Setup script to automatically update Dansguardian blacklists

Dansguardian website does not provide updates to the blacklists that were originally shipped in the Dansguardian package.  You have two options to download blacklists for your Dansguardian install:  urblacklist.com and shallalist.de. Unfortunately urlblacklist.com requires a comercial subscription if you wish to regularly download their blacklist. I would think you can download once and give it a try.

An alternative is to download it from www.shallalist.de.  Shalla lists can be downloaded for free if your are not using for comercial purposes.  As long as you do not download from their website too often you will probably not be blocked from downloading their lists.
 
As any third party list that you add to Dansguardian, you need to manually edit Dansguardian config files so that it is aware of the new lists.
 
Let's first download the list manually and configure Dansguardian files manually to include all Shalla listings.
 
1) Download shalla list called shallalist.tar.gz and extract all the listing directories to /etc/dansguardian/lists/BL
       cd /etc/dansguardian/lists
       tar -xvzf shallalist.tar.gz
       chown -R root/root /etc/dansguardian/lists/BL
 
2) Download  bigblacklist.tar.gz  urlblacklist.com and extract all the listing directories to /etc/dansguardian/lists/blacklists
       cd /etc/dansguardian/lists
       tar -xvzf bigblacklist.tar.gz
       chown -R root/root /etc/dansguardian/lists/blacklists
 
3) Create two files under /etc/dansguardian/lists/BL.  One file will be called bannedurllist_shalla and will contain includes for all urls files and the other will be called bannedsitelist_shalla and will have includes for all the domains files.
      .i.e. vim /etc/dansguardian/lists/BL/bannedurllist_shalla
                   .Include</etc/dansguardian/lists/BL/music/domains>
                   .Include</etc/dansguardian/lists/BL/webradio/domains>
                   .Include</etc/dansguardian/lists/BL/military/domains>
                   .Include</etc/dansguardian/lists/BL/isp/domains>
                     ......
            vim /etc/dansguardian/lists/BL/bannedsitelist_shalla
                   .Include</etc/dansguardian/lists/BL/music/urls>
                   .Include</etc/dansguardian/lists/BL/webradio/urls>
                   .Include</etc/dansguardian/lists/BL/military/urls>
                   .Include</etc/dansguardian/lists/BL/isp/urls>
                  ....
 
4) Create three files under /ect/dansguardian/lists/blacklists. The files will be called bannedurllist_urlblacklist, bannedsitelist_urlblacklist, and bannedregexpurllist_urlblacklist.  Like the previous step these file will contain includes for urls, domains, and regular expressions from urlblacklist.com.
        .i.e. vim /etc/dansguardian/blacklists/bannedregexpurllist_urlblacklist
                   .Include</etc/dansguardian/lists/blacklists/porn/expressions>
                   .Include</etc/dansguardian/lists/blacklists/malware/expressions>
                   .Include</etc/dansguardian/lists/blacklists/ads/expressions>
                   ....
               vim /etc/dansguardian/blacklists/bannedurllist_urlblacklist
                   .Include</etc/dansguardian/lists/blacklists/onlinegames/urls>
                   .Include</etc/dansguardian/lists/blacklists/naturism/urls>
                   .Include</etc/dansguardian/lists/blacklists/pets/urls>
                   .Include</etc/dansguardian/lists/blacklists/ecommerce/urls>
                   ....
                vim /etc/dansguardian/blacklists/bannedsitelist_urlblacklist
                   .Include</etc/dansguardian/lists/blacklists/clothing/domains>
                   .Include</etc/dansguardian/lists/blacklists/jewelry/domains>
                   .Include</etc/dansguardian/lists/blacklists/onlinegames/domains>
                   .Include</etc/dansguardian/lists/blacklists/naturism/domains>
                    ....
 
5) Edit existing dansguardian conf files to include these two new files you created.  This should increase the list of urls and domains being blocked.
      Add the following include at the end /etc/dansguardian/lists/bannedurllist file
             .Include</etc/dansguardian/lists/BL/bannedurllist_shalla>
             .Include</etc/dansguardian/lists/blacklists/bannedurllist_urlblacklist>
 
      Add the following include at the end of /etc/dansguardian/lists/bannedsitelist file
             .Include</etc/dansguardian/lists/BL/bannedsitelist_shalla>
             .Include</etc/dansguardian/lists/blacklists/bannedsitelist_urlblacklist>
 
       Add the following include at the end of /etc/dansguardian/lists/bannedregexpurllist
             .Include</etc/dansguardian/lists/blacklists/bannedregexpurllist_urlblacklist>
 
4) Restart dansguardian so that it becomes aware of the new lists by running the following in a terminal window
            dansguardian -r
 
Now that we have manually added new lists to dansguardian let's setup a process that automatically downloads and updates the list of banned urls and sites by following the steps below:
 
1) Download a helper called updshalla.tgz from Remigio Conti at http://www.shallalist.de/helpers.html. This is the only script available for Dansguardian. 
 
2) Install the helper in /etc/dansguardian/updshalla
    cd /etc/dansguardian
    tar -xzvf updshalla.tgz
 
3) Create an empty log file for updshalla script at /var/log/updshalla.log
    touch /var/log/updshalla.log
 
4) updshalla uses mailsend binary to send emails when errors happen.  This is an unsupported app that has not been updated since 2007 so we will replace with mutt
          pacman -S mutt
          vim ~/.muttrc
                set my_pass='mysecretpass'
                set my_user='myusername'
 
                set smtp_url=smtps://$my_user:$my_pass@smtp.gmail.com
                set ssl_force_tls = yes
 
                set realname="Your Name"
                set from=$my_user@gmail.com
 
          Test mutt from the command line
               mutt -s "Test Subject" myuser@gmail.com <<< "This is a test email"
 
5) Fix updshalla script to use mutt  
          cd  /etc/dansguardian/updshalla
          vim updshalla
             change MAILSEND='/usr/local/bin/mailsend'  to MAILSEND='/usr/bin/mutt'
             change CUT='/usr/bin/cut' to CUT='/bin/cut'
             change AWK='/usr/bin/awk' to AWK='/bin/awk'
             change
                    $MAILSEND -d $DOMAIN -smtp $SMTPHOST -p $SMTPPORT -t $DESTMAIL +cc +bc -f $SENDER -sub "$SUBJECT" -a "$LOGFILE,text/plain" -M "$BODY" 1>/dev/null 2>&1
             To  
                      $MAILSEND -s "$SUBJECT" -a "$LOGFILE" -- $DESTMAIL <<< "$BODY" 
             
             change
                     MATCHRES=`$DIFF -qr BL $LISTBASE|$AWK '/Only/ && !/COPYRIGHT/ && !/global_usage/ && !/processed/ && !/diff/'`
             To  
                     MATCHRES=`$DIFF -qr BL $LISTBASE|$AWK '/Only/ && !/COPYRIGHT/ && !/global_usage/ && !/processed/ && !/diff/ && !/bannedurllist_shalla/ && !/bannedsitelist_shalla/ && !/bannedregexpurllist_shalla/ && !/exceptionurllist_shalla/ && !/exceptionsitelist_shalla/'`
            change chown -R dansguardian:dansguardian * to chown -R root:root *
             
           Note:Attach a file to your message using MIME.  When attaching single or multiple files, separating filenames and recipient  addresses with  "--" is mandatory, e.g. mutt -a image.jpg -- addr1 or mutt -a img.jpg *.png -- addr1 addr2.  The -a option must  be  placed at the end of command line options.
 
6) Edit updshalla.conf to update the location of shalla black list in the local server
         vim updshalla.conf
             change LISTBASE='/etc/dansguardian/lists/blacklists/shalla'  to LISTBASE='/etc/dansguardian/lists/BL'
             change DESTMAIL='edpmanager@casone.it' to DESTMAIL='myuser@gmail.com'
             delete the following lines
                  # SMTP SERVER
                  SMTPHOST='smtp.example.net'
 
                   # SMTP PORT
                   SMTPPORT='25'
 
                   # SENDER DOMAIN 
                   DOMAIN='yourdomain'
 
                   # SENDER MAIL
                   SENDER='updshalla@example.net'
       
7)  Run updshalla every Sunday at midnight via crontab. crontab headers are minute,  hour, day, month, dayofweek, and command.
          crontab -e
          0 0 * * 0 ./etc/dansguardian/updshalla/updshalla
 
Additional documentation:

 

Rate: 

0
No votes yet