Ultimate Security Proxy With Tor

Nowadays, within the growing web 2.0 environment you may want to have some anonymity, and use other IP addresses than your own IP. Or, for some special purposes - a few IPs or more, frequently changed. So no one will be able to track you. A solution exists, and it is called Tor Project, or simply tor. There are a lot of articles and howtos giving you the idea of how it works, I'm not going to describe here onion routing and its principles, I'll rather tell you how practically pull out the maximum out of it.

So, let's start.

Major disadvantages of tor, with all its benefits like security (however, tor manufacturers advice not to rely on its strong anonymity, because there are some cases, when it is possible to track young h4x0r...) are:

1. Long enough session (60 seconds for one connection per one exit node, that means, that you will have one external IP for a minute)

2. Slow perfomance (request takes up to 20 seconds to complete, what makes surfing sites with lot of elements a disaster)

3. All the requests come through one node, and possibly route how your requests arrive can be calculated.

In order to fight these three we are going to use:

1. 8 tor processes, each using separate spool directory

2. 8 privoxy processes, each configured to talk to separate tor.

3. First squid, with malware domains blacklist, will have 8 round robin cache peers configured. (squid-in)

4. Havp, with squid-in as parent. (Anti-virus proxy, using clamav :) )

5. Second squid, that will use havp as parent (squid-out). Users will connect to this one.

Schema:

Okay, so you got it, how all it looks like. Now let's do all the required configuration.

For squid-in about 2Gb of disc cache will be enough.

For squid-out it is better to have about 12-15 Gb of disc cache to store clean elements.

So you'll need at least 20 Gb HDD.



Now about RAM.

Each tor process consumes ~ 6 Mb of Ram. 6x8 = 48 ~ 64 Mb of RAM for Tor processes.

Privoxy eats about 2 Mb each. 2x8 = 16 ~ 32 Mb of RAM.

So, 128 Mb of RAM will be enough for Tor and Privoxy processes.

Squid-in will have memory footprint approx 128 Mb. We assume it eats 256 Mb.

Same for Squid-out. We got 768 Mb of RAM total required. If it is gonna be a real system - we assume 1 Gb hardware RAM

installed.

CPU.

As you know, anti-virus scanning is very CPU demanding.

I've used a 2.0 GHz P4.

I've tested this system under concurrent load under 10 users, and it did now show it was exhausted.

Finally, we have:

CPU: 2.0 GHz

RAM: 1.0 Gb

HDD: 20 Gb

Used software:

Privoxy

privoxy --version

Privoxy version 3.0.8 (http://www.privoxy.org/)

AdvertisementAdvertisement

Tor

tor --version

Oct 11 19:49:27.394 [notice] Tor v0.1.2.19. This is experimental software. Do not rely on it for strong anonymity.

Tor version 0.1.2.19.

Squid

squid -v

Squid Cache: Version 2.7.STABLE3

configure options: '--prefix=/usr' '--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man'

'--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib'

'--sysconfdir=/etc/squid' '--libexecdir=/usr/libexec/squid' '--localstatedir=/var'

'--datadir=/usr/share/squid' '--enable-auth=basic,digest,ntlm' '--enable-removal-policies=lru,heap'

'--enable-digest-auth-helpers=password'

'--enable-basic-auth-helpers=SASL,PAM,SMB,multi-domain-NTLM,getpwnam,NCSA,MSNT'

'--enable-external-acl-helpers=wbinfo_group,ip_user,session,unix_group'

'--enable-ntlm-auth-helpers=SMB,fakeauth' '--enable-ident-lookups' '--enable-useragent-log'

'--enable-cache-digests' '--enable-delay-pools' '--enable-referer-log' '--enable-arp-acl' '--with-pthreads'

'--with-large-files' '--enable-htcp' '--enable-carp' '--enable-follow-x-forwarded-for' '--with-maxfd=8192'

'--enable-snmp' '--enable-ssl' '--enable-storeio=ufs,diskd,coss,aufs,null' '--enable-async-io'

'--enable-linux-netfilter' '--libdir=/usr/lib64' '--build=x86_64-pc-linux-gnu'

'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu' 'CC=x86_64-pc-linux-gnu-gcc'

'CFLAGS=-march=k8 -O2 -pipe -fomit-frame-pointer' 'LDFLAGS=-Wl,-O1'

Tor Configuration

/etc/tor/torrc

SocksPort 9050 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor

/etc/tor/torrc2

SocksPort 9150 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor2 ControlPort 9151

/etc/tor/torrc3

SocksPort 9250 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor3 ControlPort 9251

/etc/tor/torrc4

SocksPort 9350 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor4 ControlPort 9351

/etc/tor/torrc5

SocksPort 9450 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor5 ControlPort 9451

/etc/tor/torrc6

SocksPort 9550 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor6 ControlPort 9551

/etc/tor/torrc7

SocksPort 9650 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor7 ControlPort 9651

/etc/tor/torrc8

SocksPort 9750 # what port to open for local application connections SocksBindAddress 127.0.0.1 # accept connections only from localhost AllowUnverifiedNodes middle,rendezvous Log notice syslog RunAsDaemon 1 User tor Group tor DataDirectory /var/lib/tor8 ControlPort 9751

Configuration files for privoxy

/etc/privoxy/config

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8118 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9050 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0

/etc/privoxy/config2

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy2 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8129 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9150 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0

/etc/privoxy/config3

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy3 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8230 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9250 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0

/etc/privoxy/config4

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy4 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8321 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9350 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0

/etc/privoxy/config5

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy5 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8421 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9450 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0

/etc/privoxy/config6

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy6 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8522 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9550 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0

/etc/privoxy/config7

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy7 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8623 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9650 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0

/etc/privoxy/config8

trust-info-url http://www.example.com/why_we_block.html trust-info-url http://www.example.com/what_we_allow.html confdir /etc/privoxy logdir /var/log/privoxy8 actionsfile standard.action # Internal purpose, recommended actionsfile default.action # Main actions file actionsfile user.action # User customizations filterfile default.filter logfile logfile listen-address 127.0.0.1:8724 toggle 1 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 enforce-blocks 0 buffer-limit 4096 forward-socks4a / 127.0.0.1:9750 . forwarded-connect-retries 0 accept-intercepted-requests 0 allow-cgi-request-crunching 0 split-large-forms 0


Startup scripts

Privoxy: /etc/rc.d/privoxy (Arch linux)

#!/bin/bash # source application-specific settings [ -f /etc/conf.d/privoxy ] && . /etc/conf.d/privoxy . /etc/rc.conf . /etc/rc.d/functions #PID=`pidof -o %PPID /usr/sbin/privoxy` start() { stat_busy "Starting Privoxy, squid, havp" chown privoxy:privoxy /var/log/privoxy # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy/$i chgrp $PRIVOXY_GROUP /var/log/privoxy/$i chmod 660 /var/log/privoxy/$i done chown privoxy:privoxy /var/log/privoxy2 # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy2/$i chgrp $PRIVOXY_GROUP /var/log/privoxy2/$i chmod 660 /var/log/privoxy2/$i done chown privoxy:privoxy /var/log/privoxy3 # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy3/$i chgrp $PRIVOXY_GROUP /var/log/privoxy3/$i chmod 660 /var/log/privoxy3/$i done chown privoxy:privoxy /var/log/privoxy4 # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy4/$i chgrp $PRIVOXY_GROUP /var/log/privoxy4/$i chmod 660 /var/log/privoxy4/$i done chown privoxy:privoxy /var/log/privoxy5 # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy5/$i chgrp $PRIVOXY_GROUP /var/log/privoxy5/$i chmod 660 /var/log/privoxy5/$i done chown privoxy:privoxy /var/log/privoxy6 # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy6/$i chgrp $PRIVOXY_GROUP /var/log/privoxy6/$i chmod 660 /var/log/privoxy6/$i done chown privoxy:privoxy /var/log/privoxy7 # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy7/$i chgrp $PRIVOXY_GROUP /var/log/privoxy7/$i chmod 660 /var/log/privoxy7/$i done chown privoxy:privoxy /var/log/privoxy8 # create missing logfiles for i in logfile jarfile; do touch /var/log/privoxy8/$i chgrp $PRIVOXY_GROUP /var/log/privoxy8/$i chmod 660 /var/log/privoxy8/$i done #[ -z "$PID" ] && /usr/sbin/privoxy $PRIVOXY_ARGS 2>/dev/null /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy.pid /etc/privoxy/config /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy2.pid /etc/privoxy/config2 /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy3.pid /etc/privoxy/config3 /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy4.pid /etc/privoxy/config4 /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy5.pid /etc/privoxy/config5 /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy6.pid /etc/privoxy/config6 /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy7.pid /etc/privoxy/config7 /usr/sbin/privoxy $PRIVOXY_ARGS --pidfile /var/run/privoxy8.pid /etc/privoxy/config8 havp -c /etc/havp/havp.config squid -f /etc/squid/squid-in.conf -DYC squid -f /etc/squid/squid-out.conf -DYC if [ $? -gt 0 ]; then stat_fail else add_daemon privoxy stat_done fi } stop() { stat_busy "Stopping Privoxy, Squid, Havp" #[ ! -z "$PID" ] && kill $PID &> /dev/null killall privoxy killall squid killall havp if [ $? -gt 0 ]; then stat_fail else rm_daemon privoxy stat_done fi } case "$1" in start) start ;; stop) stop ;; restart) $0 stop sleep 1 $0 start ;; *) echo "usage: $0 {start|stop|restart}" esac exit 0

Tor startup script: /etc/rc.d/tor (Arch linux)

#!/bin/bash . /etc/rc.conf . /etc/rc.d/functions #PID=`pidof -o %PPID /usr/bin/tor` tor='/usr/bin/tor' case "$1" in start) stat_busy "Starting Tor Daemon" #[ -z "$PID" ] && /usr/bin/tor &>/dev/null ${tor} -f /etc/tor/torrc ${tor} -f /etc/tor/torrc2 ${tor} -f /etc/tor/torrc3 ${tor} -f /etc/tor/torrc4 ${tor} -f /etc/tor/torrc5 ${tor} -f /etc/tor/torrc6 ${tor} -f /etc/tor/torrc7 ${tor} -f /etc/tor/torrc8 if [ $? -gt 0 ]; then stat_fail else add_daemon tor stat_done fi ;; stop) stat_busy "Stopping Tor Daemon" #[ ! -z "$PID" ] && kill $PID &> /dev/null killall tor if [ $? -gt 0 ]; then stat_fail else rm_daemon tor stat_done fi ;; restart) $0 stop sleep 3 $0 start ;; *) echo "usage: $0 {start|stop|restart}" esac exit 0 # vim: ft=sh ts=2 sw=2

Squid-in config

/etc/squid/squid-in.conf

acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports acl malware_domains url_regex '/etc/squid/Malware-domains.txt' http_access deny malware_domains http_access allow localhost http_access deny all icp_access deny all http_port 3400 icp_port 0 hierarchy_stoplist cgi-bin ? refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache_peer localhost parent 8118 0 round-robin no-query cache_peer localhost2 parent 8129 0 round-robin no-query cache_peer localhost3 parent 8230 0 round-robin no-query cache_peer localhost4 parent 8321 0 round-robin no-query cache_peer localhost5 parent 8421 0 round-robin no-query cache_peer localhost6 parent 8522 0 round-robin no-query cache_peer localhost7 parent 8623 0 round-robin no-query cache_peer localhost8 parent 8724 0 round-robin no-query never_direct allow all always_direct deny all acl apache rep_header Server ^Apache broken_vary_encoding allow apache forwarded_for off coredump_dir /var/cache/squid-in cache_dir ufs /var/cache/squid-in 100 16 256 pid_filename /var/run/squid-in.pid access_log /var/log/squid/access.squid-in.log cache_store_log /var/log/squid/store.squid-in.log cache_log /var/log/squid/cache.squid-in.log

Squid-out config

/etc/squid/squid-out.conf

acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access allow localhost http_access deny all icp_access deny all http_port 3128 icp_port 0 hierarchy_stoplist cgi-bin ? refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache_peer localhost parent 3410 0 round-robin no-query never_direct allow all always_direct deny all acl apache rep_header Server ^Apache broken_vary_encoding allow apache forwarded_for off coredump_dir /var/cache/squid-out cache_dir ufs /var/cache/squid-out 100 16 256 pid_filename /var/run/squid-out.pid access_log /var/log/squid/access.squid-out.log cache_store_log /var/log/squid/store.squid-out.log cache_log /var/log/squid/cache.squid-out.log

Havp antivirus proxy config

/etc/havp/havp.config

USER havp GROUP havp DAEMON true PIDFILE /tmp/havp.pid SERVERNUMBER 4 MAXSERVERS 256 SCANTEMPFILE /tmp/havp-XXXXXX TEMPDIR /tmp DBRELOAD 60 PARENTPROXY localhost PARENTPORT 3400 FORWARDED_IP false X_FORWARDED_FOR false PORT 3410 BIND_ADDRESS 127.0.0.1 ENABLECLAMLIB true ENABLECLAMD false ENABLEFPROT false ENABLEAVG false ENABLEAVESERVER false ENABLESOPHIE false ENABLETROPHIE false ENABLENOD32 false ENABLEAVAST false ENABLEARCAVIR false ENABLEDRWEB false

#!/usr/bin/python # -*- coding: utf-8 -*- # Author: Bohdan Turkynewych, AKA Gh0st, 2006-2008, tb0hdan[at]gmail[dot]com # Distributed under the terms of GPLv2 # Parts of code from neutron jabber bot project, http://code.google.com/p/neutron/ from urllib2 import Request as urllib2_Request, urlopen as urllib2_urlopen from sys import exit as sys_exit from re import search as re_search from time import ctime as time_ctime, time as time_time from re import compile as re_compile def t_conv(timestamp): reply = '' seconds = timestamp % 60 minutes = (timestamp / 60) % 60 hours = (timestamp / 3600) % 60 days = timestamp / 216000 if days: reply += str(days) + ' day(s) ' if hours: reply += str(hours) + ' hour(s) ' if minutes: reply += str(minutes) + ' minute(s) ' reply += str(seconds) + ' second(s)' return reply def uptime(): global BOOTUP_TIMESTAMP if BOOTUP_TIMESTAMP: idletime = int(time_time() - BOOTUP_TIMESTAMP) reply = t_conv(idletime) else: reply = 'Unknown' return reply strip_tags = re_compile(r'<[^]+>') def decode(text): data = text.replace('<br>','

').replace(' ', ' ').replace('<', '<').replace('>', '>').replace('"', '"').replace('<br>','

').replace(' <li>','\r

') return strip_tags.sub('', data) def get_data(message): reply=[] try: for line in message.split('

'): if re_search('wbr',line): # od = re_search('',line) d0main = line[od.end():] d0main = d0main[:re_search('',d0main).start()].strip() # if d0main == '-': od = re_search('-',line) bookurl = line[od.end():] bookurl = bookurl[:re_search('',bookurl).start()].strip() bookurl = decode(bookurl) else: bookurl = decode(d0main) d0main = '' stripper = '[/|:]' if re_search(stripper,bookurl): #d0main = bookurl od = re_search(stripper,bookurl) d0main = bookurl[od.end():] d0main = bookurl[:re_search(stripper,bookurl).start()].strip() else: d0main = bookurl if not d0main in reply: reply.append(d0main) return reply except: raise def update_list(filename): req2 = urllib2_Request('http://www.malwaredomainlist.com/mdl.php?sort=Domain&ascordesc=DESC&search=&colsearch=All&quantity=All') req2.add_header = ('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.9) Gecko/20071030 SeaMonkey/1.1.6') print '[+] Update process started at ' + time_ctime() try: r = urllib2_urlopen(req2) target = r.read() urls = get_data(target) except: raise print '[-] Error occured while fetching / parsing data.' sys_exit(1) print '[+] Got ' + str(len(urls)) + ' domains/ips.' try: file = open(filename,'w') for line in urls: file.write(line + '

') file.close() except: print '[-] Error occured while flushing data to file' sys_exit(5) elapsed = uptime() print '[+] Update process finished at ' + time_ctime() + '. Took ' + elapsed BOOTUP_TIMESTAMP = time_time() update_list('/etc/squid/Malware-domains.txt')

Have fun!

P.S. Future optimisation requires 8 havp instances as well due to slowup.

P.S.S You may exclude havp and squid-out proxy from your system, and use only squid-in. That will be much, much faster.