Just a quick post about an interesting file found in a phishing kit. Bad guys use common techniques to prevent crawlers, scanners or security companies from accessing theirÂ pages. Usually, they deploy a .htaccess file to achieve this. Today, I found a phishing kit related to a bank (ANZ) with such protection. But, in this case, the attackers took the time to comment out the blocked IP addresses and user-agents. Note that they also prevent other malicious traffic (like bots) to reach them. Very interesting! Want to know who’s blocked? Have a look at the file:

<Limit GET POST> order allow,deny deny from 209.85.32.23 # totaldomaindata (checkmark) deny from 66.205.64.22 deny from 98.247.136.154 deny from 178.25.218.88 deny from 98.247.136.154 deny from 63.229.4.212 deny from 66.135.207.155 deny from 66.77.136.153 deny from 64.122.169.98 deny from 54.217.8.129 deny from 38.100.21.113 deny from 96.47.226.21 deny from 54.197.81.106 deny from 68.168.131.216 deny from 65.17.253.220 deny from 78.151.209.28 deny from 66.135.207.155 deny from 207.102.138.158 deny from 209.139.197.125 deny from 66.77.136.153 deny from 66.77.136.123 deny from 72.64.146.136 deny from 124.178.234.95 deny from 67.15.182.35 deny from 203.68. # taiwan academic network deny from 218.58.124. # china jpg giftsite spammer deny from 218.58.125. deny from 62.194.7. # NE spambot deny from 85.17.6. # netherlands deny from 194.213. # czech norway sweden etc deny from 64.27.2.18 # SEO masked as SE deny from 64.27.2.19 # SEO masked as SE deny from 212.187.116. # clown from Netherlands siphoning bible site deny from 84.87. # clown from Netherlands siphoning bible site deny from 222.252. # vietnam spammer deny from 203.160.1. # vietnam spammer deny from 82.60.1. # spamming Italy block deny from 68.46.186.93 # clown on comcast deny from 65.254.33.130 # unknown spain bot deny from 82.131.195. # hungarian BS bot deny from 217.153. # poland deny from 202.108.252. # repeated merch spam! deny from 82.208. # czech russia romania etc deny from 193.47.80.41 # BW sucking bot deny from 66.234.139.194 # bogus crawler deny from 80.96. # romania deny from 66.232.98.76 # unknown bot deny from 38.112.6.182 # cosmixcorp.com deny from 82.165.252.147 # unknown Java BW waster deny from 67.79.102.28 # blacklisted spammer deny from 220.181.26. # sohu bot deny from 64.62.136.196 # unknown stealth bot deny from 62.163. # netherlands deny from 195.113. # czech deny from 213.185.106. # nigeria deny from 213.185.107. # nigeria deny from 67.184.49.166 # blacklisted IP deny from 219.95. # malaysia deny from 66.221.106.76 # mydropbox.com deny from 81.93.165. # norway bot deny from 81.223.254. # austrian bs bot deny from 87.123.74. # patwebbot deny from 62.193.213. # french BS bot deny from 86.120. # romania deny from 86.121. deny from 86.122. deny from 86.123. deny from 86.124. deny from 86.125. deny from 86.126. deny from 86.127. deny from 220.194.54. # BS bandwidth wasting bot deny from 210.51.167. # BS bot deny from 204.14.48. # stealth bots webhost etc deny from 66.180.170.47 # development bot deny from 217.160.75.202 # bot rips way too fast deny from 84.12.54.237 # unknown clown UK deny from 65.19.154.24 # stealth bandwidth hog deny from 216.32.73.122 # stealth bot deny from 63.160.77.236 # stealth bot deny from 12.44.181.220 # unknown bot deny from 12.44.172.92 # stealth bot deny from 139.18.2. # findlinks bot deny from 70.85.193.178 # unknown bot deny from 82.80. # israel deny from 82.81. deny from 213.180.128. # poland deny from 213.180.129. deny from 213.180.130. deny from 213.180.131. deny from 66.150.55.230 # findwhat.com stealth bot deny from 67.15.175.114 # unknown bot deny from 217.113.244.119 # spanish SE deny from 194.224.199. # private spanish server deny from 81.19.66. # russia deny from 213.176.126. # iran deny from 208.223.208.181 # security-lab1.juniper.net deny from 208.223.208.182 deny from 208.223.208.183 deny from 208.223.208.184 deny from 208.223.208.185 deny from 67.167.114.21 # BS law-x.com scraper site bot deny from 194.44.42. # ukraine deny from 209.203.192. # Expedite Marketing deny from 209.203.193. deny from 209.203.194. deny from 209.203.195. deny from 209.203.196. deny from 209.203.197. deny from 209.203.198. deny from 209.203.199. deny from 209.203.200. deny from 209.203.201. deny from 209.203.202. deny from 209.203.203. deny from 209.203.204. deny from 209.203.205. deny from 209.203.206. deny from 209.203.207. deny from 64.62.175. # unknown bandwidth sucker deny from 219.136.171. # china unknown bot deny from 216.150.24.122 # sonicleads.com spambot deny from 216.150.24.123 deny from 210.14.32. # annoying philipines spammer deny from 220.132.126. # taiwan useragent = 3 deny from 66.194.6. # websense.com bandwidth waster deny from 12.17.130.27 # sitesucker deny from 65.164.129.91 deny from 207.155.199.163 deny from 208.252.91.3 deny from 198.54. # south africa scams, spam, etc deny from 66.132.132.63 # securityspace.com deny from 81.18.32. # nigeria deny from 81.18.33. deny from 81.18.34. deny from 81.18.35. deny from 81.18.36. deny from 81.18.37. deny from 81.18.38. deny from 81.18.39. deny from 81.18.40. deny from 81.18.41. deny from 81.18.42. deny from 81.18.43. deny from 81.18.44. deny from 81.18.45. deny from 81.18.46. deny from 81.18.47. deny from 192.115.134. # Israel, hacker heaven deny from 65.11.200.242 # direct revenue bot deny from 65.75.128.30 # fotopages.com deny from 204.8.168. # gator.com deny from 204.8.169. deny from 204.8.170. deny from 204.8.171. deny from 64.152.73. deny from 66.111.48.80 # spambot from russia deny from 68.211.2.61 # clown using site copier on books deny from 64.42.84.70 # addresses.com spambot deny from 67.127.13.70 # clown hitting with gethtmlcontents3 from secure site deny from 80.230. # israel deny from 80.250.32. # nigeria deny from 80.250.33. deny from 80.250.34. deny from 80.250.35. deny from 80.250.36. deny from 80.250.37. deny from 80.250.38. deny from 80.250.39. deny from 80.250.40. deny from 80.250.41. deny from 80.250.42. deny from 80.250.43. deny from 80.250.44. deny from 80.250.45. deny from 80.250.46. deny from 80.250.47. deny from 69.28.130. # quepasa.com deny from 213.8. # israel deny from 64.42.105. # unknown speed bot deny from 141.85. # romania deny from 128.238.55. # polybot deny from 67.68.89. # unknown masking bot deny from 66.36.242.25 # unknown bot deny from 81.199. # israel nigeria etc deny from 195.111. # hungary deny from 192.115.106. # clown from Israel speed downloading deny from 204.94.59. # brandimensions.com bandwidth waster deny from 12.209.181.242 # speed ripping unknown agent deny from 217.73. # romania ukraina russia etc deny from 217.218. # iran deny from 217.219. # iran deny from 216.53.84.61 # mail.mccarter.com deny from 169.132.149.100 # www.mccarter.com - new jersey law firm deny from 213.226.16. # bulgaria deny from 216.252.167. # idiot from Ghana demands free merch for many emails deny from 65.102. # WebContent Internatioanl deny from 216.163.255.1 # rpa.metlife.com bored employees deny from 67.127.164.125 # DSL bandwidth waster deny from 193.253.199. # france SE art-online.com bandwidth waster deny from 80.179.254. # clown from Israel using downloader deny from 64.37.103. # spambots and other non customers deny from 69.61.12.100 # spambot from servershost.net deny from 69.61.12.101 deny from 66.246.43.167 deny from 64.124.14. # markmonitor.com deny from 38.144.36.11 # allresearch.com deny from 38.144.36.12 deny from 38.144.36.13 deny from 38.144.36.14 deny from 38.144.36.15 deny from 38.144.36.16 deny from 38.144.36.17 deny from 206.28.72. # gettyimages.com bandwidth waster deny from 206.28.73. deny from 206.28.74. deny from 206.28.75. deny from 206.28.76. deny from 206.28.77. deny from 206.28.78. deny from 206.28.79. deny from 209.73.228.160 # allresearch.com deny from 209.73.228.161 deny from 209.73.228.162 deny from 209.73.228.163 deny from 209.73.228.164 deny from 209.73.228.165 deny from 209.73.228.166 deny from 209.73.228.167 deny from 209.73.228.168 deny from 209.73.228.169 deny from 209.73.228.170 deny from 209.73.228.171 deny from 209.73.228.172 deny from 209.73.228.173 deny from 209.73.228.174 deny from 209.73.228.175 deny from 158.108. # thailand university deny from 168.187. # kuwait ministry of communications deny from 168.188. # korea university deny from 66.207.120.221 # net-sweeper.com deny from 66.207.120.222 deny from 66.207.120.223 deny from 66.207.120.224 deny from 66.207.120.225 deny from 66.207.120.226 deny from 66.207.120.227 deny from 66.207.120.228 deny from 66.207.120.229 deny from 66.207.120.230 deny from 66.207.120.231 deny from 66.207.120.232 deny from 66.207.120.233 deny from 66.207.120.234 deny from 66.207.120.235 deny from 167.24. # usaa.com and wastemylife.com p3p client deny from 192.118.48.247 # icomverse.com (Israel, hacker heaven) deny from 192.118.48.248 deny from 192.118.48.249 deny from 67.209.128. # clown from TX, wastes bandwidth, abusive feedback deny from 12.148.209. # NameProtect.com bandwidth waster deny from 12.148.196. # NameProtect.com bandwidth waster deny from 212.19.205. # clown from Netherlands impersonating Webcrawler! deny from 206.190.171.172 # markwatch.com bandwidth waster (4 IPs) deny from 206.190.171.173 deny from 206.190.171.174 deny from 206.190.171.175 deny from 211.157. deny from 211.74. deny from 64.14.202.182 deny from 213.219.11.19 deny from 193.220.178. # abusive crawler from Benin deny from 24.77.178.1 # abusive OK cable user deny from 68.65.53.71 # unknown user (java1.4.0_03) slowly crawling whole site! deny from 198.26.120.13 # unknown .MIL user (keeps hitting one page over and over!) deny from 63.148.99. # Cyveillance.com bandwidth waster deny from 65.118.41. # Cyveillance.com bandwidth waster deny from 192.116.85. # abusive crawler, no ref, no ua, Israel? deny from 62.119.21. # sweden including picsearch.com bot deny from 80.179.100. # Israeli bot deny from 80.248.64.50 # guestbook spambot deny from 64.106.213. # some clown in Jersey, Russian name, hammering links page deny from 62.220.103. # Iran allow from all </Limit> RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^-?$ [NC,OR] # blank user-agent RewriteCond %{HTTP_USER_AGENT} "addresses\.com" [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} "agnitum" [NC,OR] # firewall sw from Cyprus RewriteCond %{HTTP_USER_AGENT} aipbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} alkaline [NC,OR] RewriteCond %{HTTP_USER_AGENT} "almaden" [NC,OR] # IBM unknown crawler RewriteCond %{HTTP_USER_AGENT} amfibi [NC,OR] # spanish SE RewriteCond %{HTTP_USER_AGENT} "anarchie" [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} anonymous [NC,OR] RewriteCond %{HTTP_USER_AGENT} "applewebkit" [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "art-online" [NC,OR] # France SE RewriteCond %{HTTP_USER_AGENT} arikus [NC,OR] # voxel.net webhost RewriteCond %{HTTP_USER_AGENT} "aspseek" [NC,OR] # unknown agent RewriteCond %{HTTP_USER_AGENT} baidu [NC,OR] # chinese language SE RewriteCond %{HTTP_USER_AGENT} "blackbox" [NC,OR] # HTML to JPG converter RewriteCond %{HTTP_USER_AGENT} "bordermanager" [NC,OR] # Novell network controller iow workers goofing off RewriteCond %{HTTP_USER_AGENT} botswana [NC,OR] # Unknown Agent RewriteCond %{HTTP_USER_AGENT} "bravobrian" [NC,OR] # unknown agent RewriteCond %{HTTP_USER_AGENT} bruinbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} btbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} "caddbot" [NC,OR] # classified ad bot RewriteCond %{HTTP_USER_AGENT} ccubee [NC,OR] # czech crawler RewriteCond %{HTTP_USER_AGENT} cfetch [NC,OR] RewriteCond %{HTTP_USER_AGENT} cfnetwork [NC,OR] RewriteCond %{HTTP_USER_AGENT} cherry.?picker [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} cjnetworkquality [NC,OR] # cj.com bot RewriteCond %{HTTP_USER_AGENT} claria [NC,OR] # gator.com RewriteCond %{HTTP_USER_AGENT} combine [NC,OR] # swedish harvester RewriteCond %{HTTP_USER_AGENT} contactbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} convera [NC,OR] # convera.com RewriteCond %{HTTP_USER_AGENT} ConveraCrawler [NC,OR] # convera.com RewriteCond %{HTTP_USER_AGENT} cosmos [NC,OR] # xyleme.com bot RewriteCond %{HTTP_USER_AGENT} cowbot [NC,OR] # korean naver bot RewriteCond %{HTTP_USER_AGENT} cuill [NC,OR] # www.cuill.com RewriteCond %{HTTP_USER_AGENT} crescent [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} dattatec [NC,OR] # argentina bot RewriteCond %{HTTP_USER_AGENT} deepak [NC,OR] # research bot from California RewriteCond %{HTTP_USER_AGENT} dloader [NC,OR] # unknown downloader RewriteCond %{HTTP_USER_AGENT} "^DA \d\.\d " [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "DTS Agent" [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "^download" [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} diamond [NC,OR] # gator.com RewriteCond %{HTTP_USER_AGENT} dtaagent [NC,OR] # bot grabs too fast RewriteCond %{HTTP_USER_AGENT} dumbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} easydl [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} e?mail.?(collector|magnet|reaper|siphon|sweeper|harvest|collect|wolf) [NC,OR] # spambots RewriteCond %{HTTP_USER_AGENT} "Educate Search" [NC,OR] # guestbook spambot RewriteCond %{HTTP_USER_AGENT} ejupiter [NC,OR] # pathetic SE RewriteCond %{HTTP_USER_AGENT} entrieva [NC,OR] RewriteCond %{HTTP_USER_AGENT} exava.com [NC,OR] RewriteCond %{HTTP_USER_AGENT} experimental [NC,OR] RewriteCond %{HTTP_USER_AGENT} expired [NC,OR] RewriteCond %{HTTP_USER_AGENT} express [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} extractor [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} faxobot [NC,OR] # faxo.com RewriteCond %{HTTP_USER_AGENT} "Fetch API Request" [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "fast firstpage retriever" [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "fetchbook\.info" [NC,OR] RewriteCond %{HTTP_USER_AGENT} findexa [NC,OR] # norway SE RewriteCond %{HTTP_USER_AGENT} findlinks [NC,OR] # german experimental bot RewriteCond %{HTTP_USER_AGENT} findwhat [NC,OR] RewriteCond %{HTTP_USER_AGENT} flashget [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} FlickBot [NC,OR] # rude bot RewriteCond %{HTTP_USER_AGENT} "Franklin Locator" [NC,OR] # guestbook spambot RewriteCond %{HTTP_USER_AGENT} gais [NC,OR] # Chinese SE RewriteCond %{HTTP_USER_AGENT} gazz/ [NC,OR] # Japanese language bot RewriteCond %{HTTP_USER_AGENT} geobot [NC,OR] # spain bot RewriteCond %{HTTP_USER_AGENT} gethtmlcontent [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} getright [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} girafabot [NC,OR] # girafa.com SE thingy RewriteCond %{HTTP_USER_AGENT} giveramp [NC,OR] RewriteCond %{HTTP_USER_AGENT} go.?zilla [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} gonzo [NC,OR] RewriteCond %{HTTP_USER_AGENT} grabber [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "green research" [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} "green research, inc." [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} gulper [NC,OR] RewriteCond %{HTTP_USER_AGENT} harvest [NC,OR] RewriteCond %{HTTP_USER_AGENT} hloader [NC,OR] # unknown downloader RewriteCond %{HTTP_USER_AGENT} hoowwwer [NC,OR] # finnish SE RewriteCond %{HTTP_USER_AGENT} html2jpg [NC,OR] # HTML to JPG converter RewriteCond %{HTTP_USER_AGENT} htmlparser [NC,OR] RewriteCond %{HTTP_USER_AGENT} "http generic" [NC,OR] # Unknown agent RewriteCond %{HTTP_USER_AGENT} httpclient [NC,OR] # OD Webdown RewriteCond %{HTTP_USER_AGENT} httprequest [NC,OR] RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} ia_archiver [NC,OR] RewriteCond %{HTTP_USER_AGENT} ichiro [NC,OR] # Japanese language bot (see gazz) RewriteCond %{HTTP_USER_AGENT} "ie plagin" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "ie plugin" [NC,OR] RewriteCond %{HTTP_USER_AGENT} imagefetch [NC,OR] # rude bot RewriteCond %{HTTP_USER_AGENT} "Indy Library" [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} "Industry Program" [NC,OR] # guestbook spambot RewriteCond %{HTTP_USER_AGENT} "^internet explorer$" [NC,OR] # BS agent RewriteCond %{HTTP_USER_AGENT} ineturl [NC,OR] RewriteCond %{HTTP_USER_AGENT} innerprise [NC,OR] # innerprise.net RewriteCond %{HTTP_USER_AGENT} irlbot [NC,OR] # research bot RewriteCond %{HTTP_USER_AGENT} ithenticate [NC,OR] # iThenticate spybot RewriteCond %{HTTP_USER_AGENT} iupui [NC,OR] # Unknown research (spam?) bot RewriteCond %{HTTP_USER_AGENT} java [NC,OR] # generic textbook bots RewriteCond %{HTTP_USER_AGENT} jetbot [NC,OR] # Unknown private SE RewriteCond %{HTTP_USER_AGENT} joedog [NC,OR] RewriteCond %{HTTP_USER_AGENT} k2spider [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} kuloko [NC,OR] # kuloko.com RewriteCond %{HTTP_USER_AGENT} lanshan [NC,OR] RewriteCond %{HTTP_USER_AGENT} lcabotaccept [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR] # unknown (spambot) RewriteCond %{HTTP_USER_AGENT} lapozz [NC,OR] # BS hungarian bot RewriteCond %{HTTP_USER_AGENT} law-x [NC,OR] # scraper site bot RewriteCond %{HTTP_USER_AGENT} linksmanager [NC,OR] # linksmanager.com spambot RewriteCond %{HTTP_USER_AGENT} linkwalker [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} lmcrawler [NC,OR] RewriteCond %{HTTP_USER_AGENT} lmqueuebot [NC,OR] RewriteCond %{HTTP_USER_AGENT} loopimprovements [NC,OR] RewriteCond %{HTTP_USER_AGENT} "lwp\:\:simple" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "lwp-trivial" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "Mac Finder" [NC,OR] # guestbook spambot RewriteCond %{HTTP_USER_AGENT} "Microsoft URL Control" [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} "mister pix" [NC,OR] # rude bot RewriteCond %{HTTP_USER_AGENT} "missauga" [NC,OR] # guestbook spambot RewriteCond %{HTTP_USER_AGENT} "missigua" [NC,OR] # guestbook spambot RewriteCond %{HTTP_USER_AGENT} madlyrics [NC,OR] # Winamp downloader RewriteCond %{HTTP_USER_AGENT} marvin [NC,OR] # danish/whoever bot RewriteCond %{HTTP_USER_AGENT} microsoftprototypecrawler [NC,OR] RewriteCond %{HTTP_USER_AGENT} minirank [NC,OR] RewriteCond %{HTTP_USER_AGENT} miva [NC,OR] RewriteCond %{HTTP_USER_AGENT} mizzu [NC,OR] # Mizzu Labs bot RewriteCond %{HTTP_USER_AGENT} mj12 [NC,OR] RewriteCond %{HTTP_USER_AGENT} majestic [NC,OR] RewriteCond %{HTTP_USER_AGENT} mogren [NC,OR] # russian bot RewriteCond %{HTTP_USER_AGENT} "mozilla\(ie compatible\)" [NC,OR] # BS agent RewriteCond %{HTTP_USER_AGENT} MSIECrawler [NC,OR] # IE's "make available offline" mode RewriteCond %{HTTP_USER_AGENT} MSFrontPage [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} msrbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} msproxy [NC,OR] # discontinued proxy software RewriteCond %{HTTP_USER_AGENT} msx [NC,OR] # unknown agent RewriteCond %{HTTP_USER_AGENT} mvaclient [NC,OR] RewriteCond %{HTTP_USER_AGENT} "my session" [NC,OR] # unknown agent RewriteCond %{HTTP_USER_AGENT} "NASA Search" [NC,OR] # bogus clown on comcast RewriteCond %{HTTP_USER_AGENT} netresearchserver [NC,OR] RewriteCond %{HTTP_USER_AGENT} netsprint [NC,OR] RewriteCond %{HTTP_USER_AGENT} netwhat [NC,OR] RewriteCond %{HTTP_USER_AGENT} nextgensearch [NC,OR] # BW waster RewriteCond %{HTTP_USER_AGENT} nusearch [NC,OR] # spider OD RewriteCond %{HTTP_USER_AGENT} nutch [NC,OR] # experimental bot RewriteCond %{HTTP_USER_AGENT} ocelli [NC,OR] # www.globalspec.com RewriteCond %{HTTP_USER_AGENT} offline [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} omniexplorer [NC,OR] # useless bot RewriteCond %{HTTP_USER_AGENT} "onsinn.de" [NC,OR] RewriteCond %{HTTP_USER_AGENT} outfoxbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} nameprotect [NC,OR] # NameProtect spybot RewriteCond %{HTTP_USER_AGENT} naver [NC,OR] # Korean robot RewriteCond %{HTTP_USER_AGENT} net.?(ants|mechanic|spider|vampire|zip) [NC,OR] # ODs RewriteCond %{HTTP_USER_AGENT} netcaptor [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} nicebot [NC,OR] # stealth bot RewriteCond %{HTTP_USER_AGENT} nicerspro [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} ninja [NC,OR] # Download Ninja OD RewriteCond %{HTTP_USER_AGENT} nobody [NC,OR] # Unknown Agent RewriteCond %{HTTP_USER_AGENT} noxtrum [NC,OR] # spanish private server RewriteCond %{HTTP_USER_AGENT} NPBot [NC,OR] # NameProtect spybot RewriteCond %{HTTP_USER_AGENT} "\ obot" [NC,OR] # Unknown bot RewriteCond %{HTTP_USER_AGENT} "^obot$" [NC,OR] # Unknown bot RewriteCond %{HTTP_USER_AGENT} openfind [NC,OR] # taiwan bot RewriteCond %{HTTP_USER_AGENT} panopy [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} patwebbot [NC,OR] # bs bot from germany RewriteCond %{HTTP_USER_AGENT} peerfactor [NC,OR] RewriteCond %{HTTP_USER_AGENT} pipeline [NC,OR] # cable account based SE RewriteCond %{HTTP_USER_AGENT} plink [NC,OR] # stealth bot RewriteCond %{HTTP_USER_AGENT} "program shareware" [NC,OR] # guestbook spambot RewriteCond %{HTTP_USER_AGENT} plantynet [NC,OR] # Korean bot RewriteCond %{HTTP_USER_AGENT} "poe-component-client" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "polybot" [NC,OR] # cis.poly.edu RewriteCond %{HTTP_USER_AGENT} psbot [NC,OR] # Picture Downloader RewriteCond %{HTTP_USER_AGENT} picsearch [NC,OR] # Picture Downloader RewriteCond %{HTTP_USER_AGENT} qarp [NC,OR] RewriteCond %{HTTP_USER_AGENT} qcreep [NC,OR] # quepasa in disguise RewriteCond %{HTTP_USER_AGENT} quepasa [NC,OR] # SouthAmerican bot RewriteCond %{HTTP_USER_AGENT} "safari" [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "^sew$" [NC,OR] # unknown agent RewriteCond %{HTTP_USER_AGENT} rampybot [NC,OR] RewriteCond %{HTTP_USER_AGENT} research [NC,OR] RewriteCond %{HTTP_USER_AGENT} sbider [NC,OR] RewriteCond %{HTTP_USER_AGENT} schibstedsok [NC,OR] RewriteCond %{HTTP_USER_AGENT} "scientec.de" [NC,OR] RewriteCond %{HTTP_USER_AGENT} scspider [NC,OR] # SpamBot RewriteCond %{HTTP_USER_AGENT} scumbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} search-o-rama [NC,OR] RewriteCond %{HTTP_USER_AGENT} searchsight [NC,OR] RewriteCond %{HTTP_USER_AGENT} searchwarp [NC,OR] RewriteCond %{HTTP_USER_AGENT} seekbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} seznambot [NC,OR] # czech bot RewriteCond %{HTTP_USER_AGENT} shim-crawler [NC,OR] RewriteCond %{HTTP_USER_AGENT} siphon [NC,OR] RewriteCond %{HTTP_USER_AGENT} sitemapper [NC,OR] RewriteCond %{HTTP_USER_AGENT} sitesell [NC,OR] RewriteCond %{HTTP_USER_AGENT} skywalker [NC,OR] RewriteCond %{HTTP_USER_AGENT} sleuth [NC,OR] RewriteCond %{HTTP_USER_AGENT} SlySearch [NC,OR] # SlySearch spybot RewriteCond %{HTTP_USER_AGENT} snagger [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} societyrobot [NC,OR] RewriteCond %{HTTP_USER_AGENT} "sohu agent" [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} sohu-search [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} sonicquest [NC,OR] RewriteCond %{HTTP_USER_AGENT} spider_pro [NC,OR] # innerprise.net RewriteCond %{HTTP_USER_AGENT} spiderku [NC,OR] RewriteCond %{HTTP_USER_AGENT} spiderman [NC,OR] RewriteCond %{HTTP_USER_AGENT} sproose [NC,OR] RewriteCond %{HTTP_USER_AGENT} sqworm [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} stackrambler [NC,OR] # russian bot RewriteCond %{HTTP_USER_AGENT} steeler [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} SurveyBot [NC,OR] # rude bot RewriteCond %{HTTP_USER_AGENT} szukacz [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} tcf [NC,OR] RewriteCond %{HTTP_USER_AGENT} tele(port|soft) [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} "test/0" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "test1" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "test 1" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "test rig" [NC,OR] RewriteCond %{HTTP_USER_AGENT} "tsw bot" [NC,OR] RewriteCond %{HTTP_USER_AGENT} terrawiz [NC,OR] # India SE RewriteCond %{HTTP_USER_AGENT} trademark [NC,OR] # bandwidth waster trademarktracker.com RewriteCond %{HTTP_USER_AGENT} transgenikbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} Turnitin [NC,OR] # Turnitin spybot RewriteCond %{HTTP_USER_AGENT} twiceler [NC,OR] # www.cuill.com RewriteCond %{HTTP_USER_AGENT} twotrees [NC,OR] # willow internet crawler RewriteCond %{HTTP_USER_AGENT} "under the rainbow" [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} "unknown origin" [NC,OR] # unknown bot RewriteCond %{HTTP_USER_AGENT} unchaos [NC,OR] # SE that spams web logs RewriteCond %{HTTP_USER_AGENT} url2file [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} usyd-nlp [NC,OR] # research spider RewriteCond %{HTTP_USER_AGENT} "vb openurl" [NC,OR] RewriteCond %{HTTP_USER_AGENT} visvo [NC,OR] RewriteCond %{HTTP_USER_AGENT} votay [NC,OR] RewriteCond %{HTTP_USER_AGENT} voyager [NC,OR] RewriteCond %{HTTP_USER_AGENT} w3crobot [NC,OR] RewriteCond %{HTTP_USER_AGENT} w3mir [NC,OR] # site copier RewriteCond %{HTTP_USER_AGENT} wbdbot [NC,OR] #sky.siraza.net RewriteCond %{HTTP_USER_AGENT} weasel [NC,OR] RewriteCond %{HTTP_USER_AGENT} weazel [NC,OR] RewriteCond %{HTTP_USER_AGENT} web.?(auto|bandit|collector|copier|devil|downloader|fetch|hook|mole|miner|mirror|reaper|sauger|sucker|site|snake|stripper|weasel|zip) [NC,OR] # ODs RewriteCond %{HTTP_USER_AGENT} webclipping [NC,OR] # bandwidth waster webclipping.com RewriteCond %{HTTP_USER_AGENT} webbug [NC,OR] RewriteCond %{HTTP_USER_AGENT} webcollage [NC,OR] RewriteCond %{HTTP_USER_AGENT} webindexer [NC,OR] # development bot RewriteCond %{HTTP_USER_AGENT} webpix [NC,OR] RewriteCond %{HTTP_USER_AGENT} webrace [NC,OR] # crawler RewriteCond %{HTTP_USER_AGENT} webspider [NC,OR] RewriteCond %{HTTP_USER_AGENT} websquash [NC,OR] # SEO RewriteCond %{HTTP_USER_AGENT} "wells search" [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} "wep search" [NC,OR] # spambot RewriteCond %{HTTP_USER_AGENT} wget [NC,OR] # OD RewriteCond %{HTTP_USER_AGENT} wise-guys.nl [NC,OR] # Clown in NL RewriteCond %{HTTP_USER_AGENT} "www.abot.com" [NC,OR] RewriteCond %{HTTP_USER_AGENT} xirq [NC,OR] RewriteCond %{HTTP_USER_AGENT} yottashopping [NC,OR] RewriteCond %{HTTP_USER_AGENT} zao/ [NC,OR] # experimental Japan crawler RewriteCond %{HTTP_USER_AGENT} zedzo [NC,OR] RewriteCond %{HTTP_USER_AGENT} zeus [NC,OR] RewriteCond %{HTTP_USER_AGENT} zspider [NC,OR] RewriteCond %{HTTP_REFERER} iaea.org [NC,OR] # spam bot RewriteCond %{HTTP_REFERER} wizard.yellowbrick.oz [NC,OR] # spam bot RewriteCond %{HTTP_REFERER} brandimensions [NC,OR] # bandidth waster RewriteCond %{HTTP_REFERER} imgurl= [NC,OR] RewriteCond %{HTTP_REFERER} imgrefurl= [NC,OR] RewriteCond %{REMOTE_ADDR} ^193.95.([1-2][0-9][0-9]). [NC,OR] # slovenia etc RewriteCond %{REMOTE_ADDR} ^203.147.([0-4][0-9]). [NC,OR] # thailand RewriteCond %{REMOTE_ADDR} ^80.87.([3-9][0-9]). [NC,OR] # ghana russia etc RewriteCond %{REMOTE_ADDR} ^80.88.(1[0-5][0-9]). [NC,OR] RewriteCond %{REMOTE_ADDR} ^203.87.(1[2-9][0-9]). [NC,OR] # philippines RewriteCond %{REMOTE_ADDR} ^218.(1[0-9][0-9]). [NC,OR] # china korea RewriteCond %{REMOTE_ADDR} ^211.([1-9][0-9]). [NC,OR] # china korea RewriteCond %{REMOTE_ADDR} ^66.150.55.(2[2-3][0-9]). [NC,OR] # findwhat.com stealth bot RewriteCond %{REMOTE_ADDR} ^64.110.([4-9][0-9]). [NC,OR] RewriteCond %{REMOTE_ADDR} ^64.110.(1[0-8][0-9]). [NC] RewriteRule .* - [F,L] Options -Indexes