Disallowing bots (was Re: Man, they'll try anything to hack your system...)

[Bill McGonigle]

On Jan 27, 2006, at 10:41, Larry Cook wrote:

> How do you keep out the bad ones, the ones that ignore robots.txt?

The bad ones usually _read_ robots.txt to figure out where the "juicy 
stuff" is.

So you can do:

  Disallow: /robottrap.html

And then have something tail your access log and instantly iptables 
anything that accesses /robottrap.html.

-Bill

-----
Bill McGonigle, Owner           Work: 603.448.4440
BFC Computing, LLC              Home: 603.448.1668
bill at bfccomputing.com           Cell: 603.252.2606
http://www.bfccomputing.com/    Page: 603.442.1833
Blog: http://blog.bfccomputing.com/
VCard: http://bfccomputing.com/vcard/bill.vcf