Bots don't honor 301 :(

[H. Kurth Bemis]

What about a perl (or python, ruby etc) script that will tail your
error_log, watching for multiple 404's coming from the same IP within a
given timeframe.  If the IP is tripping too many 404's for things that
don't exist, add them to the DROP chain.

I solved a similar problem using iptables rate limiting feature.  Just
slows down the attempts from hundreds/night to about ~8/night.

Just a thought..
~kurth

On Sat, 2009-01-10 at 15:27 +0000, VirginSnow at vfemail.net wrote:
> My httpd logs have been bombarded, lately, with probes by crackbots
> (mostly for roundcube webmail and mantis bugtracker exploits).  This
> got me wondering, "What can I do to keep these buggers off my server?"
> 
> Of course, the iptables -j TARPIT approach came to mind, but that
> didn't quite seem creative enough.  Besides, what if one of the
> compromised hosts legitimately wants to browse one of my sites?  So I
> got the idea to use status code 301 to redirect these bots to
> something fun, like:
> 
>   http://cybercrime.fbi.gov/complaints/submit_complaint.php?message=i+am+a+script+kidde+or+robot+attempting+to+compromise+a+computer+at+IP+address,+the+URL+i+am+using+to+do+this+is+$1
> 
> So, I set up my servers to trap exploit URLs and 301 them to another
> server that I control.  However, the bots didn't respect the 301, and
> seemed to treat the 301 much like a 404. :(
> 
> So, "what if I use a fastcgi program to send the bot a 200 response
> with a new Location: header", I wonder.
> 
> Has anyone on this list found any fun ways to burn these bots?
> 
> (BTW, legitimate bots, like googlebot, *do* honor status code 301.)
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/