OT: Spamming Log Files?
Jason Stephenson
jason at sigio.com
Thu May 1 09:11:08 EDT 2003
Jason Stephenson wrote:
>
> Ah-ha! But, they don't look for a robots.txt file. I have entries in my
> log file for the spiders that do look for robots.txt, and they don't
> have one. They appear to be saying one thing and doing another.
Doesn't it always work out like this? I spoke too soon. I didn't
remember seeing them looking for a robots.txt from when I checked last
night. I thought that their entries were all just HEAD requests.
After sending that message, I checked my access log, and sure enough,
they do look at robots.txt. They last checked it on the 29th.
Sorry to waste bandwidth on this. I'm going to return to looking for a
job, not that I want one, but I need one.
Heh, maybe I should try their tactic? Write a spider that spams web
companies log files with a link to my resume? ;-)
More information about the gnhlug-discuss
mailing list