robots.txt problem with google?
Bill Ricker
bill.n1vux at gmail.com
Thu Dec 14 09:30:42 EST 2006
> > My syslog tells me I have currentlt 303 hits from googlebot
> Are any of them for robots.txt? Just curious.
Good question! The follow-up is, is robots.txt 444 or 600 ?
Maybe you can see it and Google can't.
Or maybe it's the other robots.txt.
If this is the same host as your email, which it probably is since
there is a ./album/ there, you may be surprised at what you see at
http://www.syslang.net/robots.txt
- - - -
User-agent: *
-EOF-
No Disallow made it to "production".
You should have just gotten hits from pool-72.blah-blah.bos*.verizon.net ...
--
Bill
n1vux at arrl.net bill.n1vux at gmail.com
More information about the gnhlug-discuss
mailing list