robots.txt problem with google?

[Ray Cote]

At 10:34 AM -0500 12/14/06, Dan Jenkins wrote:
>Bill Ricker wrote:
>
>>>  > My syslog tells me I have currentlt 303 hits from googlebot
>>
>>>Are any of them for robots.txt?  Just curious.
>>
>There have been rumors of fake Googlebot spiders for awhile.
>I've never investigated them, so I don't know the reality of them.
>Just something to consider.

And there will probably be more fake Googlebot ids since there's a 
few articles circulating about the net showing how you can get inside 
paid sites by pretending to be Google.

This blog entry discusses the 'official' way to verify if the system 
claiming to be Googlebot is indeed Google:
<http://googlewebmastercentral.blogspot.com/2006/09/how-to-verify-googlebot.html>

-- 

Raymond Cote
Appropriate Solutions, Inc.
PO Box 458 ~ Peterborough, NH 03458-0458
Phone: 603.924.6079 ~ Fax: 603.924.8668
rgacote(at)AppropriateSolutions.com
www.AppropriateSolutions.com