Is this normal?

James A. Kuzdrall gnhlug at intrel.com
Thu Jun 13 06:44:54 EDT 2013


Greetings,

    On 09 June I gave Jerry Feldman and the LUG community a link to my example 
of a simple, "professional-looking" web page (opinions may vary).  The link 
is private in that the top page of the web site does not link to it.

    The next day, Webalizer 2.01 showed that crawl-66-249-72-95.googlebot.com, 
Google's search bot I assume, had downloaded all 16MB of technical data on 
the site.  The site will not be ready to go public until some additional 
papers are linked, so it is not a secret, just incomplete.

    Question: How did Google get the link?  gnhlug is a public bulletin board, 
but doesn't Google promise not to search email content?

    Question: Having gotten the link, what motivated them to follow it, 
especially to go beyond the index.html page referenced?

    Question: Since the site was searched 2 days ago, why don't I get my site 
as a hit when its unique terms are entered into Google search?

    About a dozen search bots check the site regularly.  However, none but the 
Google bot has downloaded anything.  Apparently Google is keeping the site 
address to themselves.

    Does anyone care to speculate?

Jim Kuzdrall 	


More information about the gnhlug-discuss mailing list