Is this normal?

[Tom Buskey]

https://www.schneier.com/blog/archives/2013/06/more_on_feudal.html


FWIW before Google when there were lots of search engines, one company
bragged that you could toggle to ignore robots.txt.  It was considered
highly unethical and I think the company suffered.

robots.txt is just like the yellow line on the highway.  Having everyone
honor it is what keeps you from having a 50 mph head on collision but the
line won't do anything to prevent it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/mailman/private/gnhlug-discuss/attachments/20130614/5bde88f9/attachment.html