Robots! What Robots?

[Glenn Brackenridge]

Hi,

going over our web hits, I keep seeing this message:

<incoming system + time> "GET /robots.txt HTTP/1.0" 404 -

What's this all about? anyone know? mail me as I don't look here too often

TIA, Glenn B

[Douglas Brick]

gl...@gbrack.demon.co.uk (Glenn Brackenridge) writes:

robots.txt is a file that tells Web searching robots what directories
they can look in and which ones they should skip, etc.

See: http://info.webcrawler.com/mak/projects/robots/robots.html

* * *

Quotation, n: The act of repeating erroneously the words of another.
Ambrose Bierce (1842-1914?): The
Devil's Dictionary

[Jeff Stevenson]

I don't know much about servers but I do recall reading somewhere in a
discussion about webcrawling robots (Lycos, Webcrawler, WWWW, etc.) that
when they "investigate" a server to document what is on it they first
search for a file that will tell them what *not* to access. Hope this
helps

Jeff
http://sapience.net/jeffs/

[David Williams]

Glenn Brackenridge wrote:
>
> Hi,

>
> going over our web hits, I keep seeing this message:
>
> <incoming system + time> "GET /robots.txt HTTP/1.0" 404 -
>
> What's this all about? anyone know? mail me as I don't look here too often
>

> TIA, Glenn B

Web-rovers or Web-bots search the internet compiling indexes and
collecting data on the net. The most notorious is digital's
scooter explorer. The robots.txt file should contain the names
of directories you do not want traversed by a "bot".

--
________________________________________________________________
//----------------------__________-------------------------------./|
|| \The Netcracker Suite\ U X \ Opinions altered and > _/|/|
|| \Symphony on the Net \ N I \ perspectives skewed / / |/|
|| \ _____________ \ I N \ near or about / _/ |/|
|| / \ (_____________)\__\ X U \ this / _/ |/|
|| / \__|____________| /__)------/\/ Signature /_/ _ |/|
|| ( )\ \ \ /\\------------------------/ /...\ |/|
|| \ / \ \ / /--------------------------- ..|-|/|
|| \ / \____________/ / da...@citylimits.com .|-|/|
|| \ / / / / http://www.citylimits.com |-|/|
|| /| / / / "The Dance of the Internet Fairy" |-|/|
|| |/^\ _ / ( /___________________________________ |-|/|
|| \ _/ / / \/____________________________________ |-|/
| \ |/ / / Voice : 909-625-4440
\ \ / / / Fax : 909-625-4441
\ \ / _ /\_/ / E-mail: dav...@citylimits.com
\ /_____/ |___\__/ Executive Vice President, Microcity
\ | / Disney & ABC, G.E. & NBC, Westinghouse & CBS; When do
\ ___|/ the Nuclear Powered Mickey Mouse Dishwasher adverts
begin?