robots.txt file

[GMiniUser]

hi,

If I want to indicate in robots.txt that only my mini can allow
certain folder. do I refer it by what is in user-agent. E.g mine says
gsa-crawler or is it something else. Also should i change the value in
the user agent and refer to that in robot.txt.


thanks

[Thiru]

Hi,

Yes. The Google Mini appliance will respect the robots.txt rules. You
are correct. The default user-agent that comes with the appliance is
"gsa-crawler". You may choose to use this string as is or change to
something else. It is up to you. More information about the robots.txt
file is explained here : http://www.robotstxt.org/orig.html

Cheers,
Thiru