robots.txt - which user agent string does the GSA respect?

Dirk Gomez

unread,

Oct 14, 2008, 5:24:46 AM10/14/08

to Google Search Appliance/Google Mini - Google Search Appliance/Google Mini

Regarding robots.txt's User-agent - what's the user agent string the
GSA respects? Is it gsa-crawler?

-- Dirk

jrewing

unread,

Oct 14, 2008, 6:26:00 AM10/14/08

to Google Search Appliance/Google Mini - Google Search Appliance/Google Mini

Hi!

you can set the name yourself. under Crawl and Index -> http headers

-Andreas

Dirk Gomez

unread,

Oct 14, 2008, 9:08:24 AM10/14/08

to Google Search Appliance/Google Mini - Google Search Appliance/Google Mini

Hi Andreas,

that's the user agent the GSA uses to crawl pages. I'd like to know
which user agent string in the robots.txt it respects - I'd assume it
is "gsa-crawler".

-- Dirk

jrewing

unread,

Oct 15, 2008, 3:57:29 AM10/15/08

to Google Search Appliance/Google Mini - Google Search Appliance/Google Mini

I think they are the same.

Regards,

Andreas

Dirk Gomez

unread,

Oct 15, 2008, 7:52:40 AM10/15/08

to Google Search Appliance/Google Mini - Google Search Appliance/Google Mini

It's in the docs:

http://code.google.com/apis/searchappliance/documentation/50/admin_crawl/Preparing.html#allow

User-Agent: gsa-crawler

-- Dirk

Reply all

Reply to author

Forward