GSA detected as Mobile Browser

[Thomas Fowler]

Hi Guys

We are trying to crawl a publicly available site (which we dont have any control over - we dont administer it) and the GSA is being detected as a mobile browser and is being presented with a mobile page (I can see this on the cached version of the crawled page).

Any ideas as to why this could be happening and if there is anything on the GSA I can change to alter the behavior?

Thanks again

Tom Fowler 

[Mathias Bierl]

This depends on the definitions an server side how a "mobile" browser is identified.

You can change some of the header information if the server reacts on them. But I guess you have to clarify this with the admin of the site

[Dave Watts]

By default, the GSA sends a user agent value of "gsa-crawler".
Presumably, the (probably poorly-designed) site you're crawling looks
for common desktop browsers and assumes anything else is mobile.

You can change the GSA's user agent value to that of a desktop browser
to avoid this problem. This is set under Crawl and Index ... HTTP
Headers.

As an aside, it is my understanding that if you use the GSA to crawl a
public site, you need permission from the owner of that site to do so,
according to the product license.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/
http://training.figleaf.com/

Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on
GSA Schedule, and provides the highest caliber vendor-authorized
instruction at our training centers, online, or onsite.

[Thomas Fowler]

Thanks guys.

Changing the user agent value to Begin with Mozilla worked like a charm.

Dave, re the permissions - we have the permissions to crawl the site. Its just a big corporate and sometimes its easier to get the issue resolved yourself without having to dig your way through corporate bureaucracy to get it sorted out.

Thanks for the help gents

Tom