Re:Crawler settings
2 views
Skip to first unread message
bigerdan
Oct 8, 2009, 11:33:22 AM10/8/09
to SOFTplus GSiteCrawler
I would to know how to set the crawler to crawl my site competely.
Joe Germann
Oct 9, 2009, 8:05:14 AM10/9/09
to gsitec...@googlegroups.com
Do not set any filters and it will crawl your entire site.
Joe
Joe
At 11:33 AM 10/8/2009, you wrote:
I would to know how to set the crawler to crawl my site competely.
MOTORHEAD extraordinaire
Professional Storage and Workspace Solutions
79 Park Road - Chelmsford, MA - 01824
Toll Free 800.618.8028 - Direct 978.618.2800 - Fax 978.418.0404
Visit our web site at www.MotorheadExtraordinaire.com and
for our latest specials, sign up for our Newsletter
webado2
Oct 9, 2009, 8:10:03 AM10/9/09
to SOFTplus GSiteCrawler
It crawls completely, subject to filters you have defined..
It also obeys robots.txt directives, robots meta tag directives and
will not crawl any links with the rel="nofollow" microformat.
It will also not find links that are opened in javascript or flash.
Finally if your navigation switches between www and non-www or jumps
around different subdomains of the same domain, only links on the
original canonical form of the domain (or subdomain) where it entered
the site will be crawled.
It also obeys robots.txt directives, robots meta tag directives and
will not crawl any links with the rel="nofollow" microformat.
It will also not find links that are opened in javascript or flash.
Finally if your navigation switches between www and non-www or jumps
around different subdomains of the same domain, only links on the
original canonical form of the domain (or subdomain) where it entered
the site will be crawled.
Reply all
Reply to author
Forward
0 new messages