sitemap
2 views
Skip to first unread message
larksaw
Oct 7, 2009, 10:11:11 AM10/7/09
to SOFTplus GSiteCrawler
I'm a real newbie to sitemaps after having my site for 3 years finally
started to try and get it better listed in the search engines. I have
done sitemaps in the past my problem is crawlers crawl to much of my
site. I know GSiteCrawler can help me correct this problem but I'm
not sure how to go about it. my site is www.poconoallseasonrentals.com
it's a php/sql database script that was created for us to list
vacation rentals. I ran GSiteCrawler on it and it list over 4000
links the site has approx 50 property detail pages not including the
main page
http://poconoallseasonrentals.com/index.php?custom_page=details&pid=108#anchor
besides the main page and a few other page the link above is a example
of the only other type pages I want the crawler to index how do I go
about doing this????
started to try and get it better listed in the search engines. I have
done sitemaps in the past my problem is crawlers crawl to much of my
site. I know GSiteCrawler can help me correct this problem but I'm
not sure how to go about it. my site is www.poconoallseasonrentals.com
it's a php/sql database script that was created for us to list
vacation rentals. I ran GSiteCrawler on it and it list over 4000
links the site has approx 50 property detail pages not including the
main page
http://poconoallseasonrentals.com/index.php?custom_page=details&pid=108#anchor
besides the main page and a few other page the link above is a example
of the only other type pages I want the crawler to index how do I go
about doing this????
webado2
Oct 7, 2009, 5:36:47 PM10/7/09
to SOFTplus GSiteCrawler
Well you need to disallow various things in the robots.txt file.
You ned to figure out what shoudl get ignored by crawlers.
# anchor fragments don't count - the actual url ends just before that.
You have to figure out what is and what is nto to be included. Only
when you have properly set up your robots.txt file woudl the site be
crawled correctly and woudl you be able to produce a good sitemap.
Sure you can fiddle with the filters in Gsitecrawler, but utimately
you have to transform all that into robots.txt directives. Google and
other search engines find and index urls by crawling the site, not
just by using the sitemap. A good robots.txt file helps focus the
crawl and discovery of urls.
Currently your robots.txt file allow everyhtings. Clearly that's not
what's needed.
On Oct 7, 10:11 am, larksaw <lark...@ptd.net> wrote:
> I'm a real newbie to sitemaps after having my site for 3 years finally
> started to try and get it better listed in the search engines. I have
> done sitemaps in the past my problem is crawlers crawl to much of my
> site. I know GSiteCrawler can help me correct this problem but I'm
> not sure how to go about it. my site iswww.poconoallseasonrentals.com
> it's a php/sql database script that was created for us to list
> vacation rentals. I ran GSiteCrawler on it and it list over 4000
> links the site has approx 50 property detail pages not including the
> main page
>
> http://poconoallseasonrentals.com/index.php?custom_page=details&pid=1...
You ned to figure out what shoudl get ignored by crawlers.
# anchor fragments don't count - the actual url ends just before that.
You have to figure out what is and what is nto to be included. Only
when you have properly set up your robots.txt file woudl the site be
crawled correctly and woudl you be able to produce a good sitemap.
Sure you can fiddle with the filters in Gsitecrawler, but utimately
you have to transform all that into robots.txt directives. Google and
other search engines find and index urls by crawling the site, not
just by using the sitemap. A good robots.txt file helps focus the
crawl and discovery of urls.
Currently your robots.txt file allow everyhtings. Clearly that's not
what's needed.
On Oct 7, 10:11 am, larksaw <lark...@ptd.net> wrote:
> I'm a real newbie to sitemaps after having my site for 3 years finally
> started to try and get it better listed in the search engines. I have
> done sitemaps in the past my problem is crawlers crawl to much of my
> site. I know GSiteCrawler can help me correct this problem but I'm
> not sure how to go about it. my site iswww.poconoallseasonrentals.com
> it's a php/sql database script that was created for us to list
> vacation rentals. I ran GSiteCrawler on it and it list over 4000
> links the site has approx 50 property detail pages not including the
> main page
>
webado2
Oct 7, 2009, 5:37:49 PM10/7/09
to SOFTplus GSiteCrawler
Also ensure all your navigation sticks to one canonical domain for:
either all www or all non-www. It seems you are mixing them up.
> > about doing this????- Hide quoted text -
>
> - Show quoted text -
either all www or all non-www. It seems you are mixing them up.
>
> - Show quoted text -
Reply all
Reply to author
Forward
0 new messages