I like GSiteCrawler so much, I installed Parallel and XP on my Mac OS X just to use it. :-)
My site uses xcart (e-store), vbulletin and linkmachine (link management). Are there any filters I should be using to not crawl registration pages, etc.
The store items go into the hundreds and there are 6,000+ forum postings, but the site registers 20K+ pages including all the php stuff. There are about 1,000 static html pages.
Thanks for in the input.
Also, this is only related -- would you recommend using an SEO package for modifying the .php pages from xCart and vBulletin. There are add-ons for both, I am considering using them. We sell stuff, and are motivated to keep moving up in Google ranking.
After more reading, I excluded store/, forum/ and linkmachine/ and gsitecrawler shot through my static html pages, created the map, ftp'd the .xml to my site and pinged google. Not bad.
Now, I need to decide how to make sure the store and forum are indexed properly. For example, there are sitemap programs for vBulletin and X-Cart, but how do I manage three sitemaps and present it to Google correctly?
> After more reading, I excluded store/, forum/ and linkmachine/ and > gsitecrawler shot through my static html pages, created the map, ftp'd > the .xml to my site and pinged google. Not bad.
> Now, I need to decide how to make sure the store and forum are indexed > properly. For example, there are sitemap programs for vBulletin and > X-Cart, but how do I manage three sitemaps and present it to Google > correctly?
Just submit several sitemaps. For instance, I have 24 sitemaps for PhotographyCorner.com - several that are created for the static pages, several for the galleries, and a bunch for vB... You can add as many sitemaps as you need to the Google Webmaster Console - just make sure you give them different names. ;)
For the X-cart, while I'm not familiar with it myself, just browse the site for a while, and look for redundant pages with different URLs, and look for which parameters to drop, and keep adding them to GSiteCrawler until you're satisfied you're not submitting dups. It'll take a couple of run-throughs at first, but once you have it right, you'll never have to worry about it until you change your site. ;)
Ah ha. Thanks. As the webmaster, how do I go about making sure they don't overlap and create duplicate information? Do you just call them sitemap1.xml, sitemap2.xml, etc.?
For example, I could build a map for the html, the forum/ and the store/. Right?
I also want to double check. You definitely think it is better installing vBSEO and creating a new forum sitemap.xml, rather than putting a series of filters on gsitecrawler.com. Right? I am dragging my feet at learning a new tool, mods, etc. FB
Yeah... I've found the auto sitemaps built for vB, at the very least, take a lot less time time crawl (done automatically through the database)... so very important if you have a large forum.
You can pretty much name them whatever you want, but sitemap1, sitemap2, vbsitemap1, etc is what I use.
> Ah ha. Thanks. As the webmaster, how do I go about making sure they > don't overlap and create duplicate information? Do you just call them > sitemap1.xml, sitemap2.xml, etc.?
> For example, I could build a map for the html, the forum/ and the > store/. Right?
> I also want to double check. You definitely think it is better > installing vBSEO and creating a new forum sitemap.xml, rather than > putting a series of filters on gsitecrawler.com. Right? I am dragging > my feet at learning a new tool, mods, etc. > FB
Just some short comments on this, here's what I would do:
- If you can split your site into logical sections and can use "internal" or database-based sitemap generators for those sections, then I would definitely aim to do it that way. It is an absolute waste of resources to crawl a site which has the URL structure in a database already :-) (plus it takes forever for a large site)
- vbSEO is the way to go -- if I were to set up a new forum, it's probably only add-on which I would want from the start. Forums are terrible with regards to SEO, absolute crawler-traps. They need a good structure and help to push the crawlers into the right areas.
- I would set it up as follows:
/sitemap-index-forum.xml - with links to the sitemap files for the forum, perhaps in a sub-directory (which is now possible), eg: - /forum/sitemap-1.xml - /forum/sitemap-2.xml
/sitemap-index-shop.xml - ditto like above
/sitemap-general-index.xml - with all the URLs which are not in the forum or shop, crawled with the GSiteCrawler, with filters set to keep it from entering the shop or forum sections.
Then you would only need to submit those 3 sitemap-index files to Google. If you add more sitemap files, you can just adjust the sitemap-index files and ping Google about it. I hope that makes sense :-)
I don't think the forum is large enough to justify a sub-directory. Just a root sitemap.xml.
So the plan is that I will:
1. leave the gsitecrawler maps at root, as is (or rename it). 2. install vBSEO and their sitemap, and ftp that to root. 3. find, buy and install an X-Cart SEO/map tool, and do the same
I will then monitor and maintain the three sitemaps.xml.
|
