SOFTplus GSiteCrawler Google Group

Re: [GSiteCrawler] Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-20T13:30:35Z

Hi again,

The developers may eventually answer this.

It doesn't remove from the good practices.

As for the method of avoiding having duplicate content urls in the first place, what I said stands.

If those urls have already been indexed, then you need to:
1) not have them in navigation where Google et al. get at them - thus modify software to not create them at all

Re: [GSiteCrawler] Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-20T11:36:02Z

Hi KKarl,

Very interesting. I'll have to dig into Google deeper regarding
blocking access with robots.txt files.

The issue on my web site is that the OS Commerce software kicks out a
lot of different links, some of which are to URL's that I want don't
want people landing on; like into a sorted list of some random

Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-20T09:06:41Z

Webado

The developers of GSC can certainly answer my question

Joe

One example can show the complex issues:

We discovered by chance for six websites (only differing in language)
that there are a lot of duplicate URLs in the index of Google:

1. We know how this DC is generated but not why!

Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-20T06:14:39Z

kkarl, I can't answer that. As I said I don't have any sites with this
issue that need special treatment. At most I dealt with a Zen Cart
site and since I had already noticed some useless urls (different sort
orders for instance), I had already handled them by modifying the
code to add rel="nofollow" to those links. I didn't have to rely on

Re: [GSiteCrawler] Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-19T16:44:23Z

Kkarl,

I am by no means the expert on GSC but I was able to recognize URL
patterns and multiple entries that were pointing to the same
effective URL. My eCommerce PHP site code spits out quite a few
different URL link patterns. Based upon what pops up in a browser's
URL window, I chose the pattern that was consistent with the user

Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-19T15:59:53Z

Webado: I agree totally with your comments and hints

Webado & Joe: How do you know that the "identical pages listed" by
GSC are all of them - completely recognized as duplicate content by
GSC?
Thats why I am interested in knowing how GSC discovers DC, what are
the criteria !

Thx
Kkarl

Re: [GSiteCrawler] Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-19T14:27:16Z

I let GSC crawl my entire PHP eCommerce site and then built a robots
file to filter out the duplicate castings that were generated
automagically by the eCommerce site. I now use the robots.txt file
to tell both the crawlers and GSC what to crawl. Every once in a
while I will run GSC without the robots directives to make sure that

Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-19T14:14:05Z

That's why you have to fix your site to avoid generating pages with
largely the same content (e.g. sorted in different ways or with a
larger view of a product image).
Proper use of a robots noindex met atag on page you dont' want
indexed, and/or rel="nofollow" on links to alternate disaplys of the

Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-19T13:59:43Z

I believe that an answer is important because in case of thousands of
duplicate pages ( e.g. for shops) the size of the sitemap is strongly
reduced and the transferred URLs to the SEs are quasi correct after
disabling the duplicate content.

To compare the content (body...) the task would be very time

Re: Criteria for identifying identical pages (Duplicate content URLs)

2009-11-19T13:25:45Z

Same text content perhaps?

I don't know exactly because I don't have such a situation anywhere to
test it.

Criteria for identifying identical pages (Duplicate content URLs)

2009-11-19T12:33:33Z

what are the criteria GSC identifies URLs as pointing to the same
content??

Thx
Kkarl

Out of the Office

2009-11-19T08:35:28Z

Thank you for your email. I am out of the office until Monday 23rd November. Should your enquiry be urgent please contact my assistant Gillian on 01438 861418.

Kind Regards

Harriet Rearden
IGLOOS Ltd
[link]

news in GSC

2009-11-18T17:53:57Z

this is message in first for GSC team

I have to say that GSC is very good and helpful program. Thank you for
that. I have a question to you. Do you prepare a new version (update)
of this program and are you going to add new feature as SFTP?

I think that in this time is SFTP very important for transfer of ftp

Sitemaps for folders

2009-11-11T10:22:59Z

If you decide to separate your site into multiple sub-sites (maybe
based on sub-folders), then you can build a site-map for each such
sub-site, according to your own schedule. Again the same deal, each
site-map can be a site-map index with individual site-maps gzipped.

Can you give me more information on how to do this?

Re: Heavy Site Map Issues

2009-11-10T13:35:24Z

Hi Karthick,

First of all I have to say I have no first hand knowledge of how to
manage such very large sites. The largest site I have has about 12000
urls, easily managed by GSC, though it takes 3 hours or so to recrawl.

GsiteCrawler has the option of making sitemap indexes for multiple
sitemaps.