SOFTplus GSiteCrawler Google Group

Sitemaps for folders

2snap...@gmail.com (MrMyckster) — Wed, 11 Nov 2009 10:22:59 UT

If you decide to separate your site into multiple sub-sites (maybe
based on sub-folders), then you can build a site-map for each such
sub-site, according to your own schedule. Again the same deal, each
site-map can be a site-map index with individual site-maps gzipped.

Can you give me more information on how to do this?

Re: Heavy Site Map Issues

web...@gmail.com (webado) — Tue, 10 Nov 2009 13:35:24 UT

Hi Karthick,

First of all I have to say I have no first hand knowledge of how to
manage such very large sites. The largest site I have has about 12000
urls, easily managed by GSC, though it takes 3 hours or so to recrawl.

GsiteCrawler has the option of making sitemap indexes for multiple
sitemaps.

Heavy Site Map Issues

sifychen...@gmail.com (Karthick) — Tue, 10 Nov 2009 07:24:50 UT

Hi There

I am karthick working for Sify Technologies India. I work for the
domain sify.com it is one of the premier portal in India we have
decided to create sitemap for our domain, We have many channels like
sports, news, finance, movies. we have planned to create separate
sitemap for all channels. The issue here is there will be thousands of

Re: [GSiteCrawler] Re: Issue with ftp upload using explicit SSL

web...@gmail.com (Christina S) — Fri, 30 Oct 2009 05:23:54 UT

I cannot check because I don't have an SSL cert anywhere.

I fail to see how the upload function can change protocol - I thought that
was established at connect time. But what do I know ...

----- Original Message -----
To: "SOFTplus GSiteCrawler" <gsitecrawler@googlegroups.com >
Sent: Thursday, October 29, 2009 9:40 AM

Re: Issue with ftp upload using explicit SSL

spen...@3ex.co.uk (spencer@3ex) — Thu, 29 Oct 2009 13:40:55 UT

I dont think the issue is with connecting to the server using ftpes as
this works. We can see from the log that we connect correctly to the
server.

Where it goes wrong is when we try to upload the file. At this point
the log indicates

534 Policy requires SSL.
Put failed: 534 Policy requires SSL.

Re: Issue with ftp upload using explicit SSL

web...@gmail.com (webado) — Wed, 28 Oct 2009 12:37:05 UT

Sorry, I cannot tell, I don't use this for my server.

But does your ftp address reflect the SSL ? Are you using ftps://....
for the host address?

Issue with ftp upload using explicit SSL

spen...@3ex.co.uk (spencer@3ex) — Wed, 28 Oct 2009 12:10:23 UT

Hello,

we are very new to gsite crawler so please bear with us.

We have to use explicit SSL to upload files to our website.

when we try to upload the test file we get the following error

FTP Connection 28/10/2009 11:54
GSiteCrawler v1.23 rev. 286
------------------------------ ----------

Re: [GSiteCrawler] Re: Recrawl question(s)

motorheadextraordina...@gmail.com (Joe Germann) — Fri, 23 Oct 2009 12:15:17 UT

Many thanks.

Joe

MOTORHEAD extraordinaire
Professional Storage and Workspace Solutions
79 Park Road - Chelmsford, MA - 01824
Toll Free 800.618.8028 - Direct 978.618.2800 - Fax 978.418.0404
Visit our web site at [link] and
for our latest specials,
<[link]>sign up

Re: [GSiteCrawler] Recrawl question(s)

web...@gmail.com (Christina S) — Fri, 23 Oct 2009 04:45:00 UT

You can delete all the currently listed urls from URL List (option Delete
all non-manual urls) and start a fresh crawl after that.

----- Original Message -----
To: "SOFTplus GSiteCrawler" <gsitecrawler@googlegroups.com >
Sent: Thursday, October 22, 2009 5:53 PM

Christina
[link]

Recrawl question(s)

motorheadextraordina...@gmail.com (Motorhead Extraordinaire) — Thu, 22 Oct 2009 21:53:18 UT

I just did a major reorganization of my eCommerce web site and moved
categories and products all around. I also generated a .htaccess file
that did a "Redirect 301 From To" for everything that moved about.

I just kicked off a recrawl with GSiteCrawler and it looks like GSC is
crawling the old URL's frorm what was last in the database. These must

Re: [GSiteCrawler] Re: robots.txt and web site remap

j...@motorheadextraordinaire.com (Joe Germann) — Tue, 20 Oct 2009 05:38:31 UT

Works great. Thanks a bunch.

Joe

Re: robots.txt and web site remap

web...@gmail.com (webado) — Mon, 19 Oct 2009 19:58:12 UT

Use [link] and put in your url.

On 19 oct, 14:17, Joe Germann <motorheadextraordina...@gmail .com>
wrote:

Re: [GSiteCrawler] Re: robots.txt and web site remap

motorheadextraordina...@gmail.com (Joe Germann) — Mon, 19 Oct 2009 18:17:45 UT

I implemented the following .htaccess and it appears to work just
fine for users. I can plug in my IP and still get access to my web site.
Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT}
^.*(Googlebot|Googlebot|Mediap artners|Adsbot|Feedfetcher)-?( Google|Image)?

Re: [GSiteCrawler] Re: robots.txt and web site remap

j...@motorheadextraordinaire.com (Joe Germann) — Mon, 19 Oct 2009 01:00:41 UT

It looks like it; just have to play around a bit to figure it all out.

Joe

Re: [GSiteCrawler] Re: robots.txt and web site remap

web...@gmail.com (Christina S) — Mon, 19 Oct 2009 00:55:38 UT

Yes, one of those examples should work for you.

----- Original Message -----
From: Joe Germann
To: gsitecrawler@googlegroups.com
Sent: Sunday, October 18, 2009 8:44 PM
Subject: [GSiteCrawler] Re: robots.txt and web site remap

Thanks for the guidance. I am investigating how to properly set us a .htaccess to do this. It looks like it is straight forward. I just have to read up a bit more and set up a test scenario. [link]