SOFTplus GSiteCrawler Google Group

Re: Heavy Site Map Issues

2009-11-10T13:35:24Z

Hi Karthick,

First of all I have to say I have no first hand knowledge of how to
manage such very large sites. The largest site I have has about 12000
urls, easily managed by GSC, though it takes 3 hours or so to recrawl.

GsiteCrawler has the option of making sitemap indexes for multiple
sitemaps.

Heavy Site Map Issues

2009-11-10T07:24:50Z

Hi There

I am karthick working for Sify Technologies India. I work for the
domain sify.com it is one of the premier portal in India we have
decided to create sitemap for our domain, We have many channels like
sports, news, finance, movies. we have planned to create separate
sitemap for all channels. The issue here is there will be thousands of

Re: [GSiteCrawler] Re: Issue with ftp upload using explicit SSL

2009-10-30T05:23:54Z

I cannot check because I don't have an SSL cert anywhere.

I fail to see how the upload function can change protocol - I thought that
was established at connect time. But what do I know ...

----- Original Message -----
To: "SOFTplus GSiteCrawler" <gsitecrawler@googlegroups.com >
Sent: Thursday, October 29, 2009 9:40 AM

Re: Issue with ftp upload using explicit SSL

2009-10-29T13:40:55Z

I dont think the issue is with connecting to the server using ftpes as
this works. We can see from the log that we connect correctly to the
server.

Where it goes wrong is when we try to upload the file. At this point
the log indicates

534 Policy requires SSL.
Put failed: 534 Policy requires SSL.

Re: Issue with ftp upload using explicit SSL

2009-10-28T12:37:05Z

Sorry, I cannot tell, I don't use this for my server.

But does your ftp address reflect the SSL ? Are you using ftps://....
for the host address?

Issue with ftp upload using explicit SSL

2009-10-28T12:10:23Z

Hello,

we are very new to gsite crawler so please bear with us.

We have to use explicit SSL to upload files to our website.

when we try to upload the test file we get the following error

FTP Connection 28/10/2009 11:54
GSiteCrawler v1.23 rev. 286
------------------------------ ----------

Re: [GSiteCrawler] Re: Recrawl question(s)

2009-10-23T12:15:17Z

Many thanks.

Joe

MOTORHEAD extraordinaire
Professional Storage and Workspace Solutions
79 Park Road - Chelmsford, MA - 01824
Toll Free 800.618.8028 - Direct 978.618.2800 - Fax 978.418.0404
Visit our web site at [link] and
for our latest specials,
<[link]>sign up

Re: [GSiteCrawler] Recrawl question(s)

2009-10-23T04:45:00Z

You can delete all the currently listed urls from URL List (option Delete
all non-manual urls) and start a fresh crawl after that.

----- Original Message -----
To: "SOFTplus GSiteCrawler" <gsitecrawler@googlegroups.com >
Sent: Thursday, October 22, 2009 5:53 PM

Christina
[link]

Recrawl question(s)

2009-10-22T21:53:18Z

I just did a major reorganization of my eCommerce web site and moved
categories and products all around. I also generated a .htaccess file
that did a "Redirect 301 From To" for everything that moved about.

I just kicked off a recrawl with GSiteCrawler and it looks like GSC is
crawling the old URL's frorm what was last in the database. These must

Re: [GSiteCrawler] Re: robots.txt and web site remap

2009-10-20T05:38:31Z

Works great. Thanks a bunch.

Joe

Re: robots.txt and web site remap

2009-10-19T19:58:12Z

Use [link] and put in your url.

On 19 oct, 14:17, Joe Germann <motorheadextraordina...@gmail .com>
wrote:

Re: [GSiteCrawler] Re: robots.txt and web site remap

2009-10-19T18:17:45Z

I implemented the following .htaccess and it appears to work just
fine for users. I can plug in my IP and still get access to my web site.
Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT}
^.*(Googlebot|Googlebot|Mediap artners|Adsbot|Feedfetcher)-?( Google|Image)?

Re: [GSiteCrawler] Re: robots.txt and web site remap

2009-10-19T01:00:41Z

It looks like it; just have to play around a bit to figure it all out.

Joe

Re: [GSiteCrawler] Re: robots.txt and web site remap

2009-10-19T00:55:38Z

Yes, one of those examples should work for you.

----- Original Message -----
From: Joe Germann
To: gsitecrawler@googlegroups.com
Sent: Sunday, October 18, 2009 8:44 PM
Subject: [GSiteCrawler] Re: robots.txt and web site remap

Thanks for the guidance. I am investigating how to properly set us a .htaccess to do this. It looks like it is straight forward. I just have to read up a bit more and set up a test scenario. [link]

Re: [GSiteCrawler] Re: robots.txt and web site remap

2009-10-19T00:44:29Z

Thanks for the guidance. I am investigating how to properly set us a
.htaccess to do this. It looks like it is straight forward. I just
have to read up a bit more and set up a test
scenario.
[link]

Thanks,
Joe

MOTORHEAD extraordinaire