adding pages to the URL list

2 views
Skip to first unread message

darren...@gmail.com

unread,
Feb 16, 2009, 6:33:29 AM2/16/09
to SOFTplus GSiteCrawler
Hi there,

New to this software but here is my problem.
I've run through the wizard and it's generated me a sitemap file that
has bout 23 urls in it. The actual website has over 120 urls. If I
click on "Import" then "Files from local directory" I can import url's
one at a time from the website folder. I was just wondering if adding
one at a time is the only way to do this as it's going to take me a
long time O_o

Thanks Darren

webado

unread,
Feb 16, 2009, 8:08:52 AM2/16/09
to SOFTplus GSiteCrawler
I don't know how else that can be done.

But I think you had better figure out why it cannot find all the urls
on your site. Google and other search engine robots will be unable to
discover your urls through crawling either, and that's much more
important than the sitemap. The sitemap can help somewhat, but with no
incoming links to them, such urls are orphans and will at best end up
in the omitted list.

Perhaps you are switching back and forth between www and non-www urls
in your navigation.

Perhaps you have some javascript or flash based navigation with no
alternative html navigation.

Perhaps you have disallowed certain pages in robots.txt which cuts off
the access to other pages.

Perhaps you have subdomains or folders with their own entry points (so
they are like separate websites) that need to be crawled separately.



On Feb 16, 6:33 am, "darrenma1...@gmail.com" <darrenma1...@gmail.com>
wrote:

darren...@gmail.com

unread,
Feb 16, 2009, 8:18:10 AM2/16/09
to SOFTplus GSiteCrawler
Thanks for the reply.
I closed my project and followed the tutorial on the website, I used
the wizard. It didn't find all my url's. Then I clicked import from
local folder and selected the first .asp page and it loaded all the
urls in the folder. This is great. I then recrawled and checked for
existence and everything is working. Thanks!!!!
Reply all
Reply to author
Forward
0 new messages