https://www.<domain>.com

18 views
Skip to first unread message

Cloud Web Interface

unread,
Nov 16, 2017, 2:41:49 PM11/16/17
to SOFTplus GSiteCrawler
I am experiencing issues with https://www.captussystems.com. Using http://www.web-site-map.com/ I was able to generate over 1,900 links. But GSiteCrawler has a lot less.

webado

unread,
Nov 16, 2017, 2:57:30 PM11/16/17
to SOFTplus GSiteCrawler

The issue is that you have a lot of urls which 301 redirect to others, so that the crawling chain is broken.


Usin Xenu I only found these urls:

https://www.captussystems.com/
https://www.captussystems.com/cdn-cgi/l/email-protection
https://www.captussystems.com/servicesgroups
https://www.captussystems.com/servicesupport
https://www.captussystems.com/contact
https://www.captussystems.com/pages/about
https://www.captussystems.com/blogs
https://www.captussystems.com/pages/termsandconditions
https://www.captussystems.com/pages/privacypolicy


And these other urls that are 301 redirected - and consequently any urls to be found further are not going to be accessible to a crawler like Xenu and like to GsiteCrawler either.


Ensure your internal navigation is free of any redirections. Use the correct final urls at all time.



Reply all
Reply to author
Forward
0 new messages