How abot decide whether to crawl this page or not?

31 views
Skip to first unread message

agarwal....@gmail.com

unread,
Jun 11, 2020, 1:58:59 PM6/11/20
to Abot Web Crawler
I am working on project and using abot crawler the website. suppose if i have url like "www.abcd.com/xyz" . This page contains thousand of links. I just to crawl the link which conatin "www.abcd.com/xyz" as prefix and dont want to crawl the other link. so which function should crawling to determine whether to crawl this page or not.

steven...@carvana.com

unread,
Jun 11, 2020, 2:10:44 PM6/11/20
to Abot Web Crawler
Reply all
Reply to author
Forward
0 new messages