Best way to do page-by-page crawling

[Anton Kheistver]

Hello,

My source site structure looks so;
http://example.com/page/1
http://example.com/page/2
and so on... But some pages may not contain what I'm looking for (images).

What is the most efficient way to crawl such a site with Abot?
For now, I use my own implementation of HyperLinkParser, which queues the next page by incrementing the URL of the current crawled page. Maybe there is a more efficient way? I'm thinking of my own implementation of Scheduler with pre-calculated URLs.

Thanks!

[sjdi...@gmail.com]

Are you asking how to crawl a site that has a predictable url structure but the links are not discoverable by just following links the abot crawls or are you asking how to avoid crawling pages that do not have images?

--
You received this message because you are subscribed to the Google Groups "Abot Web Crawler" group.
To unsubscribe from this group and stop receiving emails from it, send an email to abot-web-crawl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/abot-web-crawler/41146505-3ece-4f50-9c79-1e3bd0c49e5cn%40googlegroups.com.

[Anton Kheistver]

Yes, I'm asking how to crawl a site that has a predictable URL structure. 

Maybe my current approach with custom HyperLinkParser is not the best solution in this case.

суббота, 5 декабря 2020 г. в 05:04:49 UTC+1, sjdirect:

[sjdi...@gmail.com]

Then I do think creating an instance of the Scheduler.cs class, loading it up with the list of precalculated urls is the correct way to go.

To view this discussion on the web visit https://groups.google.com/d/msgid/abot-web-crawler/4e9c0ae2-d461-404a-ac8a-df1034d0a387n%40googlegroups.com.

[CS]

Hi 

Is there sample or reference code that has this feature?

[sjdi...@gmail.com]

Hi, 

You can just create an instance of the Scheduler class, call .Add() with every url you want to crawl then plug that into the constructor. I didn't compile this so im guessing the code would be like...

var scheduler = new Scheduler();

scheduler.Add(new SiteToCrawl("http://yahoo.com/1"));

scheduler.Add(new SiteToCrawl("http://yahoo.com/2"));

scheduler.Add(new SiteToCrawl("http://yahoo.com/3"));

var crawler = new PoliteWebCrawler(
    	null,
	null,
	null, 
	scheduler, 
	null, 
	null, 
	null, 
    	null,
	null);

To view this discussion on the web visit https://groups.google.com/d/msgid/abot-web-crawler/7d017e87-6fa5-407c-b747-81aa2c8054efn%40googlegroups.com.

[CS]

Thank-you!