how to create a crawler?

35 views
Skip to first unread message

Davood Hadiannejad

unread,
Jul 4, 2022, 6:06:49 AMJul 4
to Common Crawl
Is there any way to create script which crawls web sites for a specific word like "ebike" 

Sebastian Nagel

unread,
Jul 7, 2022, 11:21:29 AMJul 7
to common...@googlegroups.com
Hi Davood,

you may want to search for tools to build a "focused crawler" [1]:
- it's more about concepts, not words
- usually, the crawl will include pages which do not contain the
keywords itself but link to content about your topic

Best,
Sebastian

[1] https://en.wikipedia.org/wiki/Focused_crawler
Reply all
Reply to author
Forward
0 new messages