Ignore Robots.txt

9 views

Skip to first unread message

unread,

Jul 11, 2008, 12:13:41 PM7/11/08

to hounder

Hi, i need to crawl google search results. Is there a way to ignore
the robots.txt?

thanks in advance

Nicolas Bottarini

unread,

Jul 11, 2008, 2:24:49 PM7/11/08

to hounder

Hi Nico!
You can change Nutch's fetcher code to ignore the robots.txt files,
but you shoudn't.
Regards,
-- Jorge