External links

13 views
Skip to first unread message

bjorn.pa...@gmail.com

unread,
Jul 6, 2015, 9:17:21 AM7/6/15
to scrapy...@googlegroups.com
is there any setting for following external links, and crawl outside the base url?

I.e. on example.com, you find a link to example1.com, and it continues to crawl example1.com and find the link example3.com and so on. 






Nickko G

unread,
Aug 18, 2015, 7:38:24 AM8/18/15
to scrapy-users
Le lundi 6 juillet 2015 15:17:21 UTC+2, bjorn.pa...@gmail.com a écrit :
is there any setting for following external links, and crawl outside the base url?

I.e. on example.com, you find a link to example1.com, and it continues to crawl example1.com and find the link example3.com and so on. 


I'm not sure, but perhaps it's ok if you omit allowed_domain parameter.

Nickko 
Reply all
Reply to author
Forward
0 new messages