How Is robots.txt supported?

14 views
Skip to first unread message

Nick Gilmour

unread,
Jan 28, 2018, 3:37:15 PM1/28/18
to pyspider-users
Hi all,

As far as I have seen here:

and here:

https://github.com/binux/pyspider/commit/eeed857bb2c6e95e270d4aba3f9af47d577f3c1c

add robots.txt support for fetcher, enable by robots_txt=True


there is a fix for robots.txt support.

But I haven't found anything in the documentation. So, is there or not support and how it works?

Regards



Reply all
Reply to author
Forward
0 new messages