How Is robots.txt supported?

15 views

Skip to first unread message

unread,

Jan 28, 2018, 3:37:15 PM1/28/18

to pyspider-users

Hi all,

As far as I have seen here:

and here:

add robots.txt support for fetcher, enable by robots_txt=True

there is a fix for robots.txt support.

But I haven't found anything in the documentation. So, is there or not support and how it works?