On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu <
fuge...@gmail.com> wrote:
> I see bots crawling devise pages in my log:
> Should I disallow those paths in robots.txt ?
1) why?
2) do your logs show these bots *reading* robots.txt?
--
Hassan Schroeder ------------------------
hassan.s...@gmail.com
twitter: @hassan
Consulting Availability : Silicon Valley or remote