disallow devise paths in robots.txt ?
16 views
Skip to first unread message
fugee ohu
Oct 6, 2018, 2:04:05 AM10/6/18
to Ruby on Rails: Talk
I see bots crawling devise pages in my log:
141.8.143.179 - - [06/Oct/2018:00:35:45 -0400] "GET /users/sign_in HTTP/1.1" 200 4022 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
Should I disallow those paths in robots.txt ?
Hassan Schroeder
Oct 8, 2018, 12:18:27 AM10/8/18
to rubyonrails-talk
On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu <fuge...@gmail.com> wrote:
> I see bots crawling devise pages in my log:
> I see bots crawling devise pages in my log:
> Should I disallow those paths in robots.txt ?
1) why?
2) do your logs show these bots *reading* robots.txt?
--
Hassan Schroeder ------------------------ hassan.s...@gmail.com
twitter: @hassan
Consulting Availability : Silicon Valley or remote
fugee ohu
Oct 8, 2018, 7:04:42 AM10/8/18
to Ruby on Rails: Talk
Yes
Reply all
Reply to author
Forward
0 new messages