Skip to first unread message
ASIA IO
Jan 21, 2015, 11:49:18 PM1/21/15
to google-a...@googlegroups.com
This my site https://timeforums.appspot.com/forum.php on GAE
How to stop the AhrefsBot bot crawl ?I set in robots.txt ,
user-agent: AhrefsBot
disallow: /
already ,and block ip in dos.yaml
blacklist:
- subnet:
188.165.15.238
but , not work .still crawl ,
How can i do ? Anyone have any other suggestions or not.
Thank you.
Anthony Shapley
Jan 22, 2015, 11:43:47 AM1/22/15
to google-a...@googlegroups.com
Any chance its not a "real" AhrefsBot and someone else isn't just using there User Agent?
Ant
Farid Nizal
Jun 27, 2015, 11:49:27 AM6/27/15
to google-a...@googlegroups.com
please provide a complete list of ip ahrefsbot .Reply all
Reply to author
Forward
0 new messages