Annoying Bots

[Jim Breen]

The cloud service being used for the edrdg.org server (Digital Ocean)
doesn't charge for bandwidth and since the CPU load is light I'm not
too fussed about things, but glancing at my daily stats report I can't
help noticing:

13838 GET /jmwsgi/edform.py?svc
10751 GET /jmwsgi/entr.py?svc
...

That 24k hits on the database yesterday was down on the 40k+ the day
before, but it's still a lot. Looking at where they came from, I see:

2214 87.121.105.217
194 173.252.83.4
178 173.252.83.118
174 173.252.83.23
166 173.252.83.13
166 173.252.83.117
165 173.252.83.1
[plus many more from 173.252.83.*]

The first one has no reverse DNS and I have sin-binned it. The others
are more interesting. They come from a farm at "fbsv.net". AFAICT that
is part of Facebook and they scrape pages that have been mentioned in
FB posts. They also appear not to honour robots.txt directives.

Probably nothing much can be done about it, but annoying all the same.

Jim

--
Jim Breen
Adjunct Snr Research Fellow, Japanese Studies Centre, Monash University
http://www.jimbreen.org/
http://nihongo.monash.edu/