brouter.de under heavy load from china robots

[abrensch]

sorry if you encounter increased response times and interrupted requests ("operation killed by thread-priority watchdog after...")

brouter.de is currently under heavy load due to an attack by china robots

I'm not sure if they are calculating something useful, but for me the requests does not make sense. Profiles are just "rail" and "vm-forum-liegerad-schnell"

It started with a sninge IP address, but after I blocked this one it continued with a range of almost 100 IP-Adresses, all from China.

User-agent and referer a faked, so it's definitly not a friendly API-use, and it's hard to filter that out.

I have no idea why someone would do things like that,  because the effort for such camuflage is definitly higher then just running a private instance of the server...

Please let me know if usability is down to an unacceptable level.

thanx,

Arndt

[Jan Bakker]

Try this, it helped me a lot

https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/

Op dinsdag 25 mei 2021 om 20:02:27 UTC+2 schreef abrensch:

[abrensch]

I solved it the brutal way by blocking a whole bunch of IP ranges. It seems o.k. again, except of course for those affected by the blacklist. Please tell me if you are not a china hacker but affected by the blacklist.

There was'nt yet a real "high-load-day" due to all 3 german public holidays in May had rainy wheather. However, compared to last year, load from regular users is up by about a factor of 2,  with May 2nd and May 24 being the top-2 with about 200 concurrent session at the 11AM-peak and about 250.000 Routing-Requests per day. I excpect the server can handle three times more, as long as robot requests do not put additional load on it.