DSpace 7.x Site continually crashing due to bot attacks

[Carolyn Sullivan]

Hello,

We just launched our DSpace 7.x site yesterday.  It's been down for most of the time since the launch because we've had so many bots on the network.  Has anyone else experienced this issue and what did you do in response?

Thanks,

Carolyn.

[Abel Gómez]

Hello Carolyn,

Which part is crashing? The front end?

I observed severe performance issues with the DSpace Angular front end caused by crawlers and bots (not necessarily malicious). I have the DSpace Angular frontend running behind an nginx reverse proxy, and observed that a single request by a bot to our public IP was causing tenths of requests from the AngularJS application to the nginx proxy (i.e., to the public IP again). That, in the end, caused a huge RAM and CPU consumption of the Angular instances (the proxy and the DSpace backend were running fine), and finally, caused the Angular frontend to stop responding do to resource exhaust.

Since those tenths of requests were coming from the Angular frontend, I suspected it had to be related to the server-side caching features, but I was unable to find the root cause. I played with the DSpace caching configuration with no luck.

The DSpace frontend and backend run in the same VM with 16GB of RAM, with 4 cores. The Angular frontend runs 3 instances in cluster mode. Our repository is not especially big, and those specs are sensibly higher than the requirements listed in https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace for small sites. But it seems it wasn't enough.

I finally gave up, and I simply imposed strict limits to requests coming from bots in the nginx proxy (1-2 requests per minute). That's causing quite a few indexing errors in search engines (which are registering results showing a "Service unavailable" for some items in our site) but at least the site is not crashing.

So... are you experiencing the same problem? Could anybody provide any information so more requests from bots can be allowed, and thus the site can be indexed properly?

Best regards,

Abel