Bots issue consuming memory on DSpace 8.0 site

[Lewatle Johannes Phaladi]

Hello DSpace Tech, 

There is bots issue on DSpace 8.0 site, usually it happen when opened access to overseas countries that can be US, China and other countries this is causing error 504 to come out, when checking tomcat log file complaint is about memory leak, please assist by suggesting any steps to follow.

Regards,

Lewatle 

[DSpace Technical Support]

Hi Lewatle,

I'd recommend starting with our Performance Tuning Guidelines, especially those regarding "Limit which pages are processed by Server Side Rendering (SSR)" and possibly also the "Turn on caching of Server Side Rendered pages".  When a large number of bots all hit SSR, they may overwhelm both the frontend and backend... so, limiting SSR or caching it can sometimes help.  However, keep in mind, some of these settings are only available in 8.2 or later.  So, they won't help if you are running 8.0...you would need to upgrade to take advantage of them.

https://wiki.lyrasis.org/display/DSDOC8x/Performance+Tuning+DSpace

We do also have a ticket where we've started some discussions about *managing aggressive bots* (especially AI-related bots).  https://github.com/DSpace/dspace-angular/issues/4565   If these bots are overly aggressive, then it's possible the hints in our "Performance Tuning Guidelines" may not make a large impact (it may only slow down the bots slightly).  These overly aggressive bots are not something that can fully solved in DSpace code itself...but, we have a lot of external sources listed in that ticket, along with some hints from other DSpace developers on what has worked for them.

Hopefully that can at least get you started.

Tim

[Lewatle Johannes Phaladi]

Hello Tim,

Thanks very much for sharing this crucial information and definitely I will start with the upgrade to the latest DSpace 9 as you have mentioned even 8.2 accommodates remedies for the bots.

Regards,

Lewatle 

--
All messages to this mailing list should adhere to the Code of Conduct: https://lyrasis.org/code-of-conduct/
---
You received this message because you are subscribed to a topic in the Google Groups "DSpace Technical Support" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dspace-tech/YuayK2xTXzU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dspace-tech...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dspace-tech/2f095135-0618-47ff-9493-c02705ab5657n%40googlegroups.com.