Memory Spikes / Server Crashes

[Liam Lloyd-Tucker]

Hi folks!

I'm running archivematica on an AWS EC2 instance (8 cores, 16 GB memory), and about once a day the memory usage on my instance spikes above 70% and the instance becomes unreachable. I'm still digging into what exactly is causing this problem, but I'm wondering if anyone has seen something similar and might be able to point me in a fruitful direction. I'll note that I am running multiple copies of the MCPClient (I've seen the issue with between 2 and 4 copies running).

Best,
Liam

[Joseph Anderson]

We've experienced this a few times, once due to processing large video files, but in our case boosting our memory to 16gb fixed it (so probably not your issue). Another time was related to Elastic Search which we eventually disabled because we weren't really using any search functionality in AM. And the last time it was related to JHOVE getting caught in a loop when processing various TIFF files, and the solution to that ended up being that the default java heap for JHOVE was set too low by default: https://github.com/archivematica/Issues/issues/1683#issuecomment-2148066040

[Liam Lloyd-Tucker]

In my case the problem seems to have been caused by Prometheus (perhaps specifically during video processing). We weren't using Prometheus, so I turned it off and haven't seen any crashes since.

[Panna Georgina Liptak]

Hi Liam,

May I ask how did you disable Prometheus? 

We had some issues with it, tried to turn it off with some configs but for me it seems that it's not possible to avoid the directory initialization (see code) and it's taking up quite some space. Just curious if you managed to completely disable it.

Thanks,

Panna

[Liam Lloyd-Tucker]

Hi Panna,

I unset all of ARCHIVEMATICA_MCPSERVER_MCPSERVER_PROMETHEUS_BIND_PORT, ARCHIVEMATICA_MCPSERVER_MCPSERVER_PROMETHEUS_BIND_ADDRESS, ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_PROMETHEUS_BIND_PORT, ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_PROMETHEUS_BIND_ADDRESS, ARCHIVEMATICA_DASHBOARD_DASHBOARD_PROMETHEUS_ENABLED, and SS_PROMETHEUS_ENABLED. This didn't stop Archivematica from creating a `prometheus_stats` directory for me either though.

Best,
Liam

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/archivematica/3682d274-0607-45e6-abe6-8502e33f78d3n%40googlegroups.com.