ERDDAP memory usage slowly growing over time?

[Benjamin Adams]

Hi, has anyone seen ERDDAP memory usage slowly growing over time?  We've been collecting metrics on an ERDDAP process's memory usage on the Glider DAC server and have seen memory usage slowly growing over time.  We have been updating the Glider DAC so have restarted the ERDDAP process 3 days ago.  By about next week I should have a line graph with a better trend line to demonstrate what I believe is happening, namely that memory allocations are growing over time.  We're currently running ERDDAP 2.23 with max heap size of 10 GiB, and 16 vCPUs and are mostly running aggregations of 100+ NetCDF files.  I would approximate we have around a terabyte of data files or slightly less.  We're not seeing conditions like excessive load averages on the server which could lead to memory pressure and suspect there might be a memory leak.  Has anyone experienced similar issues with ERDDAP?

[Roy Mendelssohn - NOAA Federal]

Yes we are seeing the same thing and are investigating the cause - one thing is we only see it in ERDDAPs that get heavy use, our other ERDDAP have no problems. Scans that are either rmultithreading or coming from multiple IPs at once, particularly if they make a lot of large or very large request repeatedly, tend to exhibit the same behavior. My working hunch is that something is not getting garbage collected properly, such as perhaps if there is not enough memory for young generation formation those don't get cleared, but don't quote me on that. I have been playing with Tomcat settings also to see if that helps. I would be curious what happens if you increase the max heap size by fair amount, if you are willing to try. Right now I think I have our main one at 15GB.

tl,dr - yes we are aware of it, no we don't see it on all ERDDAPs, no we don't have a fix at the moment.

-Roy

> On Apr 23, 2024, at 9:05 AM, 'Benjamin Adams' via ERDDAP <erd...@googlegroups.com> wrote:
>
> Hi, has anyone seen ERDDAP memory usage slowly growing over time? We've been collecting metrics on an ERDDAP process's memory usage on the Glider DAC server and have seen memory usage slowly growing over time. We have been updating the Glider DAC so have restarted the ERDDAP process 3 days ago. By about next week I should have a line graph with a better trend line to demonstrate what I believe is happening, namely that memory allocations are growing over time. We're currently running ERDDAP 2.23 with max heap size of 10 GiB, and 16 vCPUs and are mostly running aggregations of 100+ NetCDF files. I would approximate we have around a terabyte of data files or slightly less. We're not seeing conditions like excessive load averages on the server which could lead to memory pressure and suspect there might be a memory leak. Has anyone experienced similar issues with ERDDAP?
>

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/5e243409-0111-4860-bf0d-63d13154348fn%40googlegroups.com.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new street address***
110 McAllister Way
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Men...@noaa.gov www: https://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.