pm2 restarts

[Wally Grotophorst]

When tailing pm2.log I see this happening every 30 seconds or so (this is just an excerpt, one of the  worker processes out of the 10 I have running):

 

2023-08-17T10:16:20: PM2 log: [PM2][WORKER] Process 6 restarted because it exceeds --max-memory-restart value (current_memory=117948416 max_memory_limit=8388608 [octets])
2023-08-17T10:16:20: PM2 log: App [dspace-ui:6] starting in -cluster mode-
2023-08-17T10:16:20: PM2 log: App [dspace-ui:6] online
2023-08-17T10:16:21: PM2 log: -reload- New worker listening
2023-08-17T10:16:21: PM2 log: Stopping app:dspace-ui id:_old_6
2023-08-17T10:16:21: PM2 log: App name:dspace-ui id:_old_6 disconnected
2023-08-17T10:16:21: PM2 log: App [dspace-ui:_old_6] exited with code [0] via signal [SIGINT]
2023-08-17T10:16:21: PM2 log: pid=82542 msg=process killed

Server has 16GB unused memory.  Here's my dspace-ui.json file:

"apps": [
        {
           "name": "dspace-ui",
           "cwd": "/opt/local/dspace-ui",
           "script": "dist/server/main.js",
           "instances": "max",
           "exec_mode": "cluster",
           "min_uptime": 30000,
           "out_file": "/dev/null",
           "env": {
              "NODE_ENV": "production"
           },
           "node_args": [
              "--max_old_space_size=6144"
           ]

Any suggestions on what might be wrong (I assume that much reloading is wrong...but I am very new to node.js and pm2...

[DSpace Technical Support]

Hi Wally,

Based on the error, it sounds to me like PM2 is automatically restarting anytime a process goes over the " --max-memory-restart" value.  This is a flag that can be used to tell PM2 to automatically restart anytime memory usage increases.  You may want to look more at setting that flag to a reasonable value (it looks like from the error, it might be defaulting to 8MB? "max_memory_limit=8388608")

That's my immediate guess as to what might be going on.  There are also some good guides out there for PM2 usage in general that might be helpful.  Here's one which describes the "--max-memory-restart" flag in more detail:  https://blog.appsignal.com/2022/03/09/a-complete-guide-to-nodejs-process-management-with-pm2.html

Hopefully others will add advice here if they've run into this same issue before.

Tim

[Mark H. Wood]

I'm actually relying on PM2 restarts to help manage seemingly
limitless memory growth. Front end processes were driving us hard
into swap until the OOM killer would harvest Tomcat, then a couple of
minutes later Monit would start it again. The memory limit restart
seemed to be ineffective, so I added an hourly Cron restart that seems
to keep memory demand in check. It's rather crude, though. I think
it isn't waiting for any sort of "quiet point", as I originally
supposed it might.

I'd appreciate any experience with the interplay of the cache settings
and PM2 process memory limits.

Regardless of that, I think that dspace-angular needs some
graceful-shutdown code to close the HTTP listener neatly at the end of
a response. I'm looking into doing this with 'http-terminator'. It
should handle automatic restarts by PM2, manual restarts by the
sysadmin for whatever reason, and normal system shutdown.

--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu