Memory decrease causes CPU spike

[Mike Roberts]

I've been having this problem with my production servers, haven't been able to replicate it with any other environment and it has be scratching my head daily.

About every hour or so, any and all requests start taking up 100% cpu and response times go from 0.5 to around 2 seconds. This happens for anywhere between a few minutes up to a half an hour. Then it goes back to normal. My monitoring script requests one page (the same page) per minute, and this issue can start without any traffic to the server, except for the monitor that runs the same script over and over. I've even changed the script the monitor checks and it's the same problem.

I've found that the node process drops in memory dramatically. It will go from 400mb to 100mb and this is when the problem starts. No erros shows up in any of the logs. Again, it happens seemly randomly with nothing triggering an event, at least I haven't found a trigger. It's getting fairly annoying because it's only happening in production. I've only been able to manage the symptoms using clustering, so that only one of the processes has the issue at a time.

I'm running 0.12.7 with express and mongoose.

[Jimb Esser]

If it's physical memory usage (RSS) dropping from 400mb to 100mb when things are going slow, but other memory sizes staying the same (especially if the V8 heap size is still > 100mb), that sounds like maybe your system might be out of total physical memory and a bunch of that process's memory got paged out to slower (swap) memory (although, depending on OS, that may not show up as 100% cpu usage).

[Mike Roberts]

I'm not running out of physical memory. It is the node heap size that's dropping from 400mb to 100mb.