MG-DBX memory usage in Node.js

[wdbacker]

Hi,

I discovered something strange in Node.js running QEWD.js this week: a Node.js worker was using all available internal memory on our production server: one worker had 21 GB (!) of the available 32 GB of RAM in use (I found this out in Windows task manager using the memory (private working set) column).

As soon as I gracefully stop the worker in question using QEWD Monitor, server memory usage returns to normal.

My first thought was a memory leak somewhere in my JS code. I looked further and noticed memory usage in the worker consistently goes up very fast when I issue a REST request that just returns a set of 24 products with all details from our Caché database (json data size returned = 500 KB). A single REST call/request causes a 45 MB increase in memory usage in task manager. That explains why available server memory eats up so fast.

Today I did more testing to find what's causing this. I created a testrest.js module in QEWD.js with two handlers: the first handler returns a json response with the static data result from the same 24 products request (I saved the product's json response in a test.json file and the handler simply require()'s it to return it to the browser). The second handler does a mg-dbx function call to Caché to return the product's json dynamically from the Caché database using an extrinsic function. The result is that the first handler with the static data returned does not increase memory consumption, but the second handler returning the data from Caché increases memory usage by 45 MB. As I'm using QEWD.js's workers with an mg-dbx in-process connection to our Caché, this means the memory increase must come from mg-dbx or Caché's memory usage itself. In Node.js in particular, I see the big increase mostly in the RSS (resident set size) part when you log a process.memoryUsage() call on the console in the QEWD.js handler.

It's also important to note that I made the extrinsic function called using dbx.function() now very trivial for testing: it just merges the json data to a temp global and this temp global is then returned in JS as the json response using ewd-document-store's getDocument() method. The ObjectScript code executed doesn't seem to matter, every call causes a memory increase by 45 MB in the Node.js worker and memory is not released after the call.

One major difference to note between the two handlers is that the first handler simply require()'s a static json file while the second handler does the function call to Caché and possibly possibly allocates a lot of DocumentNode instances inside ewd-document-store's getDocument() method.

As a temporary workaround, I solve this problem by restarting all workers every day.

Btw, Googling on this kind of issues, this post came to my attention: https://lightrun.com/answers/facebook-jest-bug-memory-consumption-issues-on-node-js-16110 - more specifically the --no-compilation-cache comment and https://github.com/nodejs/node/issues/40014

@Chris: do you have any clue what could be causing this memory usage pattern?

Ward

[wdbacker]

Hi,

To overcome the problem of workers allocating too much memory (when a worker keeps running for a longer time), I've written & tested an automatic memory management function you can use in your qewd.js startup code (add this code inside your own addMiddleware function):

config.addMiddleware = function(bodyParser, app, q, qx, qewdConfig) {

  // your own addMiddleware code ....

  // when QEWD.js is started, install memory limit monitoring for workers 
  q.on('started', function() {
    var checkWorkerMemoryDelay = q.userDefined.checkWorkerMemoryDelay || 300000; // default every 5 minutes
    var memoryLimitMonitor = null
    var workerMemoryLimit = q.userDefined.workerMemoryLimit || 1024; // default 1 GB max. per worker
    var monitorMemoryLimit = function() {
      // every 5 minutes, check current memory consumption of all workers
      q.handleStats(function(messageObj) {
        var checkWorkerMemoryLimit = function(workerStat) {
          // if a worker exceeds our given memory limit, send it a stopWorker message to shut it down gracefully and release it's memory
          if (+workerStat.memory.rss > workerMemoryLimit) {
            console.log(`Worker ${workerStat.pid} reached rss memory limit (${workerStat.memory.rss} MB > ${workerMemoryLimit} MB), stopping worker ...`)
            q.stopWorker(workerStat.pid)
          }
        }
        let workerStats = messageObj.worker
        if (workerStats) {
          workerStats.forEach(checkWorkerMemoryLimit)
        }
      });
    }
    // now install the memory limit monitor interval timer
    memoryLimitMonitor = setInterval(monitorMemoryLimit, checkWorkerMemoryDelay)
    console.log(`Memory limit monitoring for workers has started on master ${process.pid} (rss limit = ${workerMemoryLimit})`)
    // when QEWD.js is stopped, clear the memory limit monitor interval timer
    q.on('stop', function() {
      if (memoryLimitMonitor) {
        clearInterval(memoryLimitMonitor)
        console.log(`Memory limit monitoring for workers has stopped on master ${process.pid}`)
      }
    });
  })
};

​

HTH,

Ward

[chris....@gmail.com]

It does sound like there’s some kind of fault here.  I’m not aware of any memory leaks in mg-dbx but I’ll check this – specifically the part that invokes Cache functions – before the next release.

 

One notorious area where the V8 engine can consume an excess amount of memory is related to the garbage collector.  The V8 engine tends to be a little slow in cleaning up objects that it identifies as being out of scope.  A consequence of this is that JS scripts that quickly generate lots of JS objects can consume a lot of memory in a short period of time.  However, the garbage collector should (and usually does) spring into action before memory usage becomes critical.  Most of the time it simply appears that Node.js/V8 is a bit greedy.

 

With this ‘issue’ in mind all object classes managed by mg-dbx (global, class and cursor) all have Close and Reset methods.  The Reset method is particularly useful as it allows the same JS container to be reused – and, as such, avoids V8 allocating further memory to hold the (new) Cache object.

 

I’ll carry out some further investigation before the next update.

 

Chris.

--
You received this message because you are subscribed to the Google Groups "Enterprise Web Developer Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-web-develope...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/enterprise-web-developer-community/42422a22-7ca8-4850-a6d4-3e97e4e70a37n%40googlegroups.com.