3.2.10 using significantly more RAM than CacheSizeGB

[Nick Judson]

This is occurring on a remote windows box with a cache size set to 1GB, but where it appears to be using >3 GB. I have the metrics files but I don't really want to post them to the MongoDB JIRA since this is an older version. What is the best way to diagnose the memory usage? Thanks!

[Kevin Adistambha]

Hi Nick

Could you elaborate on how you determine that MongoDB is using >3 GB? Note that by “cache size” I’m assuming you meant the WiredTiger cache size setting.

Please note that WiredTiger cache size setting governs only the maximum cache size of WiredTiger, and not the overall MongoDB memory use. Other processes such as client connection, aggregation framework, queries, etc. uses memory on top of WiredTiger cache usage. Therefore, a busy server could use significantly more memory on top of the WiredTiger cache size setting.

Having said that, as of MongoDB 3.6.2, there is currently no built-in method to limit MongoDB memory usage, since MongoDB was designed to utilize as much memory as possible to give you the best performance.

For diagnosing memory usage, you may be able to use Memory Diagnostics for the WiredTiger Storage Engine as a starting point.

Best regards
Kevin

​

[Nick Judson]

Hi Kevin - thanks for replying.

You are correct that I am referring to the --wiredTigerCacheSizeGB (set to 1). I was sent a screenshot of task manager showing the mongo db process using >3GB of memory, as well as the metrics files (which I cannot open). I am aware of all of the settings and potential overages for MongoDB/WT, and it is entirely possible that there is some edge-case workload going on here which is causing MongoDB to bump up to 3x the wiredTigerCacheSizeGB setting. This does seem unlikely though as I have never encountered it before (in many other installations). In the past I would have posted this on the MongoDB JIRA page and the contents of the metrics file would be exploded into those nice graphs which show exactly what is occurring. I suppose I'm asking if it's possible for someone to do the same via this forum? I could PM the metrics files and then someone internal to MongoDB could post the resulting graph/prognosis.

Thanks,

Nick.

[Anthony Maire]

Hi Kevin and Nick

I'm currently experiencing the same kind of issue on linux with mongo 3.4.1

We use mongoDB for our internal monitoring system and we are currently doing some experimentation about redesigning the schema to improve our performance.

So I have 2 servers :

- The development server (mongo 3.6.1) which runs perfectly : it hosts a dump of the production database (16GB), a copy of this dump using our new schema (6GB) and a stress test database (300GB). It is started with a WT cache size of 2GB, and even when doing massive insertions on the stress test DB (on 50 connections), the resident set size (as reported by the "top" command) stay only slightly above the cache size, somewhere between 2.3 and 2.5GB. This server runs on CentOS 6, without any virtualization/containerization

- The production server (mongo 3.4.1) which experiences excessive memory consumption and crashes : it host the production database. It is running in an OpenVZ container (on top of CentOS 6, without cap on privvmpages so there shouldn't be any issue like the one that occured with older versions of OpenVZ). The whole container has 5GB of RAM available, and WT cache size is 2GB too (confirmed by executing db.serverStatus().wiredTiger.cache on the prod environment) . However, we always have a bigger resident set side for the mongod process, often greater than 4GB. This lead to have the mongo instance performs really poorly (since swap start to be used) and eventually the kernel OOM killer kicks in when the container dedicated RAM is full. This occurs several times per week

For the access patterns : our applications periodically push some new documents in the DB, but there are very few query/aggregations running, since they are very few (< 5) users that are getting data from the DB, and they only do it occasionally, most of the time there is no query at all.

So I would like to know :

- Is there any major memory leak that have been fixed between 3.4 and 3.6 that can explain this ?

- How can I investigate on the cause of this memory consumption ?

Best Regards,

Anthony

[Kevin Adistambha]

Hi Nick

Before digging into diagnostics, could you specify the environment you’re running? Such as your deployment topology (standalone, replica set, etc.), your OS version, and your hardware specification?

I was sent a screenshot of task manager showing the mongo db process using >3GB of memory

Just so we’re on the same page, are you seeing any performance issues due to this memory usage, or is this a one-time event that you’re curious about? Also, does the memory usage stabilize at some value after a while?

it is entirely possible that there is some edge-case workload going on here which is causing MongoDB to bump up to 3x the wiredTigerCacheSizeGB setting

If finding an edge-case workload is the goal, it’s generally more precise to identify the workload in question by looking at mongod logs. Things to look for could include:

• Are there any slow queries, or anything that could point to what the server was doing during the time of high memory usage? Using mloginfo from the mtools package could show you the slow queries during this period.
• Note that large spike in memory usage could be caused by incoming connections into MongoDB. Each connection into MongoDB would take about 1 MB of RAM, so if you have 1000 active connections, mongod would allocate an extra 1 GB of RAM to handle them.
• How does the memory usage change over time? Do you see this sudden spike once, or multiple times? Is there any pattern that you can observe of high memory usage vs. the server’s uptime?

Also since you’re using 3.2.10, could you try to upgrade to the latest in the 3.2 series (currently 3.2.18) and see if the issue persists?

Best regards
Kevin

​

[Kevin Adistambha]

Hi Anthony

I’m currently experiencing the same kind of issue on linux with mongo 3.4.1

Although the issue looks superficially similar, chances are it’s a different issue.

Could you please open a new thread with your issue so that we’re not talking about two different things in this thread?

Best regards
Kevin

​

[Nick Judson]

Hi Anthony,

This issue was sent to my by a customer - I do not have access to the machine itself. This is a standalone mongod (no replset) running on w2k12r2. No idea on the hardware...could well be a VM. From what I can gather, the memory increases slowly over the period of one week from 1GB to 3.5GB. I do not know if it stabilizes as the customer kills the process regularly. I also cannot upgrade the customer since this version of MongoDB is tied to the version of the product they have installed.

This particular workload has been a challenge for MongoDB in the past (it produced the bugs for the bug hunt runner up for 3.0, 3.2, and 3.4) and I've found the best way to accurately determine what is going on is to provide MongoDB with the metrics files. In the past, symptoms like this have proved to be highly technical and not something that is easily diagnosed externally. I had a look through the JIRA list and didn't see anything which might be causing this (although please correct me if you're aware of something). I thought that diagnosing issues such as this was exactly what the metric data was for?

I have manually looked at the log files and there are a few slower queries (perhaps 5 seconds) as well as some very large queries (large 'in' array) but I would expect those to require temporary spikes in memory only.

Cheers,

Nick.

[Anthony Maire]

Hi Kevin

Thank you for your answer.

I'm going to open another thread (and I have some further information to add about my issue)

Regards,

Anthony