[IMP] Multiple high memory usage alerts, MongoDB using 75% of memory for small data size

[Astro]

Hi All,

We are using MongoDB 3.2.6 with WiredTiger storageEngine. Below is the Replica set Configuration:

Replica set Configuration:

Total members: 3

Roles: Primary, Secondary, Secondary

Memory Allocated to each server: 32GB

Relative data size: 3-4GB

Issue Summary:

Memory usage of one of the secondary members has slowly grown and reached up to 75%. This triggers continuous alerts from the monitoring system used to monitor the replica set deployment. 

The memory usage keeps growing gradually hence the alerts are triggered, We expect mongodb to release memory when the usage is reduced. 

This document on WiredTigers memory usage outlines the memory usage percentage by WiredTiger cache and File system cache. Adhering to those estimations for the aforementioned deployments it should use:

WiredTiger internal cache memory usage: 60% of RAM minus 1 GB = 16.4GB

Below are some concerns we need to get addressed:

1. Will the mongod process go out of memory if the memory usage keeps growing gradually? 

2. Why does it use 75% of memory for relatively small size of data?

3. Why doesn't wiredTiger releases memory when other operations are being performed?

4. What is the best solution or workaround to mitigate the high memory usage?

Below are the server specific stats:

Memory usage:

PID      USER       PR    NI    VIRT    RES     SHR S  %CPU   %MEM    TIME+ COMMAND

24371  mongodb   20   0 25.490g 0.022t      7576 S      3.6         75.5      4520:27 mongod

mongostat:

% dirty % used flushes vsize   res

0.0        73.6     0          25.5G  22.6G

Any help will be appreciated.

Thanks in advance.

[Kevin Adistambha]

Hi,

Memory usage of one of the secondary members has slowly grown and reached up to 75%. This triggers continuous alerts from the monitoring system used to monitor the replica set deployment.

Could you post more details about your deployment:

• Could you post the output of db.serverCmdLineOpts() of all three nodes.
• Could you post the output of rs.printSlaveReplicationInfo()
• Does this happen only on a single Secondary, or to all nodes in the replica set?
• Does this happen all the time, and this situation persists even after rebooting the node?
• Are you doing Secondary read/aggregation using that Secondary?
• Is there anything in the logs that could indicate something else besides regular replication activity is going on in that node?

Will the mongod process go out of memory if the memory usage keeps growing gradually?

It is a possibility, however unless your provision a server with very small amount of memory (e.g. 1 GB or less), that should not happen. Typically, this situation manifests in the MongoDB process getting killed by Linux OOM killer.

Why does it use 75% of memory for relatively small size of data?

Secondary reads, running a large aggregation, or index building are some of the possible explanation, however it’s impossible to tell what’s going on in your server unless you post more detailed observation and data. The set of questions above should provide a starting point.

Why doesn’t wiredTiger releases memory when other operations are being performed?

There are two types of cache used by MongoDB using the WiredTiger storage engine:

• The WiredTiger cache (default size is 60% of RAM minus 1 GB, this is configured )
• The O/S filesystem cache, which is controlled by the O/S.

By default, WiredTiger will try to keep its cache usage to 80%, and the output of mongostat you posted seems to indicate that it’s at 73.6%, well below the 80% ceiling.

What is the best solution or workaround to mitigate the high memory usage?

MongoDB was designed to use the most amount of memory that is available to ensure it gives the best performance with regard to your hardware. Therefore, currently there is no built-in method to limit MongoDB memory usage.

Also, I would recommend you to upgrade to the latest version in the 3.2 series, which is currently 3.2.10. MongoDB 3.2.10 contains many improvements regarding the WiredTiger cache.

Best regards,
Kevin

​

[Astro]

Hi Kevin,

While upgraded to 3.2.10 , we're still seeing the high memory usage on secondary instances.  The memory keeps growing gradually. 

The member eventually runs out of memory.

The data size is relatively small ~8GB, the memory allocated to each node in replica set is 64GB.

Please advice.

Thanks,