MongoDB 3.2.6 Crashes with error "Invariant failure: seekRet resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp"

[tanveer madan]

Hi,

I am using MongoDB 3.2.6 on a server on a VM 4CPU and 64GB RAM

I have replica set with 2 data-bearing nodes and 1 arbiter

I am running YCSB tool to load 12M records and both the secondary and primary nodes are crashing with below error

The Database is crashing with below error

Fri Jul 15 15:49:24.293 E STORAGE  [conn30] WiredTiger (12) [1468622964:293830][2854:0x7fd90f9f1700], file:ycsbc/collection/3-8181378381775311216.wt, WT_CURSOR.search: memory allocation of 323400 bytes failed: Cannot allocate memory

Fri Jul 15 15:49:24.296 I -        [conn30] Invariant failure: seekRet resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 527

Fri Jul 15 15:49:24.332 E STORAGE  [conn32] WiredTiger (12) [1468622964:332357][2854:0x7fd90f7ef700], file:ycsbc/collection/3-8181378381775311216.wt, WT_CURSOR.search: memory allocation of 323395 bytes failed: Cannot allocate memory

Fri Jul 15 15:49:24.332 I -        [conn32] Invariant failure: seekRet resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 527

Fri Jul 15 15:49:24.344 I CONTROL  [conn30]

 0x1383c72 0x1321ae8 0x130e9cd 0x10fe667 0xc45b89 0xc0dd0a 0xc42a54 0xe99ef5 0xe9a5b9 0xe9a6b5 0xbbbb59 0xbbdc85 0xbbe04d 0xbc1178 0xbd9ac3 0xbda934 0xb36980 0xce5ca5 0xce8536 0x9ca55c 0x13311cd 0x7fd93a50ddf5 0x7fd93a23b1ad

----- BEGIN BACKTRACE -----

My config file is as below

auditLog:

  destination: file

  format: BSON

  path: /mongo/log/nfs

net:

  bindIp: poc08

  port: 27030

operationProfiling:

  mode: slowOp

  slowOpThresholdMs: 100

processManagement:

  fork: "true"

  pidFilePath: /mongo/log/nfs.pid

replication:

  oplogSizeMB: 51200

  replSetName: nfs

setParameter:

  enableTestCommands: "1"

  logLevel: "0"

storage:

  dbPath: /mongo/data/nfs

  directoryPerDB: true

  engine: wiredTiger

  journal:

    enabled: false

  wiredTiger:

    collectionConfig:

      blockCompressor: zlib

    engineConfig:

      directoryForIndexes: true

      journalCompressor: none

systemLog:

  destination: file

  logAppend: true

  logRotate: rename

  path: /mongo/log/nfs.log

  timeStampFormat: ctime

  verbosity: 0

Thanks,

Tanveer

[Tim Hawkins]

it looks like your are using NFS to connect your storage, it would be worth reading https://groups.google.com/forum/m/#!topic/mongodb-user/Kd85b2HHVn8 which advises against it. Runing database systems over nfs is not a relieable solution.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b445f460-ff87-4369-8e7b-067d0f499c72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tanveer madan]

Hi Tim,

Thanks for the reply !!!

The naming convention is nfs, but I am actually running on iSCSI storage

Thanks,

Tanveer

[Kevin Adistambha]

Hi,

Fri Jul 15 15:49:24.293 E STORAGE [conn30] WiredTiger (12) [1468622964:293830][2854:0x7fd90f9f1700], file:ycsbc/collection/3-8181378381775311216.wt, WT_CURSOR.search: memory allocation of 323400 bytes failed: Cannot allocate memory

This error means that MongoDB tried to allocate memory, but failed. The main reason is that there is not enough free memory in your system.

I have replica set with 2 data-bearing nodes and 1 arbiter
I am running YCSB tool to load 12M records and both the secondary and primary nodes are crashing with below error

Could you specify if the three mongod processes and the YCSB process all run in a single server? If all four processes run inside a single server, then it is possible that your system is experiencing a memory contention issue.

MongoDB was designed to utilize as much memory as possible to provide you with the best performance, hence it is not recommended to run more than one mongod process in a machine, or running mongod alongside any resource-hungry application.

For more information and recommended settings for MongoDB deployment, please see:

• WiredTiger Memory Use
• MongoDB 3.2 Production Notes

Best regards,
Kevin

​

[tanveer madan]

Hi Kevin,

Thanks for your reply !!!

I am running each mongod on a separate server

Moreover I am running ycsb from a different server

I fixed the issue by disbaling the setting  vm.overcommit_memory=2 on the servers

But I am confused as to, the documentation says to disable overcommit of memory to run mongodb, but when I set this parameter the mongod was crashing

Interesting point was as below

1. My server was 64GB RAM, the mongod used to grow upto 33GB ( around 60% of the RAM as mentioned in the documentation) and then crash 

2. The secondary used to become primary and then similarly after it reached usage of 33GB the mongod used to crash and it as well

I am trying to understand why mongodb is behaving like this on a recommended setting

Thanks,

Tanveer

[Kevin Adistambha]

Hi Tanveer,

I fixed the issue by disbaling the setting vm.overcommit_memory=2 on the servers
But I am confused as to, the documentation says to disable overcommit of memory to run mongodb, but when I set this parameter the mongod was crashing

The overcommit setting is set situationally depending on how you want the server to behave during memory pressure situations. As such, there is no single best setting and it depends on your systems memory usage, which depends on your workload. You should review and set the overcommit setting and appropriate ratios according to your specific needs.

1. My server was 64GB RAM, the mongod used to grow upto 33GB ( around 60% of the RAM as mentioned in the documentation) and then crash

1. The secondary used to become primary and then similarly after it reached usage of 33GB the mongod used to crash and it as well

Having the database killed by the oomkiller or return “cannot allocate memory” is an indication of high demand for memory that is beyond the server’s capacities. As default, the WiredTiger cache is set to around 60% of RAM (which you have observed). From your description, there may be other memory intensive process in your servers that resulted in MongoDB not able to allocate memory.

Best regards,
Kevin

​

[tanveer madan]

Hi Kevin,

Thanks for the reply !!!

Regards,

Tanveer