MongoDB consumes all memory

[Алексей Гаранин]

Hi guys

I use MongoDB 3.2, WiredTiger engine. I am trying to use batch inserts on 10K records, the size of one record about 1 KB. All great, but 60-70 million such records memory ends. Mongo is limited cache in 3GB, but the memory is consumed by memory map files collections and indexes. After some time Mongo CPU load at 100% and stops receiving data. What am I doing wrong? :)

OS: Windows 7

RAM: 8GB

CPU: 4 core

[Алексей Гаранин]

https://blog.clevertap.com/sleepless-nights-with-mongodb-wiredtiger-and-our-return-to-mmapv1/ :D

среда, 28 сентября 2016 г., 0:56:32 UTC+3 пользователь Алексей Гаранин написал:

[Pooja Gupta]

Hi,

Since this question has been posted sometime ago, have you found a way to solve the issues you were seeing?

I am trying to use batch inserts on 10K records, the size of one record about 1 KB. All great, but 60-70 million such records memory ends.

Could you please elaborate on:

• how many records do you have in total? Are you doing batch inserts on 10k records at a time?
• What method have you adopted for doing batch inserts (mongorestore/ mongoimport/ custom code etc.)?

Also, could you please clarify your statement a bit:

Mongo is limited cache in 3GB, but the memory is consumed by memory map files collections and indexes

• What method did you use to limit the cache size?
• How did you determine the memory consumption?

After some time Mongo CPU load at 100% and stops receiving data.

Could you post the output of diagnostic tools (e.g. mongostat) and the relevant mongod logs ?

Regards,
Pooja

​

[Алексей Гаранин]

Since this question has been posted sometime ago, have you found a way to solve the issues you were seeing?

No

• how many records do you have in total? Are you doing batch inserts on 10k records at a time?

~ 40 million. Yes,  batch inserts on 10k records.

• What method have you adopted for doing batch inserts (mongorestore/ mongoimport/ custom code etc.)? 

custom code (https://docs.mongodb.com/v3.2/reference/method/Bulk.insert/) 

• What method did you use to limit the cache size?

wiredTiger.engineConfig.cacheSizeGB

Could you post the output of diagnostic tools (e.g. mongostat) and the relevant mongod logs ?

attached 

среда, 26 октября 2016 г., 10:43:27 UTC+3 пользователь Pooja Gupta написал:

[Kevin Adistambha]

Hi

Sorry for the delay in responding. What MongoDB version are you using? Is it possible for you to upgrade to the latest in the 3.2 series (currently 3.2.10) and see if the issue is still there? It also may be worth trying to import using mongoimport and see if the import is proceeding without pauses.

Best regards,
Kevin

​