Possible memory leak

[Matt Parlane]

Hi all...

I've upgraded our off-site secondary to 3.0 with WiredTiger, but now it's constantly running out of memory. It reliably consumes all 8GB of memory on the machine, and nothing else is running. This is the logs from today:

wg@kapiti:~$ grep allocate ~/mongodb-data/mongodb.log

2015-04-10T09:32:45.200+1200 E STORAGE  [rsSync] WiredTiger (12) [1428615158:751308][22698:0x7fe75e954700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T09:32:54.359+1200 E STORAGE  [rsSync] WiredTiger (12) [1428615174:157602][22698:0x7fe75e954700], session.commit_transaction: session unable to allocate a scratch buffer: Cannot allocate memory

2015-04-10T09:32:55.604+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 289

2015-04-10T10:18:02.162+1200 E STORAGE  [rsSync] WiredTiger (12) [1428617882:162564][24583:0x7f1493b55700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T10:18:02.248+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 289

2015-04-10T10:26:18.221+1200 E STORAGE  [rsSync] WiredTiger (12) [1428618377:976358][25220:0x7f875dd3b700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T10:26:21.222+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 289

2015-04-10T11:14:17.700+1200 E STORAGE  [rsSync] WiredTiger (12) [1428621257:669768][25603:0x7fd8187cb700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T11:14:17.792+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 289

2015-04-10T11:27:45.921+1200 E STORAGE  [rsSync] WiredTiger (12) [1428622065:215770][26318:0x7f43a2e46700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T11:27:46.093+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 289

2015-04-10T12:28:19.639+1200 E STORAGE  [rsSync] WiredTiger (12) [1428625699:583247][27241:0x7f539ddf2700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T12:28:20.394+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 289

2015-04-10T12:42:50.730+1200 E STORAGE  [rsSync] WiredTiger (12) [1428626570:730219][28131:0x7f36f30ef700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T12:42:51.507+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 290

2015-04-10T13:55:59.441+1200 E STORAGE  [rsSync] WiredTiger (12) [1428630958:660924][30601:0x7f8a19f67700], session.commit_transaction: memory allocation: Cannot allocate memory

2015-04-10T13:58:42.464+1200 I -        [rsSync] Invariant failure: s->commit_transaction(s, NULL) resulted in status UnknownError 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp 290

Within about 30 minutes of starting, top looks like this:

top - 16:06:43 up 146 days,  2:09,  4 users,  load average: 15.85, 15.60, 13.21

Tasks: 139 total,   2 running, 137 sleeping,   0 stopped,   0 zombie

%Cpu(s):  3.0 us,  3.9 sy,  0.0 ni, 27.1 id, 65.7 wa,  0.0 hi,  0.3 si,  0.0 st

KiB Mem:   8176712 total,  8041280 used,   135432 free,     1908 buffers

KiB Swap:  5949436 total,  2470668 used,  3478768 free.    84044 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

32697 wg        20   0 10.350g 7.205g   3672 S  15.6 92.4   9:07.00 /home/wg/mongodb/bin/mongod --dbpath /home/wg/mongodb-data/ --noauth --fork --logpath /home/wg/mongodb-data/mongodb.log --logappend --quiet --httpinterface --rest --replSet wgnz --storageEngine wired+

The machine is currently a few days behind the primary, so it's frantically applying oplog updates -- that may be a cause, I'm not sure. 3.0.1 and 3.0.2 do the same thing. The server is often started after an unclean shutdown (because of running out of memory), but I've also made sure to shut it down clean and restart it, and the same thing happens.

I'm using --wiredTigerCacheSizeGB 1, but I've also tried 2, 3, and 4 with similar results.

Does that sound like something you'd want to look into? What other information would be useful?

Thanks,

Matt

[Dan Pasette]

Hi Matt, there are a number of memory usage issues we're trying to track down with WiredTiger right now.  If you can add details of your system, deployment configuration and a bit about your use case in the jira ticket below, that definitely would be helpful.

You can watch/comment on this issue:

https://jira.mongodb.org/browse/SERVER-17424

Thanks
Dan

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev...@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-dev.
For more options, visit https://groups.google.com/d/optout.