MongoDB crashes with "out of memory AlignedBuilder"

785 views
Skip to first unread message

theted

unread,
Dec 19, 2012, 10:51:23 AM12/19/12
to mongod...@googlegroups.com
Hello mongodb-user members,

I am hoping someone from 10gen can help with a server crash I have had twice, where a MongoDB server crashes with the error "out of memory Aligned Builder".  This occurs on my replicaset secondary, but the primary is fine.  Here are two excerpts from the logs (details below):

  Wed Oct 24 00:22:31 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder

  Wed Dec 19 03:15:39 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder

I am running MongoDB 2.2.0 in a 3-node replicaset (Primary/Secondary/Arbiter) on Amazon EC2.  The data volume for Mongo is 20 GB, of which 7 GB is used.  The system has 1.7 GB of RAM (EC2 m1.small).

Below is a tail of the log from the most recent crash (with a few tiny redactions).  I would be happy to provide more logs or other data if needed.  I am using MMS if  it contains data that would help.

I am running 2.2.0 and am aware that 2.2.2 is now available, but I have read the changelog and I do not see any fixes that are relevant to this problem.

Thanks for any help that you might be able to provide!

Ted


Wed Dec 19 03:15:13 [rsSync]  local.oplog.rs warning assertion failure _intents.size() < 2000000 src/mongo/db/dur_commitjob.h 101
0xade6e1 0x802c5a 0x78c4a0 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x625fe7 0xa60dcb 0xa61afc 0x62dc57 0x62dc98 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x280) [0x78c4a0]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails13addDeletedRecEPNS_13DeletedRecordENS_7DiskLocE+0x27) [0x625fe7]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails7compactEv+0x2eb) [0xa60dcb]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x4bc) [0xa61afc]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x27) [0x62dc57]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x38) [0x62dc98]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
Wed Dec 19 03:15:18 [rsSync]  local.oplog.rs warning assertion failure _intents.size() < 2000000 src/mongo/db/dur_commitjob.h 101
0xade6e1 0x802c5a 0x78c4a0 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0xb670ba 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f2d67e03851 0x7f2d671a911d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x280) [0x78c4a0]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x2aa) [0xb670ba]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /usr/bin/mongod() [0x7c3659]
 /lib64/libpthread.so.0(+0x7851) [0x7f2d67e03851]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d671a911d]
Wed Dec 19 03:15:20 [conn5327] end connection [IP REDACTED]:53078 (4 connections now open)
Wed Dec 19 03:15:20 [initandlisten] connection accepted from [IP REDACTED]:53080 #5336 (5 connections now open)
Wed Dec 19 03:15:20 [conn5336]  authenticate db: local { authenticate: 1, nonce: "5a5f1551b102034d", user: "__system", key: [REDACTED] }
Wed Dec 19 03:15:21 [initandlisten] connection accepted from [IP REDACTED]:59311 #5337 (6 connections now open)
Wed Dec 19 03:15:21 [conn5337] end connection [IP REDACTED]:59311 (5 connections now open)
Wed Dec 19 03:15:23 [rsSync]  local.oplog.rs warning assertion failure _intents.size() < 2000000 src/mongo/db/dur_commitjob.h 101
0xade6e1 0x802c5a 0x78c4a0 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x62dd73 0x73954c 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f2d67e03851 0x7f2d671a911d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x280) [0x78c4a0]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x113) [0x62dd73]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x6c) [0x73954c]
 /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /usr/bin/mongod() [0x7c3659]
 /lib64/libpthread.so.0(+0x7851) [0x7f2d67e03851]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d671a911d]
Wed Dec 19 03:15:23 [conn5329] end connection [IP REDACTED]:43292 (4 connections now open)
Wed Dec 19 03:15:23 [initandlisten] connection accepted from [IP REDACTED]:43298 #5338 (5 connections now open)
Wed Dec 19 03:15:23 [conn5338]  authenticate db: local { authenticate: 1, nonce: "cd59e3d5cedca2b1", user: "__system", key: [REDACTED] }
Wed Dec 19 03:15:28 [rsSync]  local.oplog.rs warning assertion failure _intents.size() < 2000000 src/mongo/db/dur_commitjob.h 101
0xade6e1 0x802c5a 0x78c4a0 0x78c4ff 0x78c7d2 0x78c8ed 0x78c95b 0xa07c1a 0x739675 0xb6708a 0x64b5eb 0x65345e 0x6538f8 0x65394a 0x653d58 0x7c3659 0x7f2d67e03851 0x7f2d671a911d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
 /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x280) [0x78c4a0]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents8_unspoolEv+0x4f) [0x78c4ff]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents7unspoolEv+0x52) [0x78c7d2]
 /usr/bin/mongod(_ZN5mongo3dur18ThreadLocalIntents4pushERKNS0_11WriteIntentE+0x6d) [0x78c8ed]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl18declareWriteIntentEPvj+0x6b) [0x78c95b]
 /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0xa07c1a]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x195) [0x739675]
 /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x27a) [0xb6708a]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0x64b5eb]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x48e) [0x65345e]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0x6538f8]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x65394a]
 /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x653d58]
 /usr/bin/mongod() [0x7c3659]
 /lib64/libpthread.so.0(+0x7851) [0x7f2d67e03851]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d671a911d]
Wed Dec 19 03:15:31 [initandlisten] connection accepted from [IP REDACTED]:59314 #5339 (6 connections now open)
Wed Dec 19 03:15:31 [conn5339] end connection [IP REDACTED]:59314 (5 connections now open)
Wed Dec 19 03:15:31 [conn5209] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:13251 reslen:3861 619ms
Wed Dec 19 03:15:31 [conn5218] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:13204 reslen:3861 621ms
Wed Dec 19 03:15:34 [journal]   warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
0xade6e1 0x802c5a 0x77dc73 0x753da5 0x7540b4 0xa09950 0xa0a779 0xa0ae24 0x7c3659 0x7f2d67e03851 0x7f2d671a911d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
 /usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x77dc73]
 /usr/bin/mongod() [0x753da5]
 /usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x7540b4]
 /usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0xa0) [0xa09950]
 /usr/bin/mongod() [0xa0a779]
 /usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
 /usr/bin/mongod() [0x7c3659]
 /lib64/libpthread.so.0(+0x7851) [0x7f2d67e03851]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d671a911d]
Wed Dec 19 03:15:39 [journal] Assertion: 13524:out of memory AlignedBuilder
0xade6e1 0x8036eb 0x77dadc 0x77db09 0x62284e 0x622c12 0xa099f1 0xa0a779 0xa0ae24 0x7c3659 0x7f2d67e03851 0x7f2d671a911d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0x8036eb]
 /usr/bin/mongod() [0x77dadc]
 /usr/bin/mongod(_ZN5mongo14AlignedBuilder8_reallocEjj+0x29) [0x77db09]
 /usr/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x4e) [0x62284e]
 /usr/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x622c12]
 /usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x141) [0xa099f1]
 /usr/bin/mongod() [0xa0a779]
 /usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
 /usr/bin/mongod() [0x7c3659]
 /lib64/libpthread.so.0(+0x7851) [0x7f2d67e03851]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d671a911d]
Wed Dec 19 03:15:39 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder
Wed Dec 19 03:15:39 dur1
Wed Dec 19 03:15:39 Got signal: 6 (Aborted).

Wed Dec 19 03:15:39 Backtrace:
0xade6e1 0x5582d9 0x7f2d670f3920 0x7f2d670f38a5 0x7f2d670f5085 0xb503f7 0xa09e1f 0xa0a779 0xa0ae24 0x7c3659 0x7f2d67e03851 0x7f2d671a911d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9]
 /lib64/libc.so.6(+0x32920) [0x7f2d670f3920]
 /lib64/libc.so.6(gsignal+0x35) [0x7f2d670f38a5]
 /lib64/libc.so.6(abort+0x175) [0x7f2d670f5085]
 /usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0xb503f7]
 /usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x56f) [0xa09e1f]
 /usr/bin/mongod() [0xa0a779]
 /usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
 /usr/bin/mongod() [0x7c3659]
 /lib64/libpthread.so.0(+0x7851) [0x7f2d67e03851]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d671a911d]

Nadir Boussoukaia

unread,
Apr 25, 2014, 6:32:23 AM4/25/14
to mongod...@googlegroups.com
I had this problem when filesystem went to 100% (due verbose logs that where not in /var).
impossible to repair the database. 
the only way is to restart from a backup, and exporting the proper tables.
to be able to re-start the DB in that situation (for mongodump or because it is prod)

You have to start Mongo without journaling (very dangerous, you should leave this mode asap)
set nojournal = true in mongod.conf
then:
rm /mnt/pathto/mongodb/journal/*
rm /mnt/pathto/mongodb/mongod.lock
chown -R  mongodb:mongodb /mnt/pathto/mongodb (in case you tried as root "mongod -f /etc/mongodb.conf --repair")
service mongodb start

DB is up & running, do your dumps a restart it WITH journaling



On Thursday, April 25, 2013 11:22:40 PM UTC+2, Shuai Yuan wrote:
I met the same problem. 
It turned out to be some python programs eating up all memory.

Hope this helps.

Asya Kamsky

unread,
Apr 28, 2014, 12:07:00 AM4/28/14
to mongodb-user
Definitely please no one ever do this.

If you have a problem recovering your database, please ask for help here, don't go deleting files as that's the surest path to data loss.

Asya



--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b8b6502c-b653-40c0-9dfc-bd9955d8405d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages