Recovering mongodb data from folder causing assertion error.

[Maisnam Raju]

A server I was using crashed because of which I couldn't run mongodb so I decided to copy the data/db folder to my local machine and try to recover it but on running 

mongod --repair command I got the following error. This is the first time I have encountered such a thing and I am not able to debug it. The data is quite large and I want to recover as much of it as possible. Any advice would be appreciated. 

 sudo mongod --repair 

2016-07-14T14:35:57.836+0530 I CONTROL  [initandlisten] MongoDB starting : pid=10704 port=27017 dbpath=/data/db 64-bit host=kseguy-X550CL

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten] db version v3.2.8

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips  1 Mar 2016

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten] allocator: tcmalloc

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten] modules: none

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten] build environment:

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten]     distmod: ubuntu1404

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten]     distarch: x86_64

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten]     target_arch: x86_64

2016-07-14T14:35:57.837+0530 I CONTROL  [initandlisten] options: { repair: true }

2016-07-14T14:35:57.873+0530 I -        [initandlisten] Detected data files in /data/db created by the 'mmapv1' storage engine, so setting the active storage engine to 'mmapv1'.

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] 

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] 

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] 

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'

2016-07-14T14:35:57.890+0530 I CONTROL  [initandlisten] 

2016-07-14T14:35:57.890+0530 I STORAGE  [initandlisten] repairDatabase Enigma_Client_Main_Data

2016-07-14T14:35:57.894+0530 I -        [initandlisten] Assertion: 10334:BSONObj size: 0 (0x0) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO

2016-07-14T14:35:57.909+0530 I CONTROL  [initandlisten] 

 0x131ce72 0x12ba578 0x12a6e58 0x12a6f0c 0x9dc0e9 0x100fbf2 0x100fa29 0xb0b201 0xaeb2ec 0xafb482 0xafcfd4 0xaffe5d 0x106bccf 0xe5d07f 0x9b214a 0x9b4e50 0x96e04d 0x7fa38ba20830 0x9b1037

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"F1CE72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EBA578","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"EA6E58","s":"_ZN5mongo11msgassertedEiPKc"},{"b":"400000","o":"EA6F0C"},{"b":"400000","o":"5DC0E9","s":"_ZNK5mongo7BSONObj14_assertInvalidEv"},{"b":"400000","o":"C0FBF2","s":"_ZNK5mongo10RecordData6toBsonEv"},{"b":"400000","o":"C0FA29","s":"_ZNK5mongo38NamespaceDetailsCollectionCatalogEntry13getAllIndexesEPNS_16OperationContextEPSt6vectorISsSaISsEE"},{"b":"400000","o":"70B201","s":"_ZN5mongo12IndexCatalog4initEPNS_16OperationContextE"},{"b":"400000","o":"6EB2EC","s":"_ZN5mongo10CollectionC2EPNS_16OperationContextENS_10StringDataEPNS_22CollectionCatalogEntryEPNS_11RecordStoreEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"6FB482","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"6FCFD4","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"6FFE5D","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},{"b":"400000","o":"C6BCCF","s":"_ZN5mongo12MMAPV1Engine14repairDatabaseEPNS_16OperationContextERKSsbb"},{"b":"400000","o":"A5D07F","s":"_ZN5mongo14repairDatabaseEPNS_16OperationContextEPNS_13StorageEngineERKSsbb"},{"b":"400000","o":"5B214A"},{"b":"400000","o":"5B4E50","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"56E04D","s":"main"},{"b":"7FA38BA00000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5B1037"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-28-generic", "version" : "#47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A53FF676E1D627BD1D9B1BF524DEFA13B667EE83" }, { "b" : "7FFEBDFC8000", "elfType" : 3, "buildId" : "384F8012518E0A03AE7DFF4C3EF240196A6EC13E" }, { "b" : "7FA38CD6C000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "822754695E4B31AE82937258BDFF3D52EFA0BA36" }, { "b" : "7FA38C911000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "B75A96C59BE1B5B54FBF1A91ED722BEC9406288E" }, { "b" : "7FA38C709000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A779DBCB3A477DC0C8D09B60FAC7335D396C19DF" }, { "b" : "7FA38C505000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "A2ADF3615338D49C702C41EB83A99AB743D2B574" }, { "b" : "7FA38C1FC000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "5C4078C04888A418F3DB0868702ECFDB35B3AD8B" }, { "b" : "7FA38BFE6000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FA38BDC9000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B77847CC9CACBCA3B5753D0D25A32E5795AFE75B" }, { "b" : "7FA38BA00000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "369DE0E1D833CAA693AF17F17C83BA937F0A4DAD" }, { "b" : "7FA38CFD5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EDFA6D46E00CA97F349FDD3333D88493D442932C" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x131ce72]

 mongod(_ZN5mongo10logContextEPKc+0x138) [0x12ba578]

 mongod(_ZN5mongo11msgassertedEiPKc+0x88) [0x12a6e58]

 mongod(+0xEA6F0C) [0x12a6f0c]

 mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x3B9) [0x9dc0e9]

 mongod(_ZNK5mongo10RecordData6toBsonEv+0x52) [0x100fbf2]

 mongod(_ZNK5mongo38NamespaceDetailsCollectionCatalogEntry13getAllIndexesEPNS_16OperationContextEPSt6vectorISsSaISsEE+0x129) [0x100fa29]

 mongod(_ZN5mongo12IndexCatalog4initEPNS_16OperationContextE+0x61) [0xb0b201]

 mongod(_ZN5mongo10CollectionC2EPNS_16OperationContextENS_10StringDataEPNS_22CollectionCatalogEntryEPNS_11RecordStoreEPNS_20DatabaseCatalogEntryE+0x3CC) [0xaeb2ec]

 mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0xA2) [0xafb482]

 mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x284) [0xafcfd4]

 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0x18D) [0xaffe5d]

 mongod(_ZN5mongo12MMAPV1Engine14repairDatabaseEPNS_16OperationContextERKSsbb+0x2EF) [0x106bccf]

 mongod(_ZN5mongo14repairDatabaseEPNS_16OperationContextEPNS_13StorageEngineERKSsbb+0xFEF) [0xe5d07f]

 mongod(+0x5B214A) [0x9b214a]

 mongod(_ZN5mongo13initAndListenEi+0x930) [0x9b4e50]

 mongod(main+0x15D) [0x96e04d]

 libc.so.6(__libc_start_main+0xF0) [0x7fa38ba20830]

 mongod(+0x5B1037) [0x9b1037]

-----  END BACKTRACE  -----

2016-07-14T14:35:57.910+0530 I STORAGE  [initandlisten] cleaning up failed repair db: Enigma_Client_Main_Data path: /data/db/_tmp_repairDatabase_0

2016-07-14T14:35:57.910+0530 I STORAGE  [initandlisten] exception in initAndListen: 10334 BSONObj size: 0 (0x0) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO, terminating

2016-07-14T14:35:57.910+0530 I CONTROL  [initandlisten] now exiting

2016-07-14T14:35:57.910+0530 I NETWORK  [initandlisten] shutdown: going to close listening sockets...

2016-07-14T14:35:57.910+0530 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock

2016-07-14T14:35:57.910+0530 I NETWORK  [initandlisten] shutdown: going to flush diaglog...

2016-07-14T14:35:57.910+0530 I NETWORK  [initandlisten] shutdown: going to close sockets...

2016-07-14T14:35:57.910+0530 I STORAGE  [initandlisten] shutdown: waiting for fs preallocator...

2016-07-14T14:35:57.910+0530 I STORAGE  [initandlisten] shutdown: closing all files...

2016-07-14T14:35:57.911+0530 I STORAGE  [initandlisten] closeAllFiles() finished

2016-07-14T14:35:57.911+0530 I STORAGE  [initandlisten] shutdown: removing fs lock...

2016-07-14T14:35:57.911+0530 I CONTROL  [initandlisten] dbexit:  rc: 100