[RocksCompactionThread] Got signal: 11 (Segmentation fault: 11).

[Mark G.]

Hi all,

I had been using percona-server-mongodb-3.4.7-1.8 and rocksdb 5.8 just fine for a few weeks. However, I get back to my office over the weekend, and the instance has segfaulted, and I can't get it up and running anymore with my data. I've bolded the errors. I've tried restarting with --repair but I get the same error. Any suggestions?

--Mark G.

2017-10-16T10:01:55.270-0500 I CONTROL  [main] ***** SERVER RESTARTED *****

2017-10-16T10:01:55.277-0500 I STORAGE  [main] [RocksDB] Block Cache Size GB: 2

2017-10-16T10:01:55.277-0500 I STORAGE  [main] [RocksDB] Compression: lz4

2017-10-16T10:01:55.277-0500 I STORAGE  [main] [RocksDB] MaxWriteMBPerSec: 1024

2017-10-16T10:01:55.277-0500 I STORAGE  [main] [RocksDB] Engine custom option: 

2017-10-16T10:01:55.277-0500 I STORAGE  [main] [RocksDB] Crash safe counters: 0

2017-10-16T10:01:55.277-0500 I STORAGE  [main] [RocksDB] Counters: 1

2017-10-16T10:01:55.277-0500 I STORAGE  [main] [RocksDB] Use SingleDelete in index: 0

2017-10-16T10:01:55.282-0500 I CONTROL  [initandlisten] MongoDB starting : pid=8419 port=27017 dbpath=/usr/local/var/mongodb 64-bit host=xxxxxxxxxx.local

2017-10-16T10:01:55.282-0500 I CONTROL  [initandlisten] db version v3.4.7-1.8

2017-10-16T10:01:55.282-0500 I CONTROL  [initandlisten] git version: ae48d6e032e8775d6cc57699e9bf210a0b99bdc9

2017-10-16T10:01:55.282-0500 I CONTROL  [initandlisten] allocator: system

2017-10-16T10:01:55.282-0500 I CONTROL  [initandlisten] modules: none

2017-10-16T10:01:55.282-0500 I CONTROL  [initandlisten] build environment:

2017-10-16T10:01:55.283-0500 I CONTROL  [initandlisten]     distarch: x86_64

2017-10-16T10:01:55.283-0500 I CONTROL  [initandlisten]     target_arch: x86_64

2017-10-16T10:01:55.283-0500 I CONTROL  [initandlisten] options: { config: "/usr/local/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, repair: true, security: { authorization: "enabled" }, storage: { dbPath: "/usr/local/var/mongodb", engine: "rocksdb", journal: { enabled: false }, rocksdb: { cacheSizeGB: 2, compression: "lz4" } }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/var/log/mongodb/mongo.log" } }

2017-10-16T10:01:55.292-0500 I STORAGE  [initandlisten] 2 dropped prefixes need compaction

2017-10-16T10:01:55.292-0500 F -        [RocksCompactionThread] Invalid access at address: 0x1b

2017-10-16T10:01:55.292-0500 I STORAGE  [initandlisten] Repairing catalog metadata

2017-10-16T10:01:55.295-0500 F -        [RocksCompactionThread] Got signal: 11 (Segmentation fault: 11).

 0x104cbbfd9 0x104cbb9ba 0x7fffe82f5b3a 0x107d7b508 0x107953e0a 0x10767aa94 0x1049b575c 0x1049b50b0 0x104c4b5ba 0x104c4d2be 0x7fffe82ff93b 0x7fffe82ff887 0x7fffe82ff08d

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"1040FE000","o":"BBDFD9","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"1040FE000","o":"BBD9BA","s":"_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv"},{"b":"7FFFE82F3000","o":"2B3A","s":"_sigtramp"},{"b":"107D49000","o":"32508","s":"je_tcache_arena_associate"},{"b":"1075FD000","o":"356E0A","s":"_ZN7rocksdb16IdentityFileNameERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE"},{"b":"1075FD000","o":"7DA94","s":"_ZNK7rocksdb6DBImpl13GetDbIdentityERNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE"},{"b":"1040FE000","o":"8B775C","s":"_ZN5mongo23CompactionBackgroundJob7compactERKNS0_9CompactOpE"},{"b":"1040FE000","o":"8B70B0","s":"_ZN5mongo23CompactionBackgroundJob3runEv"},{"b":"1040FE000","o":"B4D5BA","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"1040FE000","o":"B4F2BE","s":"_ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEENS_6__bindIMN5mongo13BackgroundJobEFvvEJPS9_EEEEEEEEPvSF_"},{"b":"7FFFE82FC000","o":"393B","s":"_pthread_body"},{"b":"7FFFE82FC000","o":"3887","s":"_pthread_body"},{"b":"7FFFE82FC000","o":"308D","s":"thread_start"}],"processInfo":{ "mongodbVersion" : "3.4.7-1.8", "gitVersion" : "ae48d6e032e8775d6cc57699e9bf210a0b99bdc9", "compiledModules" : [], "uname" : { "sysname" : "Darwin", "release" : "16.7.0", "version" : "Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/usr/local/bin/mongod", "machType" : 2, "b" : "1040FE000", "vmaddr" : "100000000", "buildId" : "A2ECB036470B3C0CA211B1135388882D" }, { "path" : "/usr/local/opt/rocksdb/lib/librocksdb.5.8.dylib", "machType" : 6, "b" : "1075FD000", "vmaddr" : "0", "buildId" : "F1733150D1DF3D63B0A90EE263A29CDA" }, { "path" : "/usr/lib/libz.1.dylib", "machType" : 6, "b" : "7FFFE7FEA000", "vmaddr" : "7FFF8AA8A000", "buildId" : "46E3FFA24328327A8D34A03E20BFFB8E" }, { "path" : "/usr/lib/libbz2.1.0.dylib", "machType" : 6, "b" : "7FFFE6C67000", "vmaddr" : "7FFF89707000", "buildId" : "ADFA329ADCE7356D8F09A3168DFC6610" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFFE6B3C000", "vmaddr" : "7FFF895DC000", "buildId" : "F18AC1E7C6F134B18069BE571B3231D4" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFFE6C76000", "vmaddr" : "7FFF89716000", "buildId" : "0B43BB5DE6EB34648DE9B41AC8ED9D1C" }, { "path" : "/usr/local/opt/snappy/lib/libsnappy.1.dylib", "machType" : 6, "b" : "107D05000", "vmaddr" : "0", "buildId" : "1875A745F9313F159DAED5BECC8433AD" }, { "path" : "/usr/local/opt/gflags/lib/libgflags.2.2.dylib", "machType" : 6, "b" : "107D14000", "vmaddr" : "0", "buildId" : "B05FC49CDD5A3727B95A7BB293BF2E40" }, { "path" : "/usr/local/opt/lz4/lib/liblz4.1.dylib", "machType" : 6, "b" : "107D33000", "vmaddr" : "0", "buildId" : "C62AC8446E343E0288B2C6633AF16AC0" }, { "path" : "/usr/local/opt/jemalloc/lib/libjemalloc.2.dylib", "machType" : 6, "b" : "107D49000", "vmaddr" : "0", "buildId" : "3277F5CF86BD3A7C9D6F416DFA306FA0" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFFE6CCD000", "vmaddr" : "7FFF8976D000", "buildId" : "BC271AD3831B362A9DA7E8C51F285FE4" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFFE800A000", "vmaddr" : "7FFF8AAAA000", "buildId" : "093A4DAB83853D47A350E20CB7CCF7BF" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFFE800F000", "vmaddr" : "7FFF8AAAF000", "buildId" : "8A64D1B0C70E385C92F0E669079FDA90" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFFE801A000", "vmaddr" : "7FFF8AABA000", "buildId" : "55D47421772A32ABB5291A46C2F43B4D" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFFE8022000", "vmaddr" : "7FFF8AAC2000", "buildId" : "819BEA3CDF113E3DA1A15A51C5BF1961" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFFE802B000", "vmaddr" : "7FFF8AACB000", "buildId" : "65D7165E2E71335DA2D633F78E2DF0C1" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFFE80AF000", "vmaddr" : "7FFF8AB4F000", "buildId" : "6582BAD6ED273B30B62090B1C5A4AE3C" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFFE80E1000", "vmaddr" : "7FFF8AB81000", "buildId" : "9B2AC56D107C3541A1279094A751F2C9" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFFE80E7000", "vmaddr" : "7FFF8AB87000", "buildId" : "7AA011A9DC213488BF733B5B14D1FDD6" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFFE80F5000", "vmaddr" : "7FFF8AB95000", "buildId" : "B856ABD2896E3DE0B2C8146A6AF8E2A7" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFFE80F6000", "vmaddr" : "7FFF8AB96000", "buildId" : "17D5D855F6C33B04B680E9BF02EF8AED" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFFE80FC000", "vmaddr" : "7FFF8AB9C000", "buildId" : "12448CC2378E35F3BE339DC395A5B970" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFFE80FF000", "vmaddr" : "7FFF8AB9F000", "buildId" : "38D4CB9C10CD30D38B7BA515EC75FE85" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFFE8101000", "vmaddr" : "7FFF8ABA1000", "buildId" : "096E42283B7C30A68B13EC909A64499A" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFFE811A000", "vmaddr" : "7FFF8ABBA000", "buildId" : "10DC540473AB35B3A277A8AFECB476EB" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFFE811B000", "vmaddr" : "7FFF8ABBB000", "buildId" : "E5AE52447D0C36AC8BB6C7AE7EA52A4B" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFFE81A9000", "vmaddr" : "7FFF8AC49000", "buildId" : "BECC01A2CA8D31E6BCDFD452965FA976" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFFE81AD000", "vmaddr" : "7FFF8AC4D000", "buildId" : "7D26DE79B424345085E1F7FAB32714AB" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFFE81B1000", "vmaddr" : "7FFF8AC51000", "buildId" : "EC6FCF07DCFB3A039CC96DD3709974C6" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFFE81CA000", "vmaddr" : "7FFF8AC6A000", "buildId" : "CC9602150B1B3822A13A3DDE96FA796F" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFFE81D1000", "vmaddr" : "7FFF8AC71000", "buildId" : "611DB84CBF703F928702B9F28A900920" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFFE81FB000", "vmaddr" : "7FFF8AC9B000", "buildId" : "34B1F16CBC9C3C5F90450CAE91CB5914" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFFE821E000", "vmaddr" : "7FFF8ACBE000", "buildId" : "86D499B5BBDC3D3B8A4E97AE8E6672A4" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFFE8266000", "vmaddr" : "7FFF8AD06000", "buildId" : "A3D15F1799A633678C7E4280E8619C95" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFFE8285000", "vmaddr" : "7FFF8AD25000", "buildId" : "369D022156CA3C3E9EDE94B41CAE77B7" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFFE82DF000", "vmaddr" : "7FFF8AD7F000", "buildId" : "B021F2B38A753633ABB0FC012B8E9B0C" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFFE82E9000", "vmaddr" : "7FFF8AD89000", "buildId" : "B8160190A0693B3ABDF62AA408221FAE" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFFE82F3000", "vmaddr" : "7FFF8AD93000", "buildId" : "897462FDB318321BA554E61982630F7E" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFFE82FC000", "vmaddr" : "7FFF8AD9C000", "buildId" : "B8FB5E20329539E2B5EBB464D1D4B104" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFFE8307000", "vmaddr" : "7FFF8ADA7000", "buildId" : "4B92EC49ACD036AEB07AA2B8152EAF9D" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFFE830B000", "vmaddr" : "7FFF8ADAB000", "buildId" : "F78B847B35653E4B98A6F7AD40392E2D" }, { "path" : "/usr/lib/system/libsystem_symptoms.dylib", "machType" : 6, "b" : "7FFFE830D000", "vmaddr" : "7FFF8ADAD000", "buildId" : "3390E07CC1CE348FADBD2C5440B45EAA" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFFE8315000", "vmaddr" : "7FFF8ADB5000", "buildId" : "AC63A7FE50D93A3096E6F6B7FF16E465" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFFE8329000", "vmaddr" : "7FFF8ADC9000", "buildId" : "3D50D8A8C460334DA5192DA841102C6B" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFFE832F000", "vmaddr" : "7FFF8ADCF000", "buildId" : "BF896DF0D8E931A8A4B301120BFEEE52" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFFE77EB000", "vmaddr" : "7FFF8A28B000", "buildId" : "70614861034032E285EDFE65759CDFFA" } ] }}

 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x104cbbfd9]

 mongod(_ZN5mongo12_GLOBAL__N_124abruptQuitWithAddrSignalEiP9__siginfoPv+0x12A) [0x104cbb9ba]

 libsystem_platform.dylib(_sigtramp+0x1A) [0x7fffe82f5b3a]

 libjemalloc.2.dylib(je_tcache_arena_associate+0x2A) [0x107d7b508]

 librocksdb.5.8.dylib(_ZN7rocksdb16IdentityFileNameERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE+0x15) [0x107953e0a]

 librocksdb.5.8.dylib(_ZNK7rocksdb6DBImpl13GetDbIdentityERNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE+0x38) [0x10767aa94]

 mongod(_ZN5mongo23CompactionBackgroundJob7compactERKNS0_9CompactOpE+0x37C) [0x1049b575c]

 mongod(_ZN5mongo23CompactionBackgroundJob3runEv+0x3B0) [0x1049b50b0]

 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x13A) [0x104c4b5ba]

 mongod(_ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEENS_6__bindIMN5mongo13BackgroundJobEFvvEJPS9_EEEEEEEEPvSF_+0x3E) [0x104c4d2be]

 libsystem_pthread.dylib(_pthread_body+0xB4) [0x7fffe82ff93b]

 libsystem_pthread.dylib(_pthread_body+0x0) [0x7fffe82ff887]

 libsystem_pthread.dylib(thread_start+0xD) [0x7fffe82ff08d]

-----  END BACKTRACE  -----

[Denis Protivenskii]

Hi Mark,

The error looks RocksDB-specific.

Did you experience this problem with previous versions of RocksDB?

Can you downgrade to, say, RocksDB 5.6.2 and try it instead?

- Denis.

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.
To post to this group, send email to percona-discussion@googlegroups.com.
Visit this group at https://groups.google.com/group/percona-discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/percona-discussion/38896669-4201-4812-8ff3-2ccad057e766%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark G.]

Denis,

Going down to 5.7.3 at least lets me start up fine. Kinda odd that it would work just fine for a week, then just break. I wasn't even doing anything over the weekend (no updates or anything) when it segfaulted. Should I bring this error up at percona/mongo-rocks?

--Mark

[Denis Protivenskii]

This error is in RocksDB. You can open an issue here: https://github.com/facebook/rocksdb/issues

--

You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.
To post to this group, send email to percona-discussion@googlegroups.com.
Visit this group at https://groups.google.com/group/percona-discussion.

To view this discussion on the web visit https://groups.google.com/d/msgid/percona-discussion/605209b4-5454-408e-a3aa-79169d71b760%40googlegroups.com.