mongod code=killed, signal=ABRT
1,602 views
Skip to first unread message
Владимир Кухлев
Aug 26, 2019, 4:09:52 AM8/26/19
to mongodb-user
После отключения электроэнергии не стартует mongod/
Ошибки:
[root@localhost ~]# service mongod status
Redirecting to /bin/systemctl status mongod.service
● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Mon 2019-08-26 10:34:47 MSK; 5min ago
Process: 9272 ExecStart=/usr/bin/mongod $OPTIONS (code=killed, signal=ABRT)
Process: 9267 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
Process: 9263 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
Process: 9260 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
Main PID: 9272 (code=killed, signal=ABRT)
Aug 26 10:34:45 localhost.localdomain systemd[1]: Starting High-performance, schema-free document-oriented database...
Aug 26 10:34:45 localhost.localdomain systemd[1]: Started High-performance, schema-free document-oriented database.
Aug 26 10:34:45 localhost.localdomain mongod[9270]: about to fork child process, waiting until server is ready for connections.
Aug 26 10:34:45 localhost.localdomain mongod[9270]: forked process: 9272
Aug 26 10:34:46 localhost.localdomain mongod[9270]: child process started successfully, parent exiting
Aug 26 10:34:47 localhost.localdomain systemd[1]: mongod.service: main process exited, code=killed, status=6/ABRT
Aug 26 10:34:47 localhost.localdomain systemd[1]: Unit mongod.service entered failed state.
Aug 26 10:34:47 localhost.localdomain systemd[1]: mongod.service failed.
LOG;
2019-08-24T14:32:19.349+0300 I CONTROL [main] ***** SERVER RESTARTED *****
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] MongoDB starting : pid=6775 port=27017 dbpath=/var/lib/mongo 64-bit host=localhost.localdomain
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] db version v3.4.7
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] allocator: tcmalloc
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] modules: none
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] build environment:
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] distmod: rhel70
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] distarch: x86_64
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] target_arch: x86_64
2019-08-24T14:32:19.403+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2019-08-24T14:32:19.403+0300 W - [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2019-08-24T14:32:19.418+0300 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-08-24T14:32:19.418+0300 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-08-24T14:32:19.418+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=15395M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2019-08-24T14:32:21.342+0300 I CONTROL [initandlisten]
2019-08-24T14:32:21.342+0300 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-08-24T14:32:21.342+0300 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-08-24T14:32:21.342+0300 I CONTROL [initandlisten]
2019-08-24T14:32:21.502+0300 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2019-08-24T14:32:21.502+0300 I NETWORK [thread1] waiting for connections on port 27017
2019-08-24T14:32:29.329+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:41204 #1 (1 connection now open)
2019-08-24T14:32:29.345+0300 I NETWORK [conn1] received client metadata from 127.0.0.1:41204 conn1: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T14:32:33.997+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:41206 #2 (2 connections now open)
2019-08-24T14:32:34.004+0300 I NETWORK [conn2] received client metadata from 127.0.0.1:41206 conn2: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T14:32:39.025+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:41208 #3 (3 connections now open)
2019-08-24T14:32:39.025+0300 I NETWORK [conn3] received client metadata from 127.0.0.1:41208 conn3: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T14:32:42.997+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:41210 #4 (4 connections now open)
2019-08-24T14:32:42.997+0300 I NETWORK [conn4] received client metadata from 127.0.0.1:41210 conn4: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T14:32:47.311+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:41216 #5 (5 connections now open)
2019-08-24T14:32:47.311+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:41218 #6 (6 connections now open)
2019-08-24T14:32:47.312+0300 I NETWORK [conn5] received client metadata from 127.0.0.1:41216 conn5: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T14:32:47.314+0300 I NETWORK [conn6] received client metadata from 127.0.0.1:41218 conn6: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T14:33:01.368+0300 E STORAGE [conn5] WiredTiger error (0) [1566646381:368787][6775:0x7f6ae0f4c700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: read checksum error for 12288B block at offset 547061760: block header checksum of 2317663528 doesn't match expected checksum of 3368889717
2019-08-24T14:33:01.368+0300 E STORAGE [conn5] WiredTiger error (0) [1566646381:368832][6775:0x7f6ae0f4c700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: collection-26-6807474155249217915.wt: encountered an illegal file format or internal value
2019-08-24T14:33:01.368+0300 E STORAGE [conn5] WiredTiger error (-31804) [1566646381:368840][6775:0x7f6ae0f4c700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-08-24T14:33:01.368+0300 I - [conn5] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-08-24T14:33:01.368+0300 I - [conn5]
***aborting after fassert() failure
2019-08-24T14:33:01.391+0300 I - [WTJournalFlusher] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2019-08-24T14:33:01.392+0300 I - [WTJournalFlusher]
***aborting after fassert() failure
2019-08-24T14:33:01.467+0300 F - [conn5] Got signal: 6 (Aborted).
0x55b565c18f81 0x55b565c18199 0x55b565c1867d 0x7f6aec45f5d0 0x7f6aec0b9207 0x7f6aec0ba8f8 0x55b564ec11d3 0x55b56592b916 0x55b564ecb4e2 0x55b564ecb6fe 0x55b564ecb956 0x55b566523585 0x55b56653d1be 0x55b5665444f3 0x55b56655cc6f 0x55b566528a69 0x55b566581a93 0x55b56591f5f4 0x55b5652258d5 0x55b565250a13 0x55b565244c96 0x55b565250a13 0x55b56555715a 0x55b565557a7b 0x55b56513c60b 0x55b56511302f 0x55b565114711 0x55b56572ca90 0x55b565330162 0x55b565332146 0x55b564f3090d 0x55b564f3123d 0x55b565b80ef1 0x7f6aec457dd5 0x7f6aec180ead
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55B5646A6000","o":"1572F81","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B5646A6000","o":"1572199"},{"b":"55B5646A6000","o":"157267D"},{"b":"7F6AEC450000","o":"F5D0"},{"b":"7F6AEC083000","o":"36207","s":"gsignal"},{"b":"7F6AEC083000","o":"378F8","s":"abort"},{"b":"55B5646A6000","o":"81B1D3","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B5646A6000","o":"1285916"},{"b":"55B5646A6000","o":"8254E2","s":"__wt_eventv"},{"b":"55B5646A6000","o":"8256FE","s":"__wt_err"},{"b":"55B5646A6000","o":"825956","s":"__wt_panic"},{"b":"55B5646A6000","o":"1E7D585","s":"__wt_bm_read"},{"b":"55B5646A6000","o":"1E971BE","s":"__wt_bt_read"},{"b":"55B5646A6000","o":"1E9E4F3","s":"__wt_page_in_func"},{"b":"55B5646A6000","o":"1EB6C6F"},{"b":"55B5646A6000","o":"1E82A69","s":"__wt_btcur_next"},{"b":"55B5646A6000","o":"1EDBA93"},{"b":"55B5646A6000","o":"12795F4","s":"_ZN5mongo21WiredTigerRecordStore6Cursor4nextEv"},{"b":"55B5646A6000","o":"B7F8D5","s":"_ZN5mongo14CollectionScan6doWorkEPm"},{"b":"55B5646A6000","o":"BAAA13","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"55B5646A6000","o":"B9EC96","s":"_ZN5mongo10LimitStage6doWorkEPm"},{"b":"55B5646A6000","o":"BAAA13","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"55B5646A6000","o":"EB115A","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"55B5646A6000","o":"EB1A7B","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"55B5646A6000","o":"A9660B","s":"_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE"},{"b":"55B5646A6000","o":"A6D02F","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"55B5646A6000","o":"A6E711","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"55B5646A6000","o":"1086A90","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"55B5646A6000","o":"C8A162"},{"b":"55B5646A6000","o":"C8C146","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"55B5646A6000","o":"88A90D","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"55B5646A6000","o":"88B23D"},{"b":"55B5646A6000","o":"14DAEF1"},{"b":"7F6AEC450000","o":"7DD5"},{"b":"7F6AEC083000","o":"FDEAD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.1.3.el7.x86_64", "version" : "#1 SMP Thu Nov 29 14:49:43 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55B5646A6000", "elfType" : 3, "buildId" : "433E85C3D902A85D7DFFD7F281B1A2C48A7ED7CD" }, { "b" : "7FFFA1816000", "elfType" : 3, "buildId" : "DF8F6BF69E976BF1266E476EA2E37CEE06F10C1D" }, { "b" : "7F6AED3F1000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AEF5E6F2240B55F90E9DF76CFBB8B9D9F5286583" }, { "b" : "7F6AECF90000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "8BD89856B64DD5189BF075EF574EDF203F93D44A" }, { "b" : "7F6AECD88000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7F6AECB84000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "67AD3498AC7DE3EAB952A243094DF5C12A21CD7D" }, { "b" : "7F6AEC882000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "918D3696BF321AA8D32950AB2AB8D0F1B21AC907" }, { "b" : "7F6AEC66C000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6B4F3D896CD0F06FCB3DEF0245F204ECE3220D7E" }, { "b" : "7F6AEC450000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3D9441083D079DC2977F1BD50C8068D11767232D" }, { "b" : "7F6AEC083000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "3C61131D1DAC9DA79B73188E7702BEF786C2AD54" }, { "b" : "7F6AED663000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5DA2D47925497B2F5875A7D8D1799A1227E2FDE4" }, { "b" : "7F6AEBE36000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B5C83BDE7ED7026835B779FA0F957FCCCD599F40" }, { "b" : "7F6AEBB4D000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8B63976509135BA73A12153D6FDF7B3B9E5D2A54" }, { "b" : "7F6AEB949000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "B4BE1023D9606A88169DF411BF94AF417D7BA1A0" }, { "b" : "7F6AEB72E000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6183129B5F29CA14580E517DF94EF317761FA6C9" }, { "b" : "7F6AEB518000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7F6AEB309000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "98F619035053EF68358099CE7CF1AA528B3B229D" }, { "b" : "7F6AEB105000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F6AEAEEC000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4C488F6E7044BB966162C1F7081ABBA6EBB2B485" }, { "b" : "7F6AEACC5000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7F6AEAA63000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b565c18f81]
mongod(+0x1572199) [0x55b565c18199]
mongod(+0x157267D) [0x55b565c1867d]
libpthread.so.0(+0xF5D0) [0x7f6aec45f5d0]
libc.so.6(gsignal+0x37) [0x7f6aec0b9207]
libc.so.6(abort+0x148) [0x7f6aec0ba8f8]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b564ec11d3]
mongod(+0x1285916) [0x55b56592b916]
mongod(__wt_eventv+0x3D7) [0x55b564ecb4e2]
mongod(__wt_err+0x9D) [0x55b564ecb6fe]
mongod(__wt_panic+0x24) [0x55b564ecb956]
mongod(__wt_bm_read+0x135) [0x55b566523585]
mongod(__wt_bt_read+0xAE) [0x55b56653d1be]
mongod(__wt_page_in_func+0x1303) [0x55b5665444f3]
mongod(+0x1EB6C6F) [0x55b56655cc6f]
mongod(__wt_btcur_next+0x399) [0x55b566528a69]
mongod(+0x1EDBA93) [0x55b566581a93]
mongod(_ZN5mongo21WiredTigerRecordStore6Cursor4nextEv+0x284) [0x55b56591f5f4]
mongod(_ZN5mongo14CollectionScan6doWorkEPm+0xC5) [0x55b5652258d5]
mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x55b565250a13]
mongod(_ZN5mongo10LimitStage6doWorkEPm+0x76) [0x55b565244c96]
mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x55b565250a13]
mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x19A) [0x55b56555715a]
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x55b565557a7b]
mongod(_ZN5mongo7FindCmd3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS8_RNS_14BSONObjBuilderE+0x106B) [0x55b56513c60b]
mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x4FF) [0x55b56511302f]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xF81) [0x55b565114711]
mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x240) [0x55b56572ca90]
mongod(+0xC8A162) [0x55b565330162]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x746) [0x55b565332146]
mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1FD) [0x55b564f3090d]
mongod(+0x88B23D) [0x55b564f3123d]
mongod(+0x14DAEF1) [0x55b565b80ef1]
libpthread.so.0(+0x7DD5) [0x7f6aec457dd5]
libc.so.6(clone+0x6D) [0x7f6aec180ead]
----- END BACKTRACE -----
2019-08-24T15:11:25.442+0300 I CONTROL [main] ***** SERVER RESTARTED *****
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] MongoDB starting : pid=6330 port=27017 dbpath=/var/lib/mongo 64-bit host=localhost.localdomain
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] db version v3.4.7
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] allocator: tcmalloc
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] modules: none
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] build environment:
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] distmod: rhel70
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] distarch: x86_64
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] target_arch: x86_64
2019-08-24T15:11:25.530+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2019-08-24T15:11:25.530+0300 W - [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2019-08-24T15:11:25.557+0300 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-08-24T15:11:25.557+0300 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-08-24T15:11:25.557+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=15395M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2019-08-24T15:11:27.036+0300 I CONTROL [initandlisten]
2019-08-24T15:11:27.036+0300 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-08-24T15:11:27.036+0300 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-08-24T15:11:27.036+0300 I CONTROL [initandlisten]
2019-08-24T15:11:27.182+0300 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2019-08-24T15:11:27.183+0300 I NETWORK [thread1] waiting for connections on port 27017
2019-08-24T15:11:28.007+0300 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2019-08-24T15:11:34.132+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:60318 #1 (1 connection now open)
2019-08-24T15:11:34.142+0300 I NETWORK [conn1] received client metadata from 127.0.0.1:60318 conn1: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T15:11:38.094+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:60320 #2 (2 connections now open)
2019-08-24T15:11:38.100+0300 I NETWORK [conn2] received client metadata from 127.0.0.1:60320 conn2: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T15:11:42.402+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:60326 #3 (3 connections now open)
2019-08-24T15:11:42.403+0300 I NETWORK [conn3] received client metadata from 127.0.0.1:60326 conn3: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T15:11:46.152+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:60332 #4 (4 connections now open)
2019-08-24T15:11:46.152+0300 I NETWORK [conn4] received client metadata from 127.0.0.1:60332 conn4: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T15:11:50.801+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:60350 #5 (5 connections now open)
2019-08-24T15:11:50.801+0300 I NETWORK [thread1] connection accepted from 127.0.0.1:60351 #6 (6 connections now open)
2019-08-24T15:11:50.801+0300 I NETWORK [conn5] received client metadata from 127.0.0.1:60350 conn5: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T15:11:50.801+0300 I NETWORK [conn6] received client metadata from 127.0.0.1:60351 conn6: { driver: { name: "mongo-java-driver", version: "unknown" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0-957.1.3.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_111-b14" }
2019-08-24T15:11:51.201+0300 I WRITE [conn3] update pvdrs.jrnl_ppozinmessage query: { _id: "6ab6d323-3aff-44be-937e-de0cde9a1a5a" } planSummary: IDHACK update: { _class: "ru.atc.pvd.rs.model.journal.JournalPpozInMessage", _id: "6ab6d323-3aff-44be-937e-de0cde9a1a5a", when: new Date(1566648711073), message: "{"requestNumber":"PKPVDMFC-2019-08-24-016384","extId":"MFC-0303/2019-48806","sender":"MFC-0303","senderType":"PKPVDMFC","statusCode":"paid","statusDes..." } keysExamined:0 docsExamined:0 nMatched:0 nModified:0 upsert:1 keysInserted:2 numYields:1 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } 123ms
2019-08-24T15:11:51.201+0300 I COMMAND [conn3] command pvdrs.$cmd command: update { update: "jrnl_ppozinmessage", ordered: false, updates: [ { q: { _id: "6ab6d323-3aff-44be-937e-de0cde9a1a5a" }, u: { _class: "ru.atc.pvd.rs.model.journal.JournalPpozInMessage", _id: "6ab6d323-3aff-44be-937e-de0cde9a1a5a", when: new Date(1566648711073), message: "{"requestNumber":"PKPVDMFC-2019-08-24-016384","extId":"MFC-0303/2019-48806","sender":"MFC-0303","senderType":"PKPVDMFC","statusCode":"paid","statusDes..." }, upsert: true } ] } numYields:0 reslen:139 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } protocol:op_query 123ms
2019-08-24T15:11:53.004+0300 I COMMAND [conn3] command pvdrs.jrnl_ppozoutrequest command: find { find: "jrnl_ppozoutrequest", filter: { requestNumber: "PKPVDMFC-2019-08-24-016384" }, limit: 1, singleBatch: true } planSummary: COLLSCAN keysExamined:0 docsExamined:169880 cursorExhausted:1 numYields:1342 nreturned:1 reslen:1434 locks:{ Global: { acquireCount: { r: 2686 } }, Database: { acquireCount: { r: 1343 } }, Collection: { acquireCount: { r: 1343 } } } protocol:op_query 1773ms
2019-08-24T15:11:53.345+0300 I WRITE [conn3] update pvdrs.search_subject query: { appealId: "acad6180-1fb3-49bd-a348-e0c81704d20a" } planSummary: IXSCAN { appealId: 1 } update: { $set: { dirty: true } } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 numYields:1 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } 158ms
2019-08-24T15:11:53.345+0300 I COMMAND [conn3] command pvdrs.$cmd command: update { update: "search_subject", ordered: false, updates: [ { q: { appealId: "acad6180-1fb3-49bd-a348-e0c81704d20a" }, u: { $set: { dirty: true } }, multi: true } ] } numYields:0 reslen:59 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } protocol:op_query 158ms
2019-08-24T15:12:06.532+0300 E STORAGE [conn6] WiredTiger error (0) [1566648726:532443][6330:0x7f854b482700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: read checksum error for 8192B block at offset 551112704: calculated block checksum of 1040589911 doesn't match expected checksum of 1131592315
2019-08-24T15:12:06.532+0300 E STORAGE [conn6] WiredTiger error (0) [1566648726:532520][6330:0x7f854b482700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: collection-26-6807474155249217915.wt: encountered an illegal file format or internal value
2019-08-24T15:12:06.532+0300 E STORAGE [conn6] WiredTiger error (-31804) [1566648726:532546][6330:0x7f854b482700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-08-24T15:12:06.532+0300 I - [conn6] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-08-24T15:12:06.532+0300 I - [conn6]
***aborting after fassert() failure
2019-08-24T15:12:06.557+0300 I - [WTJournalFlusher] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2019-08-24T15:12:06.557+0300 I - [WTJournalFlusher]
***aborting after fassert() failure
2019-08-24T15:12:06.616+0300 F - [conn6] Got signal: 6 (Aborted).
0x56433fa16f81 0x56433fa16199 0x56433fa1667d 0x7f8556a965d0 0x7f85566f0207 0x7f85566f18f8 0x56433ecbf1d3 0x56433f729916 0x56433ecc94e2 0x56433ecc96fe 0x56433ecc9956 0x564340321585 0x56434033b1be 0x5643403424f3 0x56434035ac6f 0x564340326a69 0x56434037fa93 0x56433f71d5f4 0x56433f0238d5 0x56433f04ea13 0x56433f042c96 0x56433f04ea13 0x56433f35515a 0x56433f355a7b 0x56433ef3a60b 0x56433ef1102f 0x56433ef12711 0x56433f52aa90 0x56433f12e162 0x56433f130146 0x56433ed2e90d 0x56433ed2f23d 0x56433f97eef1 0x7f8556a8edd5 0x7f85567b7ead
----- BEGIN BACKTRACE -----
Oleg Toropov
Aug 26, 2019, 6:19:23 PM8/26/19
to mongod...@googlegroups.com
Vladimir
Looks like your storage (WiredTiger files) is corrupted:
2019-08-24T14:33:01.368+0300 E STORAGE [conn5] WiredTiger error (0) [1566646381:368787][6775:0x7f6ae0f4c700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: read checksum error for 12288B block at offset 547061760: block header checksum of 2317663528 doesn't match expected checksum of 33688897172019-08-24T14:33:01.368+0300 E STORAGE [conn5] WiredTiger error (0) [1566646381:368832][6775:0x7f6ae0f4c700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: collection-26-6807474155249217915.wt: encountered an illegal file format or internal value2019-08-24T14:33:01.368+0300 E STORAGE [conn5] WiredTiger error (-31804) [1566646381:368840][6775:0x7f6ae0f4c700], file:collection-26-6807474155249217915.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic2019-08-24T14:33:01.368+0300 I - [conn5] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 3612019-08-24T14:33:01.368+0300 I - [conn5]
There are some tools that may recover your database: https://docs.mongodb.com/manual/tutorial/recover-data-following-unexpected-shutdown/
but it does not work all the time.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/0a7e9984-d3a4-46c4-845d-51227a2f88ba%40googlegroups.com.
--
//
// Oleg V.Toropov, MCP | oleg.t...@gmail.com
// -----------------------------------------------------------------
// I have discovered that if I read enough stuff that's over my head,
// I actually begin to understand some of it.
//
// Oleg V.Toropov, MCP | oleg.t...@gmail.com
// -----------------------------------------------------------------
// I have discovered that if I read enough stuff that's over my head,
// I actually begin to understand some of it.
//
Reply all
Reply to author
Forward
0 new messages