redis keeps losing connection to master
276 views
Skip to first unread message
Jan
Aug 5, 2023, 11:34:29 AM8/5/23
to Redis DB
Hello,
Can someone where should I look for the solution of my issue? I have 3 redis nodes with redis-sentinel, all default settings. When finally clients were connected after some short time redis sentinel is switching master nodes. Sometimes the loop takes 3h, sometimes just once a day.
logs:
1013003:C 02 Aug 2023 06:28:03.167 * Fork CoW for RDB: current 22 MB, peak 22 MB, average 13 MB
434:S 02 Aug 2023 06:28:03.328 * Background saving terminated with success
434:M 02 Aug 2023 06:28:36.368 # Connection with master lost.
434:M 02 Aug 2023 06:28:36.368 * Caching the disconnected master state.
434:S 02 Aug 2023 06:28:36.369 * Connecting to MASTER 10.13.0.27:6379
434:S 02 Aug 2023 06:28:36.369 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:28:36.369 * REPLICAOF 10.13.0.27:6379 enabled (user request from 'id=10509472 addr=10.13.0.27:39224 laddr=10.13.0.28:6379 fd=12 name=sentinel-44d0c53a-cmd age=607 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=344 qbuf-free=20130 argv-mem=4 multi-mem=178 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 tot-mem=23598 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:28:36.392 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:28:36.393 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:28:36.394 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:28:36.396 * Trying a partial resynchronization (request 0bda6e46ec3e27e264c5e7ac6394239071161f6e:39027397143).
434:S 02 Aug 2023 06:28:36.397 * Successful partial resynchronization with master.
434:S 02 Aug 2023 06:28:36.397 # Master replication ID changed to 5d0a0d69dafc3e5225cadf78b3a3ece6c1cf5b7c
434:S 02 Aug 2023 06:28:36.398 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
434:S 02 Aug 2023 06:30:12.850 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:30:12.894 * Background saving started by pid 1013054
1013054:C 02 Aug 2023 06:30:25.627 * DB saved on disk
1013054:C 02 Aug 2023 06:30:25.664 * Fork CoW for RDB: current 27 MB, peak 27 MB, average 17 MB
434:S 02 Aug 2023 06:30:25.810 * Background saving terminated with success
434:S 02 Aug 2023 06:31:26.020 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:31:26.072 * Background saving started by pid 1013079
434:M 02 Aug 2023 06:31:27.411 # Connection with master lost.
434:M 02 Aug 2023 06:31:27.411 * Caching the disconnected master state.
434:M 02 Aug 2023 06:31:27.411 * Discarding previously cached master state.
434:M 02 Aug 2023 06:31:27.411 # Setting secondary replication ID to 5d0a0d69dafc3e5225cadf78b3a3ece6c1cf5b7c, valid up to offset: 39031081016. New replication ID is 6c80e9b301191e2866ec410ae9352a615694d217
434:M 02 Aug 2023 06:31:27.411 * MASTER MODE enabled (user request from 'id=10511284 addr=10.13.0.28:60236 laddr=10.13.0.28:6379 fd=10 name=sentinel-6d24eb3f-cmd age=171 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=188 qbuf-free=20286 argv-mem=4 multi-mem=169 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 tot-mem=23589 events=r cmd=exec user=default redir=-1 resp=2')
434:M 02 Aug 2023 06:31:27.454 # CONFIG REWRITE executed with success.
1013079:C 02 Aug 2023 06:31:37.920 * DB saved on disk
1013079:C 02 Aug 2023 06:31:37.985 * Fork CoW for RDB: current 13 MB, peak 13 MB, average 9 MB
434:M 02 Aug 2023 06:31:38.168 * Background saving terminated with success
434:M 02 Aug 2023 06:31:41.195 * Replica 10.13.0.29:6379 asks for synchronization
434:M 02 Aug 2023 06:31:41.196 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '0bda6e46ec3e27e264c5e7ac6394239071161f6e', my replication IDs are '6c80e9b301191e2866ec410ae9352a615694d217' and '5d0a0d69dafc3e5225cadf78b3a3ece6c1cf5b7c')
434:M 02 Aug 2023 06:31:41.196 * Delay next BGSAVE for diskless SYNC
434:M 02 Aug 2023 06:31:46.849 * Starting BGSAVE for SYNC with target: replicas sockets
434:M 02 Aug 2023 06:31:46.901 * Background RDB transfer started by pid 1013080
1013080:C 02 Aug 2023 06:31:58.838 * Fork CoW for RDB: current 21 MB, peak 21 MB, average 15 MB
434:M 02 Aug 2023 06:31:58.838 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
434:M 02 Aug 2023 06:31:58.977 * Background RDB transfer terminated with success
434:M 02 Aug 2023 06:31:58.977 * Streamed RDB transfer with replica 10.13.0.29:6379 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming
434:M 02 Aug 2023 06:31:58.977 * Synchronization with replica 10.13.0.29:6379 succeeded
434:M 02 Aug 2023 06:32:49.274 * 10000 changes in 60 seconds. Saving...
434:M 02 Aug 2023 06:32:49.324 * Background saving started by pid 1013104
1013104:C 02 Aug 2023 06:33:02.737 * DB saved on disk
1013104:C 02 Aug 2023 06:33:02.771 * Fork CoW for RDB: current 12 MB, peak 12 MB, average 8 MB
434:M 02 Aug 2023 06:33:02.865 * Background saving terminated with success
434:M 02 Aug 2023 06:37:31.374 * 10000 changes in 60 seconds. Saving...
434:M 02 Aug 2023 06:37:31.406 * Background saving started by pid 1013183
434:M 02 Aug 2023 06:37:31.408 # Connection with replica client id #10511890 lost.
434:S 02 Aug 2023 06:37:42.409 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
434:S 02 Aug 2023 06:37:42.409 * Connecting to MASTER 10.13.0.29:6379
434:S 02 Aug 2023 06:37:42.409 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:37:42.409 * REPLICAOF 10.13.0.29:6379 enabled (user request from 'id=10513017 addr=10.13.0.27:46996 laddr=10.13.0.28:6379 fd=8 name=sentinel-44d0c53a-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=198 qbuf-free=20276 argv-mem=4 multi-mem=178 rbs=1024 rbp=1024 obl=45 oll=0 omem=0 tot-mem=22574 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:37:42.423 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:37:42.423 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:37:42.424 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:37:42.425 * Trying a partial resynchronization (request 6c80e9b301191e2866ec410ae9352a615694d217:39039868221).
1013183:C 02 Aug 2023 06:37:44.353 * DB saved on disk
1013183:C 02 Aug 2023 06:37:44.389 * Fork CoW for RDB: current 70 MB, peak 70 MB, average 56 MB
434:S 02 Aug 2023 06:37:44.493 * Background saving terminated with success
434:S 02 Aug 2023 06:37:47.360 * Full resync from master: 8693844ab2bba2ba427baf357c179cf75f4d407c:39042175434
434:S 02 Aug 2023 06:37:47.452 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
434:S 02 Aug 2023 06:37:58.782 * Discarding previously cached master state.
434:S 02 Aug 2023 06:37:58.783 * MASTER <-> REPLICA sync: Flushing old data
434:S 02 Aug 2023 06:38:04.438 * MASTER <-> REPLICA sync: Loading DB in memory
434:S 02 Aug 2023 06:38:04.450 * Loading RDB produced by version 7.0.11
434:S 02 Aug 2023 06:38:04.450 * RDB age 17 seconds
434:S 02 Aug 2023 06:38:04.450 * RDB memory usage when created 1017.19 Mb
434:S 02 Aug 2023 06:38:12.274 * Done loading RDB, keys loaded: 1136219, keys expired: 0.
434:S 02 Aug 2023 06:38:12.274 * MASTER <-> REPLICA sync: Finished with success
434:S 02 Aug 2023 06:38:45.080 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:38:45.133 * Background saving started by pid 1013207
1013207:C 02 Aug 2023 06:38:56.044 * DB saved on disk
1013207:C 02 Aug 2023 06:38:56.078 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 2 MB
434:S 02 Aug 2023 06:38:56.212 * Background saving terminated with success
434:M 02 Aug 2023 06:39:14.632 # Connection with master lost.
434:M 02 Aug 2023 06:39:14.632 * Caching the disconnected master state.
434:S 02 Aug 2023 06:39:14.632 * Connecting to MASTER 10.13.0.27:6379
434:S 02 Aug 2023 06:39:14.632 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:39:14.633 * REPLICAOF 10.13.0.27:6379 enabled (user request from 'id=10513061 addr=10.13.0.28:49898 laddr=10.13.0.28:6379 fd=7 name=sentinel-6d24eb3f-cmd age=92 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=358 qbuf-free=20116 argv-mem=4 multi-mem=178 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 tot-mem=23598 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:39:14.662 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:39:14.663 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:39:14.664 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:39:14.666 * Trying a partial resynchronization (request 8693844ab2bba2ba427baf357c179cf75f4d407c:39044392211).
434:S 02 Aug 2023 06:39:14.667 * Successful partial resynchronization with master.
434:S 02 Aug 2023 06:39:14.667 # Master replication ID changed to bb00b5860578b65794fdd441ee1051434dd1b712
434:S 02 Aug 2023 06:39:14.668 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
434:S 02 Aug 2023 06:40:06.125 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:40:06.172 * Background saving started by pid 1013232
1013232:C 02 Aug 2023 06:40:18.441 * DB saved on disk
1013232:C 02 Aug 2023 06:40:18.475 * Fork CoW for RDB: current 20 MB, peak 20 MB, average 11 MB
434:S 02 Aug 2023 06:40:18.583 * Background saving terminated with success
434:S 02 Aug 2023 06:41:19.097 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:41:19.143 * Background saving started by pid 1013239
1013239:C 02 Aug 2023 06:41:30.209 * DB saved on disk
1013239:C 02 Aug 2023 06:41:30.245 * Fork CoW for RDB: current 5 MB, peak 5 MB, average 3 MB
434:S 02 Aug 2023 06:41:30.342 * Background saving terminated with success
434:S 02 Aug 2023 06:42:31.002 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:42:31.048 * Background saving started by pid 1013264
1013264:C 02 Aug 2023 06:42:42.107 * DB saved on disk
1013264:C 02 Aug 2023 06:42:42.160 * Fork CoW for RDB: current 14 MB, peak 14 MB, average 9 MB
434:S 02 Aug 2023 06:42:42.244 * Background saving terminated with success
434:M 02 Aug 2023 06:43:19.429 # Connection with master lost.
434:M 02 Aug 2023 06:43:19.429 * Caching the disconnected master state.
434:S 02 Aug 2023 06:43:19.429 * Connecting to MASTER 10.13.0.29:6379
434:S 02 Aug 2023 06:43:19.430 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:43:19.430 * REPLICAOF 10.13.0.29:6379 enabled (user request from 'id=10513683 addr=10.13.0.28:36550 laddr=10.13.0.28:6379 fd=7 name=sentinel-6d24eb3f-cmd age=143 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=344 qbuf-free=20130 argv-mem=4 multi-mem=178 rbs=4096 rbp=45 obl=45 oll=0 omem=0 tot-mem=25646 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:43:19.452 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:43:19.453 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:43:19.454 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:43:19.456 * Trying a partial resynchronization (request bb00b5860578b65794fdd441ee1051434dd1b712:39049943612).
434:S 02 Aug 2023 06:43:24.289 * Full resync from master: d9a370c0925d8da5aead542c748b42f1c02fec73:39050680157
434:S 02 Aug 2023 06:43:24.398 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
434:S 02 Aug 2023 06:43:36.277 * Discarding previously cached master state.
434:S 02 Aug 2023 06:43:36.277 * MASTER <-> REPLICA sync: Flushing old data
434:S 02 Aug 2023 06:43:40.030 * MASTER <-> REPLICA sync: Loading DB in memory
434:S 02 Aug 2023 06:43:40.049 * Loading RDB produced by version 7.0.11
434:S 02 Aug 2023 06:43:40.049 * RDB age 16 seconds
434:S 02 Aug 2023 06:43:40.049 * RDB memory usage when created 1010.90 Mb
434:S 02 Aug 2023 06:43:47.727 * Done loading RDB, keys loaded: 1133405, keys expired: 0.
434:S 02 Aug 2023 06:43:47.728 * MASTER <-> REPLICA sync: Finished with success
434:S 02 Aug 2023 06:44:44.545 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:44:44.627 * Background saving started by pid 1013312
1013312:C 02 Aug 2023 06:44:55.346 * DB saved on disk
1013312:C 02 Aug 2023 06:44:55.383 * Fork CoW for RDB: current 14 MB, peak 14 MB, average 10 MB
434:S 02 Aug 2023 06:44:55.477 * Background saving terminated with success
434:S 02 Aug 2023 06:45:56.091 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:45:56.142 * Background saving started by pid 1013317
1013317:C 02 Aug 2023 06:46:08.767 * DB saved on disk
1013317:C 02 Aug 2023 06:46:08.827 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 2 MB
434:S 02 Aug 2023 06:46:08.944 * Background saving terminated with success
434:S 02 Aug 2023 06:47:21.986 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:47:22.019 * Background saving started by pid 1013344
1013344:C 02 Aug 2023 06:47:32.733 * DB saved on disk
1013344:C 02 Aug 2023 06:47:32.789 * Fork CoW for RDB: current 32 MB, peak 32 MB, average 27 MB
434:S 02 Aug 2023 06:47:32.914 * Background saving terminated with success
434:S 02 Aug 2023 06:48:33.050 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:28:03.328 * Background saving terminated with success
434:M 02 Aug 2023 06:28:36.368 # Connection with master lost.
434:M 02 Aug 2023 06:28:36.368 * Caching the disconnected master state.
434:S 02 Aug 2023 06:28:36.369 * Connecting to MASTER 10.13.0.27:6379
434:S 02 Aug 2023 06:28:36.369 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:28:36.369 * REPLICAOF 10.13.0.27:6379 enabled (user request from 'id=10509472 addr=10.13.0.27:39224 laddr=10.13.0.28:6379 fd=12 name=sentinel-44d0c53a-cmd age=607 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=344 qbuf-free=20130 argv-mem=4 multi-mem=178 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 tot-mem=23598 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:28:36.392 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:28:36.393 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:28:36.394 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:28:36.396 * Trying a partial resynchronization (request 0bda6e46ec3e27e264c5e7ac6394239071161f6e:39027397143).
434:S 02 Aug 2023 06:28:36.397 * Successful partial resynchronization with master.
434:S 02 Aug 2023 06:28:36.397 # Master replication ID changed to 5d0a0d69dafc3e5225cadf78b3a3ece6c1cf5b7c
434:S 02 Aug 2023 06:28:36.398 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
434:S 02 Aug 2023 06:30:12.850 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:30:12.894 * Background saving started by pid 1013054
1013054:C 02 Aug 2023 06:30:25.627 * DB saved on disk
1013054:C 02 Aug 2023 06:30:25.664 * Fork CoW for RDB: current 27 MB, peak 27 MB, average 17 MB
434:S 02 Aug 2023 06:30:25.810 * Background saving terminated with success
434:S 02 Aug 2023 06:31:26.020 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:31:26.072 * Background saving started by pid 1013079
434:M 02 Aug 2023 06:31:27.411 # Connection with master lost.
434:M 02 Aug 2023 06:31:27.411 * Caching the disconnected master state.
434:M 02 Aug 2023 06:31:27.411 * Discarding previously cached master state.
434:M 02 Aug 2023 06:31:27.411 # Setting secondary replication ID to 5d0a0d69dafc3e5225cadf78b3a3ece6c1cf5b7c, valid up to offset: 39031081016. New replication ID is 6c80e9b301191e2866ec410ae9352a615694d217
434:M 02 Aug 2023 06:31:27.411 * MASTER MODE enabled (user request from 'id=10511284 addr=10.13.0.28:60236 laddr=10.13.0.28:6379 fd=10 name=sentinel-6d24eb3f-cmd age=171 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=188 qbuf-free=20286 argv-mem=4 multi-mem=169 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 tot-mem=23589 events=r cmd=exec user=default redir=-1 resp=2')
434:M 02 Aug 2023 06:31:27.454 # CONFIG REWRITE executed with success.
1013079:C 02 Aug 2023 06:31:37.920 * DB saved on disk
1013079:C 02 Aug 2023 06:31:37.985 * Fork CoW for RDB: current 13 MB, peak 13 MB, average 9 MB
434:M 02 Aug 2023 06:31:38.168 * Background saving terminated with success
434:M 02 Aug 2023 06:31:41.195 * Replica 10.13.0.29:6379 asks for synchronization
434:M 02 Aug 2023 06:31:41.196 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '0bda6e46ec3e27e264c5e7ac6394239071161f6e', my replication IDs are '6c80e9b301191e2866ec410ae9352a615694d217' and '5d0a0d69dafc3e5225cadf78b3a3ece6c1cf5b7c')
434:M 02 Aug 2023 06:31:41.196 * Delay next BGSAVE for diskless SYNC
434:M 02 Aug 2023 06:31:46.849 * Starting BGSAVE for SYNC with target: replicas sockets
434:M 02 Aug 2023 06:31:46.901 * Background RDB transfer started by pid 1013080
1013080:C 02 Aug 2023 06:31:58.838 * Fork CoW for RDB: current 21 MB, peak 21 MB, average 15 MB
434:M 02 Aug 2023 06:31:58.838 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
434:M 02 Aug 2023 06:31:58.977 * Background RDB transfer terminated with success
434:M 02 Aug 2023 06:31:58.977 * Streamed RDB transfer with replica 10.13.0.29:6379 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming
434:M 02 Aug 2023 06:31:58.977 * Synchronization with replica 10.13.0.29:6379 succeeded
434:M 02 Aug 2023 06:32:49.274 * 10000 changes in 60 seconds. Saving...
434:M 02 Aug 2023 06:32:49.324 * Background saving started by pid 1013104
1013104:C 02 Aug 2023 06:33:02.737 * DB saved on disk
1013104:C 02 Aug 2023 06:33:02.771 * Fork CoW for RDB: current 12 MB, peak 12 MB, average 8 MB
434:M 02 Aug 2023 06:33:02.865 * Background saving terminated with success
434:M 02 Aug 2023 06:37:31.374 * 10000 changes in 60 seconds. Saving...
434:M 02 Aug 2023 06:37:31.406 * Background saving started by pid 1013183
434:M 02 Aug 2023 06:37:31.408 # Connection with replica client id #10511890 lost.
434:S 02 Aug 2023 06:37:42.409 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
434:S 02 Aug 2023 06:37:42.409 * Connecting to MASTER 10.13.0.29:6379
434:S 02 Aug 2023 06:37:42.409 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:37:42.409 * REPLICAOF 10.13.0.29:6379 enabled (user request from 'id=10513017 addr=10.13.0.27:46996 laddr=10.13.0.28:6379 fd=8 name=sentinel-44d0c53a-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=198 qbuf-free=20276 argv-mem=4 multi-mem=178 rbs=1024 rbp=1024 obl=45 oll=0 omem=0 tot-mem=22574 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:37:42.423 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:37:42.423 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:37:42.424 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:37:42.425 * Trying a partial resynchronization (request 6c80e9b301191e2866ec410ae9352a615694d217:39039868221).
1013183:C 02 Aug 2023 06:37:44.353 * DB saved on disk
1013183:C 02 Aug 2023 06:37:44.389 * Fork CoW for RDB: current 70 MB, peak 70 MB, average 56 MB
434:S 02 Aug 2023 06:37:44.493 * Background saving terminated with success
434:S 02 Aug 2023 06:37:47.360 * Full resync from master: 8693844ab2bba2ba427baf357c179cf75f4d407c:39042175434
434:S 02 Aug 2023 06:37:47.452 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
434:S 02 Aug 2023 06:37:58.782 * Discarding previously cached master state.
434:S 02 Aug 2023 06:37:58.783 * MASTER <-> REPLICA sync: Flushing old data
434:S 02 Aug 2023 06:38:04.438 * MASTER <-> REPLICA sync: Loading DB in memory
434:S 02 Aug 2023 06:38:04.450 * Loading RDB produced by version 7.0.11
434:S 02 Aug 2023 06:38:04.450 * RDB age 17 seconds
434:S 02 Aug 2023 06:38:04.450 * RDB memory usage when created 1017.19 Mb
434:S 02 Aug 2023 06:38:12.274 * Done loading RDB, keys loaded: 1136219, keys expired: 0.
434:S 02 Aug 2023 06:38:12.274 * MASTER <-> REPLICA sync: Finished with success
434:S 02 Aug 2023 06:38:45.080 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:38:45.133 * Background saving started by pid 1013207
1013207:C 02 Aug 2023 06:38:56.044 * DB saved on disk
1013207:C 02 Aug 2023 06:38:56.078 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 2 MB
434:S 02 Aug 2023 06:38:56.212 * Background saving terminated with success
434:M 02 Aug 2023 06:39:14.632 # Connection with master lost.
434:M 02 Aug 2023 06:39:14.632 * Caching the disconnected master state.
434:S 02 Aug 2023 06:39:14.632 * Connecting to MASTER 10.13.0.27:6379
434:S 02 Aug 2023 06:39:14.632 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:39:14.633 * REPLICAOF 10.13.0.27:6379 enabled (user request from 'id=10513061 addr=10.13.0.28:49898 laddr=10.13.0.28:6379 fd=7 name=sentinel-6d24eb3f-cmd age=92 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=358 qbuf-free=20116 argv-mem=4 multi-mem=178 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 tot-mem=23598 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:39:14.662 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:39:14.663 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:39:14.664 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:39:14.666 * Trying a partial resynchronization (request 8693844ab2bba2ba427baf357c179cf75f4d407c:39044392211).
434:S 02 Aug 2023 06:39:14.667 * Successful partial resynchronization with master.
434:S 02 Aug 2023 06:39:14.667 # Master replication ID changed to bb00b5860578b65794fdd441ee1051434dd1b712
434:S 02 Aug 2023 06:39:14.668 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
434:S 02 Aug 2023 06:40:06.125 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:40:06.172 * Background saving started by pid 1013232
1013232:C 02 Aug 2023 06:40:18.441 * DB saved on disk
1013232:C 02 Aug 2023 06:40:18.475 * Fork CoW for RDB: current 20 MB, peak 20 MB, average 11 MB
434:S 02 Aug 2023 06:40:18.583 * Background saving terminated with success
434:S 02 Aug 2023 06:41:19.097 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:41:19.143 * Background saving started by pid 1013239
1013239:C 02 Aug 2023 06:41:30.209 * DB saved on disk
1013239:C 02 Aug 2023 06:41:30.245 * Fork CoW for RDB: current 5 MB, peak 5 MB, average 3 MB
434:S 02 Aug 2023 06:41:30.342 * Background saving terminated with success
434:S 02 Aug 2023 06:42:31.002 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:42:31.048 * Background saving started by pid 1013264
1013264:C 02 Aug 2023 06:42:42.107 * DB saved on disk
1013264:C 02 Aug 2023 06:42:42.160 * Fork CoW for RDB: current 14 MB, peak 14 MB, average 9 MB
434:S 02 Aug 2023 06:42:42.244 * Background saving terminated with success
434:M 02 Aug 2023 06:43:19.429 # Connection with master lost.
434:M 02 Aug 2023 06:43:19.429 * Caching the disconnected master state.
434:S 02 Aug 2023 06:43:19.429 * Connecting to MASTER 10.13.0.29:6379
434:S 02 Aug 2023 06:43:19.430 * MASTER <-> REPLICA sync started
434:S 02 Aug 2023 06:43:19.430 * REPLICAOF 10.13.0.29:6379 enabled (user request from 'id=10513683 addr=10.13.0.28:36550 laddr=10.13.0.28:6379 fd=7 name=sentinel-6d24eb3f-cmd age=143 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 qbuf=344 qbuf-free=20130 argv-mem=4 multi-mem=178 rbs=4096 rbp=45 obl=45 oll=0 omem=0 tot-mem=25646 events=r cmd=exec user=default redir=-1 resp=2')
434:S 02 Aug 2023 06:43:19.452 # CONFIG REWRITE executed with success.
434:S 02 Aug 2023 06:43:19.453 * Non blocking connect for SYNC fired the event.
434:S 02 Aug 2023 06:43:19.454 * Master replied to PING, replication can continue...
434:S 02 Aug 2023 06:43:19.456 * Trying a partial resynchronization (request bb00b5860578b65794fdd441ee1051434dd1b712:39049943612).
434:S 02 Aug 2023 06:43:24.289 * Full resync from master: d9a370c0925d8da5aead542c748b42f1c02fec73:39050680157
434:S 02 Aug 2023 06:43:24.398 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
434:S 02 Aug 2023 06:43:36.277 * Discarding previously cached master state.
434:S 02 Aug 2023 06:43:36.277 * MASTER <-> REPLICA sync: Flushing old data
434:S 02 Aug 2023 06:43:40.030 * MASTER <-> REPLICA sync: Loading DB in memory
434:S 02 Aug 2023 06:43:40.049 * Loading RDB produced by version 7.0.11
434:S 02 Aug 2023 06:43:40.049 * RDB age 16 seconds
434:S 02 Aug 2023 06:43:40.049 * RDB memory usage when created 1010.90 Mb
434:S 02 Aug 2023 06:43:47.727 * Done loading RDB, keys loaded: 1133405, keys expired: 0.
434:S 02 Aug 2023 06:43:47.728 * MASTER <-> REPLICA sync: Finished with success
434:S 02 Aug 2023 06:44:44.545 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:44:44.627 * Background saving started by pid 1013312
1013312:C 02 Aug 2023 06:44:55.346 * DB saved on disk
1013312:C 02 Aug 2023 06:44:55.383 * Fork CoW for RDB: current 14 MB, peak 14 MB, average 10 MB
434:S 02 Aug 2023 06:44:55.477 * Background saving terminated with success
434:S 02 Aug 2023 06:45:56.091 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:45:56.142 * Background saving started by pid 1013317
1013317:C 02 Aug 2023 06:46:08.767 * DB saved on disk
1013317:C 02 Aug 2023 06:46:08.827 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 2 MB
434:S 02 Aug 2023 06:46:08.944 * Background saving terminated with success
434:S 02 Aug 2023 06:47:21.986 * 10000 changes in 60 seconds. Saving...
434:S 02 Aug 2023 06:47:22.019 * Background saving started by pid 1013344
1013344:C 02 Aug 2023 06:47:32.733 * DB saved on disk
1013344:C 02 Aug 2023 06:47:32.789 * Fork CoW for RDB: current 32 MB, peak 32 MB, average 27 MB
434:S 02 Aug 2023 06:47:32.914 * Background saving terminated with success
434:S 02 Aug 2023 06:48:33.050 * 10000 changes in 60 seconds. Saving...
Reply all
Reply to author
Forward
0 new messages