Redis 4.0 partial synchronization not working

[George Chilumbu]

We are currently running redis 4.0.7, and when we stopped the server for maintenance, and later restarted the server, we noticed that it was doing a full sync to the master instead of the documented partial (incremental) sync. Below is our log info:

21285:S 07 Feb 15:34:13.927 * MASTER <-> SLAVE sync started
21285:S 07 Feb 15:34:13.927 * Non blocking connect for SYNC fired the event.
21285:S 07 Feb 15:34:13.928 * Master replied to PING, replication can continue...
21285:S 07 Feb 15:34:13.928 * Trying a partial resynchronization (request 8e140967bf3de9874ea45acfc8aed1270310a486:458034995495).
21285:S 07 Feb 15:34:14.163 * Full resync from master: 8e140967bf3de9874ea45acfc8aed1270310a486:474431691184
21285:S 07 Feb 15:34:14.164 * Discarding previously cached master state.
21285:S 07 Feb 15:36:05.510 * MASTER <-> SLAVE sync: receiving 5818151093 bytes from master
21285:S 07 Feb 15:37:14.281 * MASTER <-> SLAVE sync: Flushing old data
21285:S 07 Feb 15:37:57.365 * MASTER <-> SLAVE sync: Loading DB in memory
21285:S 07 Feb 15:38:54.950 * MASTER <-> SLAVE sync: Finished with success

[Salvatore Sanfilippo]

Hello, you should please post the logs on the master side to see why
it was not possible for the master to just send an incremental
payload. Thanks.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.



--
Salvatore 'antirez' Sanfilippo
open source developer - Redis Labs https://redislabs.com

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.

[George Chilumbu]

Below is the log from the master:

439:M 07 Feb 15:30:06.567 # Connection with slave client id #3987733 lost.

439:M 07 Feb 15:34:13.928 * Slave 10.1.9.100:6379 asks for synchronization

439:M 07 Feb 15:34:13.928 * Unable to partial resync with slave 10.1.9.100:6379 for lack of backlog (Slave request was: 458034995495).

439:M 07 Feb 15:34:13.928 * Starting BGSAVE for SYNC with target: disk

439:M 07 Feb 15:34:14.163 * Background saving started by pid 5040

5040:C 07 Feb 15:36:05.096 * DB saved on disk

5040:C 07 Feb 15:36:05.234 * RDB: 0 MB of memory used by copy-on-write

439:M 07 Feb 15:36:05.509 * Background saving terminated with success

439:M 07 Feb 15:37:13.533 * Synchronization with slave 10.1.9.100:6379 succeeded

> email to redis-db+unsubscribe@googlegroups.com.

> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

--
Salvatore 'antirez' Sanfilippo
open source developer - Redis Labs https://redislabs.com

"If a system is to have conceptual integrity, someone must control the
concepts."
       — Fred Brooks, "The Mythical Man-Month", 1975.

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/cJ5o6F1_Ys0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+unsubscribe@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--

Purpose is the key to success. You can easily be successful in the wrong things. 

[Salvatore Sanfilippo]

The relevant part is:

439:M 07 Feb 15:34:13.928 * Unable to partial resync with slave
10.1.9.100:6379 for lack of backlog (Slave request was: 458034995495).

Edit your redis.conf and give it a bit more backlog, and the next time
you should have more luck.
By default we use a very small backlog because it uses memory and we
are not sure about what are the user requirements.

On Wed, Feb 7, 2018 at 9:30 AM, George Chilumbu

>> > email to redis-db+u...@googlegroups.com.

>> > To post to this group, send email to redi...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/redis-db.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Salvatore 'antirez' Sanfilippo
>> open source developer - Redis Labs https://redislabs.com
>>
>> "If a system is to have conceptual integrity, someone must control the
>> concepts."
>> — Fred Brooks, "The Mythical Man-Month", 1975.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Redis DB" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/redis-db/cJ5o6F1_Ys0/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> redis-db+u...@googlegroups.com.

>> To post to this group, send email to redi...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/redis-db.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Purpose is the key to success. You can easily be successful in the wrong
> things.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to redis-db+u...@googlegroups.com.

[喬治吉倫布]

Do you mean the repl-backlog-size or the tcp-backlog? I am guessing its the repl-backlog-size since we are dealing with replication and memory here am i right?

Messages in this email and any attachments hereof are confidential, intended solely for the use of the named recipient(s) above and may contain confidential information and/or privileged communications and/or otherwise exempt from disclosure under applicable law. Any unauthorized review, use, copying, disclosure or dissemination of any information contained in or attached to this email is STRICTLY PROHIBITED. If you are not the intended recipient(s), or have received this email in error, please immediately notify the sender by email or telephone and delete this email and any attached documents from your computer and mail server. Receipt by anyone other than the intended recipient(s) is not a waiver of confidentiality and/or any legal privilege. Thanks for your cooperation.

本郵件所載之訊息及其附件係屬機密且僅供本郵件指定之收件人使用，其內容可能包含本公司之機密資訊、特權溝通及/或法令豁免揭露之內容，嚴格禁止任何未經授權而參閱、使用、複製、揭露或散布本郵件資訊或附件之行為。若您非本郵件寄件人指定之收件者或誤收本郵件，請立即以郵件或電話通知寄件人，並自您電腦及郵件伺服器中完全刪除本郵件及相關附件。本郵件倘有誤寄之情事並非代表本公司拋棄保密之主張及/或法律上特權。謝謝您的配合。

[George Chilumbu]

I changed the  repl-backlog-size from 64mb to 512, but now i get the message below which is different from the previous message the directly pointed at the backlog bot set high enough:

5824:M 07 Feb 23:54:41.088 * Partial resynchronization not accepted: Requested offset for second ID was 474439652356, but I can reply up to 474439649725

Which value 474496... is it referring to? How do i tune this value?

[George Chilumbu]

I was doing the upgrade and it seems that the partial synch only works if all nodes are using 4.0, or at at least the slave and the master involved in the failover are both using 4.0.