Basic questions about slave replication

[Dbz Fan]

When a slave connects to master for first time then always full synchronization is performed?

During full synchronization fork() is performed .How much RAM spike should I expect (in full synchronization) in worst case scenario? Also what is the disk space used by the RDB file created on  full synchronization.
How is the RDB file created shared between multiple slaves.?

Now coming to partial replication.
How to estimate the "repl-backlog-size" one should have? I know the rate at which i will be receiving commands but how much space will an individual write command occupy in replication backlog buffer?

What happens if during partial synchronization the repl-backlog-size is exceeded ? Will redis go for full synchronization in this case?

[Dbz Fan]

One more query in addition to the above is the RDB file deleted from master once trasmitted to slave because if there are multiple full synchronizations to be performed then no. of rdb files will keep on increasing. Also is the RDB file on slave side deleted after slave takes it in memory from disk?

[Josiah Carlson]

Replies inline.

On Sun, May 3, 2015 at 4:36 AM, Dbz Fan <manass...@gmail.com> wrote:

One more query in addition to the above is the RDB file deleted from master once trasmitted to slave because if there are multiple full synchronizations to be performed then no. of rdb files will keep on increasing. Also is the RDB file on slave side deleted after slave takes it in memory from disk?

If operating properly, the master (and slaves) should have 1-2 RDB snapshots at any one time: 1) the most recently completed and finished snapshot, 2) a snapshot being created as part of a SAVE, BGSAVE, SHUTDOWN, or SYNC.

After the new snapshot is finished being created, Redis renames the new snapshot over the old snapshot name, deleting the old snapshot, leaving one.

On Sunday, May 3, 2015 at 4:55:02 PM UTC+5:30, Dbz Fan wrote:

When a slave connects to master for first time then always full synchronization is performed?

Yes.

During full synchronization fork() is performed .How much RAM spike should I expect (in full synchronization) in worst case scenario?

2x memory, this is in the documentation. But Redis will tell you how much additional memory was used as part of the logging output.

 

Also what is the disk space used by the RDB file created on  full synchronization.

It depends on how much data is stored in Redis. Generally speaking, the on-disk snapshot size will be roughly 5-20% of the in-memory data size (RSS).

 

How is the RDB file created shared between multiple slaves.?

It is transmitted from the master, but each slave may get a different snapshot at a different time. But as long as all slaves are caught up with the master, they should have the same data.

Now coming to partial replication.
How to estimate the "repl-backlog-size" one should have?

1. Know how many write commands you are sending and how much data they take up

2, Know how long you want to be able to support a partial resync on connection

3. (write bytes per second) * (seconds to allow for a partial resync) = (repl-backlog-size)

 

I know the rate at which i will be receiving commands but how much space will an individual write command occupy in replication backlog buffer?

Exactly the size of the whole command with arguments on the wire. Redis stores the entire command as-is in the backlog buffer, just like the AOF, and just like the replication stream.

What happens if during partial synchronization the repl-backlog-size is exceeded ?

If the backlog is exceeded while waiting for a slave to reconnect, the backlog will be flushed, and that slave can't use the partial resync until after it has done a full sync.

 

Will redis go for full synchronization in this case?

On that slave, yes. But if you have a 2nd slave that disconnects and reconnects before the buffer is exceeded again, then that slave should be able to reconnect without a partial resync (but won't make the first slave be able to do so).

 - Josiah

[Greg Andrews]

During full synchronization fork() is performed .How much RAM spike should I expect (in full synchronization) in worst case scenario?

2x memory, this is in the documentation. But Redis will tell you how much additional memory was used as part of the logging output.

2x is the generally accepted figure, but there have been reports of higher consumption.  The factors that produce the growth don't really limit it to 2x:

• The time it takes to complete the four stages of a full synchronization
  
• The rate of writes to the master during the synchronization
  

The sync sequence is a four stage process:

1. Master forks a child process that writes a snapshot of the database to local disk
2. Master reads the snapshot from disk, transfers it through network to slave. which writes the snapshot to its local disk
3. Slave reads the snapshot from disk
4. Master transfers to the slave the new writes that were received during stages 1-3 and buffered in RAM
   

Visualize what's going on in each stage and you'll see that the write and read performance of the disk on the master and the slave are important, as is the network between the master and slave.  Bottlenecks in these places will make the sync process take more time.

On the master, Redis uses copy-on-write memory management when it forks a child process so the act of writing to disk doesn't block the writes (or reads) coming in from the client.  However, this means writes will cause RAM growth as the memory blocks for the written keys are copied.  (the child process keeps the original memory block unchanged so it can be saved to disk, and the master has a new memory block with the changed data)  I don't know when that child process exits and frees up its copy of the memory - it might stay running for all four stages, or it might exit sooner.

If the writes that come in during the sync only change existing keys (e.g., increment counters), then the worst case scenario is 2x the ram consumption.  However, if writes add new keys or add data to existing keys (e.g., add new elements to JSON data), then it's possible to go beyond 2x.

My own recommendation for a Redis master server is to keep the total RAM 3x the size of the Redis consumption.  Or more.  That allows for more growth than the customary 2x without negative effects, and allows the OS to buffer the disk files during the reads and writes in Stage 1 and 2.

[Dbz Fan]

I was observing the used_memory_rss and used_memory during full synchronization .There is no change in both these parameters (my repl-backlog-buffer is intentionally set to 1bytes to observe any RAM spike just due to fork()).However the master log say the following:

Slave 10.55.yyy.zz:6379 asks for synchronization

[7884] 04 May 20:47:43.498 * Full resync requested by slave 10.55.yyy.zz:6379

[7884] 04 May 20:47:43.498 * Starting BGSAVE for SYNC with target: disk

[7884] 04 May 20:47:43.500 * Background saving started by pid 8066

[7884] 04 May 20:47:43.696 - DB 0: 556868 keys (556868 volatile) in 1048576 slots HT.

[7884] 04 May 20:47:43.696 - 5 clients connected (1 slaves), 222635320 bytes in use

[8066] 04 May 20:47:45.568 * DB saved on disk

[8066] 04 May 20:47:45.568 * RDB: 222 MB of memory used by copy-on-write

[7884] 04 May 20:47:45.608 * Background saving terminated with success

[7884] 04 May 20:47:46.819 * Synchronization with slave 10.55.yyy.zz:6379 succeeded

Does this mean the fork() operation during full synchronization is not consuming any memory?

 also what does this particular line mean RDB: 222 MB of memory used by copy-on-write .
What is the 222MB stated here(my used_memory_rss is close to 227MB has it go to do anything with that or is it the extra RAM consumed  during fork() which wasnt reported in used_memory or used_memory_rss or anything else?).

[Greg Andrews]

The output of the INFO command (after the sync has been performed) has one or two lines that show stats related to the memory used by the fork.  That might be useful to look at.

The previous two lines from the log you posted may also help explain:

[7884] 04 May 20:47:43.696 - 5 clients connected (1 slaves), 222635320 bytes in use

[8066] 04 May 20:47:45.568 * DB saved on disk

[8066] 04 May 20:47:45.568 * RDB: 222 MB of memory used by copy-on-write

The "222635320 bytes in use" seems to match the 222 MB reported for copy-on-write.  The process ID at the start of the line indicates the two lines that mention memory use came from different processes.  Likely the parent and child.  Since the elapsed time is only a couple of seconds and you haven't mentioned that your clients write/update at the rate of 100MB/sec, It doesn't appear that there was significant memory growth during the sync.  The message "RDB: 222 MB of memory used by copy-on-write" is just the child process reporting its memory size before exiting.

If you want to know the source of the additional 5MB reported in used_memory_rss (227MB vs. 222MB), well perhaps there's a line in the INFO output that will show it.  It's less than 1% of the total, so it may be hard to find the precise explanation.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Dbz Fan]

Is the 2x memory stated calculated simply as parent + child memory used.

As per my logs 222 mb is memory used by parent and again another 222 mb by child process
Basically 2x memory is seemingly applicable here ?

I am assuming that replication backlog size has to be accounted seperately and not to be included in that 2x factor
Does it seem right?

Also what is the life span of the child process?

[Josiah Carlson]

Small correction to Greg. The line that reads:

[8066] 04 May 20:47:45.568 * RDB: 222 MB of memory used by copy-on-write

Tells you how much *additional* non-shared memory was used by the child process to create the snapshot. So in this case, it looks a lot like *something* is touching all of the memory and making the fork() shared memory unshared, by writing to it. Which version of Redis are you running? Older Redis versions would manipulate reference counts during SYNC/BGSAVE/BGREWRITEAOF, which would result in the behavior you are seeing.

Replication backlog, client backlog, incoming command buffers, ... should all be accounted for separately compared to the 2x worst-case fork() overhead. But assuming you are running the most recent edition of Redis (or at lest 2.8.19), and haven't set your backlogs to be large (relative to your data size), the biggest factor will generally be the amount of extra memory used during SYNC/BGSAVE/BGREWRITEAOF forks. The exception comes if/when you have huge amounts of data and high write volumes, then the replication backlog can take up more memory than the child process.

The child process lasts long enough to create the new snapshot and overwrite the old snapshot. This time varies depending on how much data you have, disk IO speed, etc.

 - Josiah

[Dbz Fan]

Yes I am using 2.8.19 version.
So Josiah by unshared memory do you mean that the child process memory is not accounted in 2x factor right?

If so then in general how much RAM would the child process consume?

And again where can I exactly see the Ram spike in master caused due to fork

[Josiah Carlson]

The 2x that is listed in the docs is *because* of the forked child process that could eventually not share any memory and result in 2x memory utilization. More specifically, the 2x accounts for the 222635320 in the line below (really the used_memory_rss from INFO), plus the 222MB in the other line below:

[7884] 04 May 20:47:43.696 - 5 clients connected (1 slaves), 222635320 bytes in use

[8066] 04 May 20:47:45.568 * RDB: 222 MB of memory used by copy-on-write

Generally speaking, child process memory usage during the snapshot operation should be at most the used_memory_rss from the parent process' INFO output. But parent process memory utilization may go up due to various buffers, additional data writes, etc.

I don't know what commands or scripts you are running, or what your data looks like, but for one reason or another, a fairly large subset of your data in memory is being changed in the 2 seconds it takes to snapshot your dataset.

You won't see a ram spike in the main Redis server master process itself (unless your commands are coming in fast enough to cause replication buffers to grow really fast, but then that is additional memory growth), but you *may* observe it being used by the forked child if your snapshot operations take long enough. But because it only takes 2 seconds for Redis to snapshot your dataset, the extra 222 MB of memory used by the forked child is reclaimed almost immediately, and your monitoring is unlikely to observe it (unless you are sitting in top refreshing constantly to see the child process appear, grow in memory, then disappear - all in 2 seconds).

 - Josiah

[Dbz Fan]

If mutiple slaves say 'N' no. of slaves try to reconnect to redis master parallely then will 'N' forks be performed ,resulting in N child processes ?

[Dbz Fan]

Ok i tried reconnecting multiple slaves(specifically 2 slaves) parallely(less than a second of difference) and in master log i see only single child process id performing BGSAVE. 
What can be the reason for this?
Here are the logs :

[15261] 13 May 16:50:47.395 - Accepted 10.zz.yyy.33:46379

[15261] 13 May 16:50:47.398 * Slave 10.zz.yyy.33:6379 asks for synchronization

[15261] 13 May 16:50:47.398 * Full resync requested by slave 10.zz.yyy.33:6379

[15261] 13 May 16:50:47.398 * Starting BGSAVE for SYNC with target: disk

[15261] 13 May 16:50:47.399 * Background saving started by pid 16337

[15261] 13 May 16:50:47.741 - Accepted 10.zz.yyy.39:44881

[15261] 13 May 16:50:47.746 * Slave10.zz.yyy.39:6379 asks for synchronization

[15261] 13 May 16:50:47.746 * Full resync requested by slave 10.zz.yyy.39:6379

[15261] 13 May 16:50:47.746 * Waiting for end of BGSAVE for SYNC

[16337] 13 May 16:50:48.600 * DB saved on disk

[16337] 13 May 16:50:48.600 * RDB: 134 MB of memory used by copy-on-write

[15261] 13 May 16:50:48.602 * Background saving terminated with success

[15261] 13 May 16:50:49.435 * Synchronization with slave 10.zz.yyy.33:6379 succeeded

[15261] 13 May 16:50:49.721 * Synchronization with slave 10.zz.yyy.33:6379 succeeded

[15261] 13 May 16:50:52.324 - DB 0: 329808 keys (329808 volatile) in 524288 slots HT.

[15261] 13 May 16:50:52.324 - 8 clients connected (2 slaves), 168504504 bytes in use

[Greg Andrews]

Dbz Fan <manass...@gmail.com> wrote:

Ok i tried reconnecting multiple slaves parallely(less than a second of difference) and in master log i see only single child process id performing BGSAVE. 

What can be the reason for this?

Since around version 2.4.15 (I don't have the exact version number in front of me), when multiple slaves request a full sync at the same time, the master starts just one background save process and keeps just one buffer of updates for it, which it delivers to those slaves that requested the sync at the same time.  Since those slaves essentially need the same data, it doesn't make sense to save duplicate snapshot files and keep duplicate buffers of updates for each slave.

If you delay the re-connection of one slave by the right amount of time, you'll force the master to fork a new child process to make a new snapshot.  You'll probably have to wait until the first slave is finished with the sync process.

  -Greg