Multiple nodes writing to the same append only file

[David Dikman]

Hi everyone!

This is my first post here so I do apologise if I get the format wrong or should turn elsewhere.

I've inherited a Redis cluster with 3 servers having 3 master nodes and 2 slaves on each server for the other masters. We use both rdb and append only file, each node write to a separate rdb file but they share the same append only file.

This seems wrong to me, I think that would mean that on restart each node would read all data as opposed to the 1/3 it needs. The reason we found this is that we're hunting performance issues and amongst other things we found this setting and I was thinking maybe it leads to write contention on the append only file (we're also looking into decreasing reads to the rdb file which is another issue).

I've searched around but I can't find anything that either says you can or you can't share the same append file for different nodes although in all examples I find nodes are using their own append only files.

My simple question is (though maybe the answer isn't as simple), can I share the same append only file or is this invalid configuration?

Best regards,

David

[Itamar Haber]

Hi David and welcome - no apologies needed and helping is what this list is about,

What you're describing is an invalid configuration if I've ever heard of one :) All persistency files, rdb and aof alike, that Redis generates are meant to be per process (whether master or slave) and not shared with others.

Your suspicions are very much in place and from the sound of it you've not only inherited the cluster, but also a little tangle to unknot. That said, once you've reconfigured the instances (can be done during runtime with `CONFIG SET` and `CONFIG REWRITE`) and the aof files have been rewritten (see `BGREWRITEAOF`) you should be ok... in that perspective at least.

Stay vigilant and good luck,

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--

Itamar Haber | Technology Evangelist
Redis Labs ~/redis

Mobile: +972 (54) 567 9692
Twitter: @itamarhaber
Skype: itamar.haber

[hva...@gmail.com]

Welcome!

You're describing an invalid configuration.  It's possible in theory for multiple processes to write to the same file and not corrupt the writes from the other processes, but the software would have to be written with this configuration in mind, and thoroughly tested.  I'm not part of Redis Labs, but I'm confident that no such design or testing has been done with the Redis server code.  So I see it as a case of data corruption that's waiting to happen.  Or has already happened, but not yet noticed because your organization hasn't needed to restore from one of these AOF files yet.

[David Dikman]

Hi!

Thank you very much for the answers! Then I'm not going mad :) (in that respect at least)

And sorry for the late answer, I must've messed up the notifications on this thread.

We're going to move over to just using rdb files in place of the aof for performance reasons so I hope we can leave this issue behind :)

Again thank you very much.

Best regards,

David