Is it possible to make priority of events synchronization?

[Dmitriy Kaloshin]

Hi all,

 

it looks like nobody yet asked for a subject except one post Is replication strictly sequential? where Josiah Carlson answered YES.

 

To understand my point I have to present a small and simple example:

 

1. We have Master and Slave.

2. On Master we have large transaction of some data.

3. Then again we have small transaction of another data but very important one.

4. We expect to get it on the Slave but unfortunately due to #1 we have to wait until time when first data processes its synchronization.

5. If we have bad internet connection then sync time of second transaction becomes critical for us.

 

I made a small research using Windows port of Redis Source 2.8.17 (MSVS 2013) where I built a local environment with master and slave and played over in debugger to see how everything goes on.

 

At first side (May be I am wrong) I discovered that each change on master side fires creation of aeFileEvent and then it processes through method aeProcessEvents.

 

It looks like it is not so difficult to add another field with name Priority to struct aeFileEbent and at the same time to make some sorting inside of aeProcessEvents.

 

It would be very nice to have such feature as an option like

 

A. Probably priority will be switched off in config by default.

B. When it is switched on then set priority depending on DB. Like db0 has highest priority, db1 lower priority and so on.

C. Probably to create another command with name SETDBPRIORITY where user will configure DBs at rin time.

 

Does it make sense?

 

brgds

Dmitriy

 

 

[Josiah Carlson]

There is one single reason to not implement what you describe: data integrity.

Let's imagine there was the possibility for a user to say, "replicate this data first, if possible". Whether that is done with an additional command, using DB numbers (as you describe), or something else, it doesn't matter.

Given this semantic, I can always create a scenario where executing #1 then #2 results in a different system state than executing #2 then #1. Given that #1 was executed first on the master, if #2 was executed first on *any* slave, you would have data inconsistencies between the master and that slave. Given multiple slaves, there is a chance that some slaves would execute #2, #1 while others executed #1, #2 (this depends on replication speed).

All in all, I wouldn't want this feature in Redis due to the ease of breaking data integrity on accident.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Dmitriy Kaloshin]

 

Hi Josiah,

 

Thanks for Your reply.

 

And I think You are right and wrong at the same time. :)

 

Yes, I am agreed with You regarding sequence of data replication. Especially if we have data splitted between different databases  (db0, db1, db2 etc). but commited in a single transaction on some Master.

In this case if to imagine suggested feature is used and during replication from Master to Slave of db0 (high priority data as suggested) and db1 (more low priority) some other data of some other, more late transaction of db0 may easily come to Slave before data of first transaction for db1.

 

But... this is a question of correct data organization. Of course if I switch on this possible feature I must understand all consequences that may happen if I start to push data with the same importance into different databases with different priority. :)

 

I think nobody will do like this :)

 

I do my request because I want to put into some db only data of same priority. And of course replication sequence for same db will have the same order as it was commited on a Master side. Sorting should affect only sequences of different databases.

 

Again I remind You that by default such feature should be switched off. And I think if somebody decided to use SETDBPRIORITY and creates above mentioned incorrect scenario then appropriate error should be fired.

 

So could You please clear if such thing may be implemented by suggested way (my first mail)?

 

I am not so familar with Redis source and it will take much more time for me to answer this question.

 

Thanks in advance,
Dmitriy

пятница, 14 ноября 2014 г., 9:39:10 UTC+2 пользователь Dmitriy Kaloshin написал:

[Josiah Carlson]

On Mon, Nov 17, 2014 at 1:21 PM, Dmitriy Kaloshin <kaloshin...@gmail.com> wrote:

 

Hi Josiah,

 

Thanks for Your reply.

 

And I think You are right and wrong at the same time. :)

That happens sometimes, but I am not wrong here.

Yes, I am agreed with You regarding sequence of data replication. Especially if we have data splitted between different databases  (db0, db1, db2 etc). but commited in a single transaction on some Master.

In this case if to imagine suggested feature is used and during replication from Master to Slave of db0 (high priority data as suggested) and db1 (more low priority) some other data of some other, more late transaction of db0 may easily come to Slave before data of first transaction for db1.

The existence of MOVE means that any key in any DB can be moved to any other DB. And that means that any command that relies on keys (almost all of them) could have its meaning changed by altering the order of command replication and the resulting command execution order. Even worse, Lua scripts can arbitrarily choose databases during execution, allowing for effectively unlimited data manipulation within a Lua script involving keys from many DBs.

Prioritization of replication based on *any* criteria can lead to data corruption. Redis operates under the "let's not corrupt people's data if we can help it" approach, and being that this could corrupt data fairly easily with existing code that relies on MOVE and/or Lua scripts and involves key movement between databases, I don't believe it has a place in Redis.

But... this is a question of correct data organization. Of course if I switch on this possible feature I must understand all consequences that may happen if I start to push data with the same importance into different databases with different priority. :)

Having documentation is not sufficient. People don't read documentation, and they barely read the comments in the configuration file. How do I know? I have answered countless questions about the documentation and information sitting in the configuration file.

 

 I think nobody will do like this :)

Whether people like or don't like it, most people won't realize that having this feature could destroy data.

I do my request because I want to put into some db only data of same priority. And of course replication sequence for same db will have the same order as it was commited on a Master side. Sorting should affect only sequences of different databases.

But operations can involve multiple databases at the same time, which could result in some operations being replicated out of order with respect to the execution order in a given database. This is a problem.

Again I remind You that by default such feature should be switched off. And I think if somebody decided to use SETDBPRIORITY and creates above mentioned incorrect scenario then appropriate error should be fired.

I think something different: why introduce a feature that intends to purposefully break sequential execution guarantees for the benefit of replicating some "more important" commands at the cost of introducing data corruption edge cases?

To me, there is no win here. I *want* sequential execution. And despite what you believe, you *want* sequential execution too! You are just hung up on your slow commands. My advice is the same as before: make your slow commands faster, or at least write them to perform work small amounts at a time so that your replication stream isn't backed up with a single slow operation.

So could You please clear if such thing may be implemented by suggested way (my first mail)?

I wouldn't implement it, and I would advise both Matt and Salvatore to reject the pull request for every reason I've outlined in this post. It can and will lead to data corruption. And Redis will get the blame for the data corruption, not the user who turned on the configuration option.

 - Josiah

I am not so familar with Redis source and it will take much more time for me to answer this question.

 

Thanks in advance,
Dmitriy

пятница, 14 ноября 2014 г., 9:39:10 UTC+2 пользователь Dmitriy Kaloshin написал:

Hi all,

 

it looks like nobody yet asked for a subject except one post Is replication strictly sequential? where Josiah Carlson answered YES.

 

To understand my point I have to present a small and simple example:

 

1. We have Master and Slave.

2. On Master we have large transaction of some data.

3. Then again we have small transaction of another data but very important one.

4. We expect to get it on the Slave but unfortunately due to #1 we have to wait until time when first data processes its synchronization.

5. If we have bad internet connection then sync time of second transaction becomes critical for us.

 

I made a small research using Windows port of Redis Source 2.8.17 (MSVS 2013) where I built a local environment with master and slave and played over in debugger to see how everything goes on.

 

At first side (May be I am wrong) I discovered that each change on master side fires creation of aeFileEvent and then it processes through method aeProcessEvents.

 

It looks like it is not so difficult to add another field with name Priority to struct aeFileEbent and at the same time to make some sorting inside of aeProcessEvents.

 

It would be very nice to have such feature as an option like

 

A. Probably priority will be switched off in config by default.

B. When it is switched on then set priority depending on DB. Like db0 has highest priority, db1 lower priority and so on.

C. Probably to create another command with name SETDBPRIORITY where user will configure DBs at rin time.

 

Does it make sense?

 

brgds

Dmitriy

 

 

--

[Максим Синёв]

Hi Josiah

I am working with Dmitriy and  we want to use Redis for data distribution. First I like to explain why we think it is necessary to have prioritisation. We have several data types of different urgency e,g. stock quotes, various news, various documents. Those data are not linked in any way and stored in different DBs. Clients are connected via reasonably slow links while sources are pushing data independently, we want Stock data to be replicated before news and news before documents even if there are set of new documents to be replicated.

As far as I understand we should try to implement this feature our self if you could not suggest better option for our needs. As I understand your concern - in case of "data QOS" enabled we should limit/disable using inter-database commands . Could you see any other reasons why it might be problematic to implement ?

Sincerely, Maxim Sinev 

вторник, 18 ноября 2014 г., 1:43:35 UTC+3 пользователь Josiah Carlson написал:

[Josiah Carlson]

If your data is completely independent of each other, then there is no reason not to just use multiple Redis server processes. You can then use network/OS-level prioritization on those flows to ensure that your prioritized data gets priority.

 - Josiah

[Максим Синёв]

Well, yes - we thought of such  "plan-B", but downside of solution is much heavier clients, need for more resources (especially connections) and need to manage it all on both sides. Managing priority on network level is not easy either because the volume of such connections (packs of connections of different priorities) and the fact that internet routers would just ignore the external  QoS suggestions. Using internal Redis QoS provides much cleaner solution at least at first glance.

Are there any commands beside MOVE and EVAL(SHA) we should be potentially afraid of? Using SELECT inside MULTI block?   Regarding transactional integrity between DBs I would not afraid too much at least if it is not leading to unrecoverable data corruption. And to prevent data corruption we just need to either avoid commands that act between databases or force syncing those two/all DBs if such commands popped up  (pause higher priority to restore original sequence). 
BTW is it possible to replicate changes of such commands (say the EVAL command) instead of the command itself (Sorry if I am mistaken in concepts as I am quite new to Redis internals). Effectively translating them to sequence of DEL/(M/L)SETs ? Purpose - creation really slim read only clients that supports only subset of RESP protocol commands e.g. have no lua engine.
 

Sincerely, Maxim

вторник, 18 ноября 2014 г., 19:22:07 UTC+3 пользователь Josiah Carlson написал:

[Josiah Carlson]

On Tue, Nov 18, 2014 at 3:49 PM, Максим Синёв <msi...@gmail.com> wrote:

Well, yes - we thought of such  "plan-B", but downside of solution is much heavier clients, need for more resources (especially connections) and need to manage it all on both sides.

I have used databases to segment my data for years, and my client stopped supporting the SELECT command a long time ago (SELECT was found to leave connections in a state that was unknown to future callers). Instead, you create a new connection for every database (similar to how regular databases require you to create a new connection for each database). What I did was to write a simple function that hides any/all connection details:

conn = get_redis_connection('cache')

With a small amount of work, you can write an equivalent function in your programming language of choice. This shouldn't be substantially more difficult connection management than what you are currently doing, but if it is, I might recommend refactoring your connection management process.

Also, if you are concerned about multiplying your number of connections by your number of databases (assuming you are running with the default 16 databases), then you already have a connection count problem and should be looking into ways of doing connection pooling. If you are using a stack that can't do connection pooling (PHP inside an Apache prefork configuration), then I would suggest either installing Twemproxy/Nutcracker on your local machines to minimize outgoing connections (if your command set is supported by Twemproxy/Nutcracker), or migrating to a platform that can do connection pooling (almost literally every other stack).

 

Managing priority on network level is not easy either because the volume of such connections (packs of connections of different priorities) and the fact that internet routers would just ignore the external  QoS suggestions. Using internal Redis QoS provides much cleaner solution at least at first glance.

QoS is only a problem because you are trying to run a Windows Redis server. If you use Linux or BSD, you can force certain packets out onto the network first (look into the 'tc' command for Linux). Once they are on the network first, they will likely* get to the destination first. And with *nix prioritization, generally you can guarantee that all priority data (in network buffers) hits the network before non-priority data.

* The critical segment is your outgoing connection to the internet. Once it hits the internet, reordering happens, but it is already out of your slow network segment (which was the limiting factor). As long as you can get it from your server to your slow network connection in a prioritized fashion, you should be fine.

In addition to all of this, even without network prioritization, if you split your data into multiple Redis servers, then as long as your DB 1 equivalent isn't flooding the network with traffic, operations on DB 0 should be able to be replicated without significant difficulty or delay. This is simply because what is executing in DB 0 will be executing and replicated independent of what is executing or replicated in DB 1.

Are there any commands beside MOVE and EVAL(SHA) we should be potentially afraid of? Using SELECT inside MULTI block?   Regarding transactional integrity between DBs I would not afraid too much at least if it is not leading to unrecoverable data corruption. And to prevent data corruption we just need to either avoid commands that act between databases or force syncing those two/all DBs if such commands popped up  (pause higher priority to restore original sequence).

Off the top of my head, MOVE, EVAL(SHA), and SELECT inside a MULTI/EXEC transaction look to be the big ones. There may be others, but I don't have the time to go through the full command list to verify.

That said, who are you building this for? If you are building it for yourself, build whatever you want. If you are building it for inclusion in Redis core, then working on Windows Redis is not going to get your code into Redis core, never mind that the sequential execution nature of Redis precludes this from ever getting in**.

**  This would violate a wide variety of assumptions and established requirements for Redis replication, and would necessarily violate failover assumptions.

 

BTW is it possible to replicate changes of such commands (say the EVAL command) instead of the command itself (Sorry if I am mistaken in concepts as I am quite new to Redis internals).

Initial versions of Lua scripting in Redis propagated the *results* of scripts instead of the scripts themselves. At some point before Redis 2.8 (it may have even been before 2.6), Redis started propagating the script/sha + arguments to replicas. For many Lua scripts, replicating the script + arguments is significantly smaller than replicating individual command results, which is why it is done that way.

 

Effectively translating them to sequence of DEL/(M/L)SETs ? Purpose - creation really slim read only clients that supports only subset of RESP protocol commands e.g. have no lua engine.

It would seem that you don't understand Lua scripting actually does. Redis Lua scripting runs the Lua script *inside* Redis, allowing you to do basically anything***. Your clients just need to know how to pass the script/sha plus arguments as described: http://redis.io/commands/eval or http://redis.io/commands/evalsha

*** There are several caveats about this, which are described in the EVAL documentation.

On a related note: are you at least compressing the data between your master and slave? There are several examples of compression reducing Redis replication traffic to roughly 10-15% of the original size, which may change this entire conversation.

 - Josiah