[New Feature Design] JedisCluster with Pipeline

[임정택]

Hello all.

Current JedisCluster lacks Pipeline feature so we need to send each command separately.

Pipeline feature has greatest throughput (even though Async!) so we need this feature to make JedisCluster more amazing!

But it's not that simple because we should route commands to each host according to slot, and handle MOVED, ASK when Redis responses.

To make our issue simple, I wonder we're fine to allow pipeline only all of commands are willing to access "same" "one" slot.

(It's same to handle commands with multi keys.)

If we're fine with this, we're good to go!

Since I don't design this feature yet, I'll share some sketches when available.

I wish to discuss this with users / devs, so please feel free to give an opinion!

Thanks.

Regards.

Jungtaek Lim (HeartSaVioR)

[Jonathan Leibiusky]

Hi Jungtaek!

That's a very important feature indeed.
Do you know if other clients are offering something similar and how they do it?

--
You received this message because you are subscribed to the Google Groups "Jedis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[임정택]

Hi, Jonathan!

No, I can't find any clients that supports Redis Cluster with Pipeline.

Actually only a few clients support Redis Cluster for now.

Here is some interesting document from redis-py-cluster. 

If pipeline is used with Cluster, it doesn't actually play like pipeline, but keeps interface so users can enjoy like they're using pipeline.

https://github.com/Grokzen/redis-py-cluster/blob/f857e647cbd92837b68c7a5484b9389fcc378984/docs/Limits_and_differences.md

I think we can support pipeline with Cluster, with some limitation I've stated, or none.

Let me know if you find any clients supporting this.

Regards.

Jungtaek Lim (HeartSaVioR)

2014년 11월 3일 월요일 오후 9시 57분 22초 UTC+9, Jonathan L. 님의 말:

[임정택]

I made a sketch to this feature.

First of all, since we already have Pipeline, I'm trying to respect its usage.

By user side, it would be equal to current Pipeline.

```

// assume JedisCluster named "cluster" is initialized

ClusterPipeline p = cluster.pipelined();

Response<String> string = p.get("string{1}");

Response<String> list = p.lpop("list{1}");

// ...

p.sync();

String stringVal = string.get();

String listVal = list.get();

// ...

```

By ClusterPipeline, it has a queue storing commands (transformed to raw, byte[]) and Response<?> for building.

Since we should treat MOVED, and ASK, and we should resend all commands when MOVED and ASK occur, so we cannot use current strategy for Pipeline. (adding commands to OutputStream and flush it when sync() called)

When ClusterPipeline.cmdxxx() (ex. set()) is called,

1. check slot number

2. convert command with arguments to raw byte[] .

3. generate Response<?> with getResponse(Builder), traditional approach.

4. push (1, 2, 3) to queue

5. return 3 (generated, but not built Response<?>)

When ClusterPipeline.sync() is called, (assume ClusterPipeline has reference of JedisCluster instance)

1. Check all command's slots

1.a. If slots count is 2 or upper, it throws JedisClusterCrossSlotException from #687 currently waiting for merge.

2. Get a connection related to slot

2.a. If it doesn't exist from slot-node cache, get a random connection

3. Send all commands at once, by joining all byte[]s to one.

4. Build responses

5. Check first response

5.a. If first response contains MOVED or ASK, update slot-node cache and go to 2. (with increasing redirect count)

6. Return (Responses should be built, so users can play with them.)

Please review my sketch, and let me know there're sth. to add, or sth. to be fixed.

Regards.

Jungtaek Lim (HeartSaVioR)

2014년 11월 4일 화요일 오전 7시 21분 47초 UTC+9, 임정택 님의 말:

[Jonathan Leibiusky]

There is a possible state where a slot is moving and keys will be in 2 different servers. In this case how your algorithm will work?

[임정택]

Actually I couldn't think about it. I was thinking commands within pipeline pointed to same slot is atomic, but it's wrong.

Btw, if we can allow that commands are executed without sequences, we can build response, and filter those which need to retry, and run again.

How about this alternative approach?

2014년 11월 11일 화요일, Jonathan Leibiusky<iona...@gmail.com>님이 작성한 메시지:

You received this message because you are subscribed to a topic in the Google Groups "Jedis" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jedis_redis/u6j8slokO3E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jedis_redis...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior

LinkedIn : http://www.linkedin.com/in/heartsavior

[Jonathan Leibiusky]

I guess we can add an additional condition to fix this.

1. All commands in the pipeline must be of slots that lives in the same node

2. Slots must not be in a moving state

To know 2 we need to ask the cluster when we start the pipeline. So we can also use this opportunity to update our slot routing table.

Since we now know where the slot lives, we just get a connection to the right node and pipeline all the commands as a regular pipeline.

Does this make sense or am I missing something?

[임정택]

There're race conditions between asking slot state and sending pipeline, and each command inside of pipeline.

No ensuring of transaction, we can't rely on 2 though we know slot is not moving now.

I wonder Redis cluster has a mechanism to support it, it would be mandatory.

[Jonathan Leibiusky]

Do you mind explaining a bit more about the race condition you mentioned?

[임정택]

Redis ensures that only single command (includes lua script) or multi block is atomic, that means others are not atomic.

So we ask Redis Cluster to check slot is not moving, but it doesn't ensure that slot is not moving when we send commands with pipeline, especially Redis executes those commands.

In addition, Redis doesn't ensure that all commands in pipeline runs atomically, so slot moving could be occured between commands.

Please correct it if it misses something.

Thanks!

[임정택]

I posted regarding Redis Cluster with Pipeline to Redis google groups, so we can hear some voices from post.
https://groups.google.com/d/msg/redis-db/4I0ELYnf3bk/Lrctk0ULm6AJ

2014년 11월 11일 화요일 오후 11시 23분 13초 UTC+9, 임정택 님의 말:

To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Jedis" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jedis_redis/u6j8slokO3E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jedis_redis+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior

LinkedIn : http://www.linkedin.com/in/heartsavior

--
You received this message because you are subscribed to the Google Groups "Jedis" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Jedis" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jedis_redis/u6j8slokO3E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jedis_redis+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior

LinkedIn : http://www.linkedin.com/in/heartsavior

--
You received this message because you are subscribed to the Google Groups "Jedis" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Jedis" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jedis_redis/u6j8slokO3E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jedis_redis+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[임정택]

After long conversation to Josiah Carlson, 

I've noticed that my solution can work fine but it would be blamed due to break executing sequence, whether we document it or not.

I agree with him, so I decide to discard my idea.

Btw, Josiah give me new approach, MULTI-EXEC.

Though it should be batched (not pipeline), we can safe with race conditions.

Below is Josiah's comment from another thread.

When Redis receives EXEC, it:
1. looks at all of the commands that were sent after the most recent MULTI
2. extracts the keys from those commands
3. verifies that all of the keys are in the same slot
4. verifies that the slot is in this machine
5. verifies that all of the keys are available in this machine (which handles in-progress slot migration)
No execution happens inside a MULTI/EXEC block unless Redis cluster is pretty sure that it will complete successfully. Which means that you shouldn't get an error at all.

And slot migration is performed with MIGRATE all keys in slot one by one,

(see https://github.com/antirez/redis/blob/unstable/src/redis-trib.rb#L764-L809)

we can ensure migrate command isn't executed when EXEC command is processed.

I can see Josiah's approach can work perfectly, though we should deal with TRYAGAIN in addition to CROSSSLOT, ASK, MOVED.

How do you think about Josiah's approach?

Regards.

Jungtaek Lim (HeartSaVioR)

2014년 11월 15일 토요일 오전 9시 20분 0초 UTC+9, 임정택 님의 말:

[Jonathan Leibiusky]

I think I prefer that approach. But I wonder what exactly we need to support from the client side. 

[임정택]

Currently JedisCluster doesn't support Transaction.
I was thinking Pipeline for gaining performance, but with Redis Cluster restrictions, we can't use Pipeline.
But I was thinking we cannot use multi in Redis Cluster, but after investigating we can use it.
Then we can add feature, JedisCluster with Transaction, and let users use it instead.

To unsubscribe from this group and all its topics, send an email to jedis_redis...@googlegroups.com.

[Jonathan Leibiusky]

Ok. That sounds good. 
Very good research!

[임정택]

Thanks!

Btw, in the middle of slot migrating, Jedis Cluster slot cache can be changed.
It means that if we're sending multi, and send commands to being queued, commands may be sent to 2 or more nodes and making some errors, which is critical for us.
In order to prevent it, we should "batch" commands and send all to only one node which serves slot at that time.
When Redis throws MOVED, ASK, TRYAGAIN, it's up to us to make it retry automatically, or delegate handling to users.
(What's more appreciated approach?)

Regards.
Jungtaek Lim (HeartSaVioR)

[mrazola]

Hi!

Any news regarding this feature request? It would be very interesting to have JedisCluster suport transactions (if keys belong to the same node).

[Guillermo Casanova]

I would love to see this implemented, did this get any further? Regards!

[Akshay Viswanathan]

Hi, 

Are there any updates on this feature request? 

It seems like many other major redis client libraries support this, so perhaps we can use a similar approach?