RE: pipelining, how does redis keep track?

[S Ahmed]

With redis pipelining, how does redis keep track that the last 4 calls that were pipelined are to be returned in the single response back?

To they get wrapped in some sort of a "transaction" that redis keeps appending to and then sends the response back grouped together?

[Marc Gravell]

If you are **just** pipelining, there is nothing to keep track of: it sends each result separately. Pipelining does not cause a single response (quite the opposite).

If you mean "multi"/"exec" (which is different to pipelining), then at the simplest level it simply buffers all the commands until "exec" (responding with a "pending" as they arrive); when you call "exec", it is only at that point that it actually executes them, so again: it isn't overly complex.

Marc

On 3 December 2012 14:42, S Ahmed <sahme...@gmail.com> wrote:

With redis pipelining, how does redis keep track that the last 4 calls that were pipelined are to be returned in the single response back?

To they get wrapped in some sort of a "transaction" that redis keeps appending to and then sends the response back grouped together?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
Regards,

Marc

[S Ahmed]

Ok just a bit confused, see: 

https://github.com/xetorthio/jedis/wiki/AdvancedUsage

"Sometimes you need to send a bunch of different commands. A very cool way to do that, and have better performance than doing it the naive way, is to use pipelining. This way you send commands without waiting for response, and you actually read the responses at the end, which is faster."

Pipeline p = jedis.pipelined();
p.set("fool", "bar"); 
p.zadd("foo", 1, "barowitch");  p.zadd("foo", 0, "barinsky"); p.zadd("foo", 0, "barikoviev");
Response<String> pipeString = p.get("fool");
Response<Set<String>> sose = p.zrange("foo", 0, -1);
p.sync(); 

int soseSize = sose.get().size();
Set<String> setBack = sose.get();

[Derek Williams]

That's just a poor pipeline implementation. It doesn't need to be done in a batch like that.

Derek Williams

[S Ahmed]

Derek,

Are you referring to the example is bad or the actual java client implementation is bad?  (in what way?)

[Derek Williams]

The java client implementation is poor because it requires waiting for all the given commands to respond  before getting the first response. I thought that was the whole point of your confusion? Or was it something else?

[S Ahmed]

My confusion surround around understand what pipelining is, and I am using the jedis client so looking at the example I assumed they implementation was based on how redis pipelining works.

"With redis pipelining, how does redis keep track that the last 4 calls that were pipelined are to be returned in the single response back?

To they get wrapped in some sort of a "transaction" that redis keeps appending to and then sends the response back grouped together?"

[S Ahmed]

Reading this: http://redis.io/topics/pipelining

The jedis client seems to be doing this, " read the replies in a single step"

"A Request/Response server can be implemented so that it is able to process new requests even if the client didn't already read the old responses. This way it is possible to send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step."

[Derek Williams]

Reading it all as a single step isn't required at all and could potentially be harmful if there is lots of data being buffered waiting to be read. I don't know if the Jedis client begins reading before the call to 'sync', or if it actually does wait until then to read all the data.

Pipelining in Redis is very similar to pipelining with HTTP, which pretty much means a request does not have to wait for the previous response to be received. Responses are received in the same order as the requests, but there is no need to stop and wait for all the responses in order to receive just one of them.

So the documentation on redis.io seems to poorly explain this also.

[Marc Gravell]

The implementation of pipeling at the client is a client/library concern. Typically it simply involves keeping a queue of outstanding expected replies, and processing them as they arrive. One approach of that (meaning: the way I do it) might be to return a "future" from the original operation, and then provide the *result* of the "future" later. Callers can then either wait on the future's result, or use an async-callback to process it when it arrives without blocking.

---

From: Derek Williams
Sent: 03/12/2012 16:12
To: redi...@googlegroups.com
Subject: Re: pipelining, how does redis keep track?

[Derek Williams]

Yes, that is the same pattern I use as well (return a future for each request), and from what I have seen so far it provides the greatest flexibility to allow the user to decide how to handle the responses.

[Derek Williams]

On Mon, Dec 3, 2012 at 1:05 PM, Sam Hendley <sam.h...@gmail.com> wrote:

Just to defend the jedis library a little, the "Response" class in jedis is actually a future, its name should probably be changed to better indicate that. I have worked with a number of client libraries and I am very impressed with the simplicity and design of this library. I have a pull request out to add the async reading you mentioned if anyone is interested.

I had a quick read through of your pull request, and yes, that does appear to fix pipelining. Good job!

--

Derek Williams