explaining how pipeline works

[Gitted]

Can someone explain to me how pipeline works?

When I use pipelining, say like the wiki shows:

Pipeline p = jedis.pipelined();
p.set("fool", "bar");
p.zadd("foo", 1, "barowitch"); p.zadd("foo", 0, "barinsky");
p.zadd("foo", 0, "barikoviev");
Response<String> pipeString = p.get("fool");
Response<Set<String>> sose = p.zrange("foo", 0, -1);
p.sync();

int soseSize = sose.get().size();
Set<String> setBack = sose.get();

When I call p.set("foo", "bar"), does this call get sent over the
network to the redis server, or does it send all the calls together?

I would really appreciate if someone could explain what is happening
after each call in the above snippet.

thanks!

[Baishampayan Ghose]

Pipelining is a way of sending multiple commands to the server without waiting for the responses individually and then reading all the responses in one shot. It's better to use a pipeline if you want to set multiple keys in a hash-map, for example.

More here - http://redis.io/topics/pipelining

Regards,

BG

--
Baishampayan Ghose
b.ghose at gmail.com

[S Ahmed]

So in the snippet I sent, these calls will be sent out over the network to the redis server individually still right?

p.set("fool", "bar");

p.zadd("foo", 1, "barowitch");  p.zadd("foo", 0, "barinsky");
p.zadd("foo", 0, "barikoviev");

So that is still 3 calls, but it doesn't send back responses back for each one, rather you get 1 back for all 3 calls.

[Sam Hendley]

Yes the command will be sent individually and may be processed interleaved with other requests.

There is nothing magical about pipelining, its entirely a client side feature. We stream a set of requests to the server, it streams us the responses and we defer the parsing of the results until we are ready. It just allows us to wait and decide when we are going to wait for the response for the server. If we are doing many commands in a row without needing the results to any of the requests we can be much faster.

Keep in mind that this means the requests can be interleaved with other clients, use MULTI/EXEC as well if you want the commands to be processed in a single shot. 

Sam

BTW: I believe this note from the pipelining page is a bit misleading. Isn't the queue/storage of the results effectively the TCP/IP stream and socket buffers? It reads like redis server is doing something special for pipelining.

IMPORTANT NOTE: while the client sends commands using pipelining, the server will be forced to queue the replies, using memory. So if you need to send many many commands with pipelining it's better to send this commands up to a given reasonable number, for instance 10k commands, read the replies, and send again other 10k commands and so forth. The speed will be nearly the same, but the additional memory used will be at max the amount needed to queue the replies for this 10k commands.

[S Ahmed]

Sam,

I actually asked about pipelining on the redis group, and some replies suggested that the jedis implementation might not be ideal: 

(they didn't say for certain, but from the github wiki snippet on pipelining led them to question some things).

https://groups.google.com/group/redis-db/browse_thread/thread/959bc6381b827398

I'd love for your feedback/comments.

[Sam Hendley]

I commented there, I think the jedis implementation is solid. If my async patch gets accepted it would solve the one issue the current design has.

https://github.com/xetorthio/jedis/pull/349

Sam