Is mass insertion the fastest technique to insert large amounts of data to redis?
Nachi Vpn
Marc Gravell
Firstly I disagree with the assertion (stated in words and mathematically) that pipelining cuts RTT in half : it does much, much better than that. It means you only pay *1 of* RTT - most easily considered as "the last one", since most socket libraries buffer at the NIC and deal with that anyway - allowing to to keep writing.
If I had to guess, the difference you are seeing is *packet fragmentation*. Obviously I don't have your test to repro, but it sounds consistent with the approach you are taking. If your "single command" approach is also "single packet", then yes - that will be less effective on networks with high MTUs. What you could investigate there as a quick test is simply: batching; rather than send one command each iteration send 10 in a batch (this can still be pipelined). Obviously this would depend on your client library supporting pipelined batching!
Marc
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.
Marc Gravell
Just to clarify - by "packet fragmentation" here, I simply mean: sending lots and lots of tiny packets, rather than a smaller number of larger packets. One packet per command is pretty inefficient in terms of bandwidth, but will get a faster initial response than if you wait for more work - obviously, the server can't process what you haven't sent it yet! Clearly that isn't important when pipelining, because you *don't care about* (in terms of latency) the response to the first message; you care more about the response to the *last*.
For comparison, what SE.Redis does (one of the .NET clients) is: commands are always sent to a dedicated writer thread. There is always some tiny time involved in the writer thread waking up, which means there might *already* be multiple things in the pipe waiting to be sent. It then additionally writes to a buffered stream, which automatically sends to the socket whenever it hits the chosen size. Whenever the writer runs out of things to write, it does a very short spin-wait; if there still isn't anything to do, it'll send the short packet - but the idea is that *when under heavy load* it biases towards packets with multiple messages in them (possibly even nicely full packets, where "full" simply represents our arbitrary send size).
Marc
Nachi Vpn
Marc Gravell
Time taken for a command is:
- client outbound processing
- latency (client-server)
- bandwidth (client-server)
- server-side processing
- latency (server-client)
- bandwidth (server-client)
- client inbound processing
The only thing that pipelining changes is: latency. Rather than paying latency per-request, you *only* pay the latency cost once - because you *don't keep waiting*. All other costs remain the same.
As for your "why is it more expensive" - pipelining does involve additional things like keeping a queue of outstanding messages, and often async IO. However, compared to the *huge* cost of latency, that is a good trade. Latency is usually the single most expensive cost.
I haven't looked at how your rig is structured, but done properly: pipelining should be very very efficient. If you saturate the network with tiny packets, that will be diminished somewhat.
Does that make sense?
Marc