latency
116 views
Skip to first unread message
S Ahmed
Nov 9, 2012, 3:06:52 PM11/9/12
to redi...@googlegroups.com
I'm not sure how to phrase this correctly, but when you connect to a dedicated redis server that is in the same datacenter (or even rack), generally what kind of latency should one expect? Is it in the ms range or even less?
When I was benchmarking, I was seeing a range of 1-5ms (the calls where mostly incrementing a counter).
Assuming no data is being returned, is there a general range and in reality how much fluxuation do you see and what are the general causes of this flucation? (assuming you are not reaching the e.g. 20K requests per secodn limit that your redis has)
Jeremy Zawodny
Nov 9, 2012, 3:08:29 PM11/9/12
to redi...@googlegroups.com
I typically see 0.5ms but it'll depend on your network gear, I suspect.
Jeremy
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Salvatore Sanfilippo
Nov 9, 2012, 3:09:04 PM11/9/12
to Redis DB
redis-cli --latency may be useful in this regard.
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org
Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
--
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org
Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter
Marc Gravell
Nov 9, 2012, 3:14:15 PM11/9/12
to S Ahmed, redi...@googlegroups.com
Iirc, the normal timing a few years ago for LAN latency was normally about 0.3ms, although that may depend on hardware, and may be outdated. Network congestion will make that worse, obviously. You certainly shouldn't be seeing 5ms if the nodes are pretty close to each-other.
To reduce the *perceived* latency, you can use the pipelined nature of redis. Specifically, you can send dozens of commands to the server before you've got the first response back. That way, for "n" operations, instead of paying "n * latency + bandwidth" you just pay "latency + bandwidth".
Marc
To reduce the *perceived* latency, you can use the pipelined nature of redis. Specifically, you can send dozens of commands to the server before you've got the first response back. That way, for "n" operations, instead of paying "n * latency + bandwidth" you just pay "latency + bandwidth".
Marc
From: S Ahmed
Sent: 09/11/2012 20:07
To: redi...@googlegroups.com
Subject: latency
S Ahmed
Nov 9, 2012, 3:18:01 PM11/9/12
to redi...@googlegroups.com
So would there a any noticeable difference between say the ruby and java clients?
Or would it be more of a an difference in the time it takes to generate the request, but once it is fired off and on wire it really is not a language issue (as I right this I am realizing I have answered it but I am still going to ask just in case i learn something new!)
Yiftach Shoolman
Nov 9, 2012, 3:30:26 PM11/9/12
to redi...@googlegroups.com
From our experience in EC2:
1. when your App and Redis instances are in the same zone - 0.5msec RTT
2. when your App and Redis instances are in the same region but in different zone - 1msec RTT
Josiah Carlson
Nov 9, 2012, 3:40:18 PM11/9/12
to redi...@googlegroups.com
Hard to say without testing. Run the Redis latency tests with
redis-cli, and try to calculate the same from Ruby or Java.
- Josiah
redis-cli, and try to calculate the same from Ruby or Java.
- Josiah
Salvatore Sanfilippo
Nov 10, 2012, 7:23:42 AM11/10/12
to Redis DB
The latency you pay is the sum of the following:
1) Time for your client to process the command, that is, everything
from funciton/method call to the write(2) syscall to the socket.
2) Time for the kernel to write the TCP buffer on the physical link.
This is why not having TCP_NODELAY set as socket option is a problem.
3) Time for the TCP packet to reach the other end. This is the
physical link latency neat cost.
4) Time for the receiving TCP/IP stack to pass data to the application
(the Redis server). Here different kinds of multiplexing APIs will
affect the performance of course, usually here you have Linux+epoll
API.
5) Time for Redis to process the request and write it to the client
output buffer.
6) Time for Redis to call the callback for the writable event for this
client, so finally data hit write(2) against the socket.
7) "2" again, bot on the other side.
8) "3" again
9) "4" again, here it is important that your client is ready waiting
in read(2) if it's a blocking client, to get the data ASAP.
10) Time needed to convert the data from the Redis protocol to your
programming language return value.
As you can see there are a lot of things that can go wrong, and I did
not mentioned everything. At lower level interrupts need to fire,
hardware may detect collisions, packets can get lost, and so forth.
So for good latency you need:
1) A good client.
2) Redis running on Linux, with a modern kernel and good drivers for
your hardware.
3) A fast ethernet that is not near capacity.
4) As little hops as possible between your client and your server, and
good switches & co.
4) Fast CPUs on both side help also. This is mostly I/O bound but CPU
plays a role.
Cheers,
Salvatore
1) Time for your client to process the command, that is, everything
from funciton/method call to the write(2) syscall to the socket.
2) Time for the kernel to write the TCP buffer on the physical link.
This is why not having TCP_NODELAY set as socket option is a problem.
3) Time for the TCP packet to reach the other end. This is the
physical link latency neat cost.
4) Time for the receiving TCP/IP stack to pass data to the application
(the Redis server). Here different kinds of multiplexing APIs will
affect the performance of course, usually here you have Linux+epoll
API.
5) Time for Redis to process the request and write it to the client
output buffer.
6) Time for Redis to call the callback for the writable event for this
client, so finally data hit write(2) against the socket.
7) "2" again, bot on the other side.
8) "3" again
9) "4" again, here it is important that your client is ready waiting
in read(2) if it's a blocking client, to get the data ASAP.
10) Time needed to convert the data from the Redis protocol to your
programming language return value.
As you can see there are a lot of things that can go wrong, and I did
not mentioned everything. At lower level interrupts need to fire,
hardware may detect collisions, packets can get lost, and so forth.
So for good latency you need:
1) A good client.
2) Redis running on Linux, with a modern kernel and good drivers for
your hardware.
3) A fast ethernet that is not near capacity.
4) As little hops as possible between your client and your server, and
good switches & co.
4) Fast CPUs on both side help also. This is mostly I/O bound but CPU
plays a role.
Cheers,
Salvatore
S Ahmed
Nov 10, 2012, 11:23:16 AM11/10/12
to redi...@googlegroups.com
How do you monitor if your ethernet is the bottleneck?
Jay A. Kreibich
Nov 11, 2012, 9:19:02 PM11/11/12
to redi...@googlegroups.com
On Sat, Nov 10, 2012 at 11:23:16AM -0500, S Ahmed scratched on the wall:
almost guaranteed the physical link is *not* a significant contributor
to the overall latency. Twisted pair and fiber Ethernet signals
propagate at nearly the speed of light, making them insignificant
when inside the same building. Switches add some latency, but a
full store-and-forward switch (the slowest method) doing full size
Ethernet frames at 100Mbps only adds about 0.12ms to the transport
latency.
Unless the links are running very close to capacity (creating link
contention), have a auto-sense, duplex mismatch, or some other
abnormality, almost all the "socket to socket" latency is going to be
inside the two machines on either end of the link, mostly in kernel
processing, buffer copies, and context switches. The latency in
actually moving the bits only starts to get interesting when the
machines are in different buildings.
-j
--
Jay A. Kreibich < J A Y @ K R E I B I.C H >
"Intelligence is like underwear: it is important that you have it,
but showing it to the wrong people has the tendency to make them
feel uncomfortable." -- Angela Johnson
> How do you monitor if your ethernet is the bottleneck?
If you have two machines on the same local Ethernet fabric, it is
almost guaranteed the physical link is *not* a significant contributor
to the overall latency. Twisted pair and fiber Ethernet signals
propagate at nearly the speed of light, making them insignificant
when inside the same building. Switches add some latency, but a
full store-and-forward switch (the slowest method) doing full size
Ethernet frames at 100Mbps only adds about 0.12ms to the transport
latency.
Unless the links are running very close to capacity (creating link
contention), have a auto-sense, duplex mismatch, or some other
abnormality, almost all the "socket to socket" latency is going to be
inside the two machines on either end of the link, mostly in kernel
processing, buffer copies, and context switches. The latency in
actually moving the bits only starts to get interesting when the
machines are in different buildings.
-j
Jay A. Kreibich < J A Y @ K R E I B I.C H >
"Intelligence is like underwear: it is important that you have it,
but showing it to the wrong people has the tendency to make them
feel uncomfortable." -- Angela Johnson
M. Edward (Ed) Borasky
Nov 11, 2012, 9:45:01 PM11/11/12
to redi...@googlegroups.com
On Sun, Nov 11, 2012 at 6:19 PM, Jay A. Kreibich <j...@kreibi.ch> wrote:
[snip]
> Unless the links are running very close to capacity (creating link
> contention),
Indeed ... there's a certain point when too many stations yammering at
once on a link will create enough contention to cause real problems.
Other than that, you're usually good. Sniffers normally can spot this
and warn you.
--
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/
How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?
[snip]
> Unless the links are running very close to capacity (creating link
> contention),
once on a link will create enough contention to cause real problems.
Other than that, you're usually good. Sniffers normally can spot this
and warn you.
--
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/
How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?
Jay A. Kreibich
Nov 12, 2012, 12:32:59 PM11/12/12
to redi...@googlegroups.com
On Sun, Nov 11, 2012 at 06:45:01PM -0800, M. Edward (Ed) Borasky scratched on the wall:
10-base-5 (thick-net), multi-port AUI transceivers, 10-base-2, and
repeated 10-base-T & 100-base-TX. And wireless-- but I assume you're
not running servers off wireless. Most of the problems can be
attributed to the shared nature of the media and the high-latency
collision recovery process.
The issue is much, much less true in today's world of switched,
point-to-point, full-duplex links (the only mode supported at gigabit
Ethernet and better speeds). In that environment, the only point of
contention is the switch trying to serialize outgoing Ethernet frames
onto a link. This is most common when the switch is trying to
multiplex many client links onto one server link. This type of
contention is a serialization issue, however, not a link-access issue.
It is not a collision, and therefore does not require exponential
back-off or any of the lengthy, latency-inducing recovery protocols.
Quality switching gear can simply buffer the frames and send them
out one after another, as fast as possible.
As a result, you typically need very high link utilization rates
to see noticeable latency issues that can be attributed to congestion.
-j
> On Sun, Nov 11, 2012 at 6:19 PM, Jay A. Kreibich <j...@kreibi.ch> wrote:
> > Unless the links are running very close to capacity (creating link
> > contention),
>
> Indeed ... there's a certain point when too many stations yammering at
> once on a link will create enough contention to cause real problems.
> Other than that, you're usually good. Sniffers normally can spot this
> and warn you.
That was certainly true back in the days of shared media such as
> > Unless the links are running very close to capacity (creating link
> > contention),
>
> Indeed ... there's a certain point when too many stations yammering at
> once on a link will create enough contention to cause real problems.
> Other than that, you're usually good. Sniffers normally can spot this
> and warn you.
10-base-5 (thick-net), multi-port AUI transceivers, 10-base-2, and
repeated 10-base-T & 100-base-TX. And wireless-- but I assume you're
not running servers off wireless. Most of the problems can be
attributed to the shared nature of the media and the high-latency
collision recovery process.
The issue is much, much less true in today's world of switched,
point-to-point, full-duplex links (the only mode supported at gigabit
Ethernet and better speeds). In that environment, the only point of
contention is the switch trying to serialize outgoing Ethernet frames
onto a link. This is most common when the switch is trying to
multiplex many client links onto one server link. This type of
contention is a serialization issue, however, not a link-access issue.
It is not a collision, and therefore does not require exponential
back-off or any of the lengthy, latency-inducing recovery protocols.
Quality switching gear can simply buffer the frames and send them
out one after another, as fast as possible.
As a result, you typically need very high link utilization rates
to see noticeable latency issues that can be attributed to congestion.
-j
Kevin Burton
Nov 13, 2012, 1:40:29 PM11/13/12
to redi...@googlegroups.com
How are you measuring the latency?
Reply all
Reply to author
Forward
0 new messages