What do people usually use to monitor latency in prod like environment, with minimal overhead
For my current project I was considering simply timestamping incoming /outgoing messages on the server and recording the latency numbers on a hdrHisTogram, having an embedded http server to send histogram data when requested. What do people think?
--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/I4JfZQ1GYi8/unsubscribe.
To unsubscribe from this group and all of its topics, send an email to mechanical-symp...@googlegroups.com.
Could you please explain what important information Metrics can miss?
I understand what you’re saying, but if my history is one minute, or one second even, does it make a difference? Suppose I occasionally get an unacceptable latency - say 3 seconds instead of 50 millis. That number is going to show up as “max” in the Metrics data. If Metrics reports every second, and the effect happens every 10 seconds, I will get an undesired max value every ten seconds.
If bad latency happens more than once a second, then it will not only effect the max value, but maybe the 99.9th percentile as well.
I get a new histogram that describes the latency distribution of, say, 10K operations every second. So even if the histogram for those 10K operations is off, I still get a new histogram every second. Could I miss a serious problem in the app? After all, that’s what Metrics is for: production performance monitoring that’s supposed to alert me of trouble; it is not a benchmarking tool.
Could you please explain what important information Metrics can miss?
I understand what you’re saying, but if my history is one minute, or one second even, does it make a difference? Suppose I occasionally get an unacceptable latency - say 3 seconds instead of 50 millis. That number is going to show up as “max” in the Metrics data. If Metrics reports every second, and the effect happens every 10 seconds, I will get an undesired max value every ten seconds.
If bad latency happens more than once a second, then it will not only effect the max value, but maybe the 99.9th percentile as well.
I get a new histogram that describes the latency distribution of, say, 10K operations every second. So even if the histogram for those 10K operations is off, I still get a new histogram every second. Could I miss a serious problem in the app? After all, that’s what Metrics is for: production performance monitoring that’s supposed to alert me of trouble; it is not a benchmarking tool.
To unsubscribe from this group and all of its topics, send an email to mechanical-sympathy+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/I4JfZQ1GYi8/unsubscribe.
To unsubscribe from this group and all of its topics, send an email to mechanical-sympathy+unsub...@googlegroups.com.
To unsubscribe from this group and all of its topics, send an email to mechanical-symp...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/I4JfZQ1GYi8/unsubscribe.
To unsubscribe from this group and all of its topics, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/I4JfZQ1GYi8/unsubscribe.
To unsubscribe from this group and all of its topics, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.
To unsubscribe from this group and all of its topics, send an email to mechanical-sympathy+unsub...@googlegroups.com.