Poor man's continuous profiler using stopwatch approach?

[Kevin Burton]

Wanted feedback on this as I just thought of it now :)

90% of our performance hotspots in production are obvious issues.

For example:

- making a REST API call

- performing a DNS request

- sending a message to a queue

- fetching a URL.

Some of these are fast for one invocation, say 5ms, but give a significant number of requests, they can add up... 100k of them would take 50 seconds for example.

So if you were to code these by wrapping them in a stopwatch, something like:

Stopwatch stopwatch = Stopwatch.createStarted();

try { 

   doSomethingReasonablySlow();

} finally { 

    stopwatch.stop();

}

then what you could do is keep a log of these operations. 

Then you could have a report generated at the end tracing your programs latency.

[Greg Young]

Sure using metrics like this is useful and these things should be in your code.

Also take a look at hdrhistogram quite useful to push values into there as well

> --
> You received this message because you are subscribed to the Google Groups
> "mechanical-sympathy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mechanical-symp...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Studying for the Turing test

[Michael Barker]

You can use tools like btrace to patch in this sort of code on the fly.

[John Meyer]

I've had very good results using the codahale metrics library for measuring off system latencies like you listed. For greater control over the range and precision for very tight timings, I would use hdrhistogram.

[Marshall Pierce]

You can use hdrhistogram as the in-memory storage for metrics with https://bitbucket.org/marshallpierce/hdrhistogram-metrics-reservoir. This is applicable to Histogram, Timer, etc.

[John Meyer]

Oh, that's very nice. No need to vary timer implementations based on expected latencies. Thank you!

[Trask Stalnaker]

Glowroot is also a nice option for instrumenting these timings on the fly (warning: biased author’s opinion).

You can add this type of method-based timer directly in the UI (e.g. https://demo.glowroot.org/config/instrumentation?new).

Glowroot will roll up these timings per named transaction (e.g. per url into your application), which provides a nice breakdown of what’s happening in the system (e.g. https://demo.glowroot.org/transaction/metrics).

HdrHistogram is used to track transaction latencies (e.g. https://demo.glowroot.org/transaction/overview) though only a couple of specific percentiles are displayed in the UI currently.

Thanks,

Trask

[Kevin Burton]

I agree.. we use codahale/dropwizard metrics too.. I think what I'm primarily thinking of something as kind of a combination between the two and maybe even lightweight.  maybe like linux perf where you have exported counters.  

Tie in a little webapp and I think it could be pretty powerful.  

> email to mechanical-sympathy+unsub...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
Studying for the Turing test

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Kevin Burton]

I was also thinking that Java 8 lambdas would make wrapping the code easier because you could just implement a macro and wrap it around anything you want timed.  

[Gary Mulder]

On 28 April 2015 at 21:25, Kevin Burton <burto...@gmail.com> wrote:

then what you could do is keep a log of these operations. 

Then you could have a report generated at the end tracing your programs latency.

For every way-point of interest in your log, collecting at least two metrics from Little's Law (queue length, effective arrival rate, system processing time) allows you to compute the third.

These metrics are invaluable at identifying primary and secondary bottlenecks.

Gary