Visualizing latency

[Philip]

Hey,

There was a thoughtful article about benchmarking at High Scalability recently. It mentioned the TechEmpower Framework Benchmarks, and included a number of opinions on what a performance metric might look like if it included both client load and latency.

http://highscalability.com/blog/2016/4/13/10-stack-benchmarking-dos-and-donts.html

I think many of the author's suggestions are very good. It would be great if we could get reliable latency numbers from the load generation tool (any news there?).

What I found most useful was the visualization. A cumulative distribution plot with logarithmic scale of percentile along the x-axis, and linear latency scale on the y-axis. Here is an example:

You can add as many lines as you can visually parse. It can get hard to read, but grouping things by color can help.

It might be nice to pick up the conversation about latency _and_ throughput again. Should we consider adding a new type of constant, defined load test?

Just riding the pine over here,

Philip

[Naoki INADA]

I agree it is nice article.

Especially, he says

> Avoid ...  then report some (usually bogus) latency stats at those max throughout modes.

The author must know why I'm against showing response time of wrk in public result page.

To measure response time spectram, we should use constant load tool, not wrk.

https://github.com/rakyll/boom is one tool I know.

Any other suggestion for easy to use, simple command line constant load generator?

[Naoki INADA]

Two things you should know is:

1. Benchmarking your (one or two) application and hundreds of frameworks is different thing.

2. Max throughput is useful enough.

While I agree "compare response time / latency spectrum at given loads." is nice,

it's hard job.

I don't know techempower people have enough time to do it.

I prefer focusing throughput to next round never comes.

[Aliaksandr Valialkin]

There is also wrk2 - https://github.com/giltene/wrk2 .