My ZuriHac 2014 Project

[Alfredo Di Napoli]

Hi guys,

I've blogged (and there is a screencast included) about my ZuriHac 2014 project; I've basically worked on a version of snaplet-perf which uses acid-state instead of zoom-cache:

http://www.alfredodinapoli.com/posts/2014-06-09-my-zurihac-project-revamping-snaplet-perf.html

What you reckon?
Alfredo

[Gregory Collins]

It looks cool -- the graphs are a little strange though, for aggregates like "average response time" you'd prefer to see a different graph, preferably bar or box plot. If you keep a larger number of quantiles you can do really cool effects here like a blended-color stack plot for latency, something like this:

    http://www.appneta.com/blog/wp-content/uploads/2011/10/application-performance-3.png

but with more color bands, like this:

    http://www.vijayp.ca/movies/images/nogrey.colour.nolight.nosat.all.png

If you pick the color maps with narrow saturation peaks around the 50% and 95% level then you can get a really nice at-a-glance understanding of latency behaviour this way.

Re: the storage backend, not sure why you would choose acid-state here, as this basically rules it out for serious production use. The reason Doug picked zoom-cache in the first place is that it's designed with this format in mind: you have timeseries data that you would like to:

  a) stream to disk

  b) summarize and index this data in a way (successive power-of-2 aggregation/precomputation) that allows you to read an arbitrary time range with a constant or log-factor amount of disk (and RAM) I/O.

A service doing 10qps (pretty meager these days), assuming an acid-state overhead of 256 bytes per request (not unreasonable?), is going to generate 211MB of log data every day. It's not practical to store this on the heap.

G

Alfredo

--

---
You received this message because you are subscribed to the Google Groups "Snap Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snap_framewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Gregory Collins <gr...@gregorycollins.net>

[Alfredo Di Napoli]

On Tuesday, 10 June 2014, Gregory Collins <gr...@gregorycollins.net> wrote:
> It looks cool -- the graphs are a little strange though, for aggregates like "average response time" you'd prefer to see a different graph, preferably bar or box plot.

The js library I have used, Rickshaw, supports different renderer. This is a quick preview using the "bar" renderer. plotting is supported as well:

http://imgur.com/xUjK02h

> Re: the storage backend, not sure why you would choose acid-state here, as this basically rules it out for serious production use.

I'm well aware of the reasons why Doug choose zoom-cache in the first place, I wrote the rationale behind using acid-state at the hackaton in my blog post, so for saving some typing I'll redirect to the read of the full blog post. I do agree though that the current solution is probably hackaton-quality.

Alfredo