Trying to improve ratio collection using by and project combo with many by partitions...

[Carl Sandland]

Hello, 

Nice to find a group for riemann, I have a rule that boils down to;

(by :this)(by :that)(project [preds] children ... reinject) 

this approach uses a lot of memory as the number of :this and :that values grows (1000's).

the point of it all is to calculate some per 'scope' ratio metrics for cassadra scopes.

Been thinking maybe this is not a good approach, any ideas for a different way? I'm thinking of maybe doing it outside of riemann somehow but want ideas.

Thanks for any help, promise to share if I figure it out.

Cheers,

Carl

[Sanel Zukan]

Hi,

Carl Sandland <carl.s...@instaclustr.com> writes:
> Hello,
>
> Nice to find a group for riemann, I have a rule that boils down to;
>
> (by :this)(by :that)(project [preds] children ... reinject)
>
> this approach uses a lot of memory as the number of :this and :that values
> grows (1000's).

(by) is expensive for values that grows [1] and is more suitable for
something "fixed", like hostnames or service/metric names.

> the point of it all is to calculate some per 'scope' ratio metrics for
> cassadra scopes.
>
> Been thinking maybe this is not a good approach, any ideas for a different
> way? I'm thinking of maybe doing it outside of riemann somehow but want
> ideas.

Without more details, I'll try with something, but it might be a total
nonsense :)

X is a number of seconds events will be aggregated and sent down the
stream:

(by [:service :host]
(coalesce X
(fn [events]
;;
;; Assumed every event comes with :scope, so event structure
;; looks like: {:host "Foo" :service "bar" :scope "abc" ...}
;;
;; Now group events by :scope, using clojure 'group-by'. 'mps' is
;; a vector of maps with 'scope' scope.
;;
(doseq [[scope mps] (group-by :scope events)]
(let [calculated-whatever (reduce ... mps ...)]
(inject-riemann-event-for-scope scope calculated-whatever))))))

Also, instead of reinjecting events back to Riemann core, you can just
pass them down to the children, which is, I think more efficient. Also
use 'where' to limit metrics, I assume not all of them has :scope.

You can wrap all of this in a clojure function, like:

(defn by-scope [N & children]
(where (and (service #"^Magic-*")
(not (expired? event)))
(by [:service]
(coalesce N
(fn [events]
(doseq [[scope mps] (group-by :scope events)]
(let [calculated ... ;; use reduce/map to calculate on mps
tmp-event (first mps)] ;; get one event as a template
(call-rescue (assoc event :scope scope :ratio calculated) children))))))))

;; aggregate metrics every 10 seconds, calculate something
;; on them and emit a single metric with :scope and :ratio and store
;; that in Cassandra

(by-scope 10 cassandra)

> Thanks for any help, promise to share if I figure it out.
>
> Cheers,
> Carl

Best regards,
Sanel

[1] http://riemann.io/api/riemann.streams.html#var-by

[Carl Sandland]

Thanks for the ideas (which is what I was looking for) and the time to send such a detailed response, it's appreciated!

Some brief background; I've 30+ years coding but just 1 in clojure/riemann, and it's been a real learning curve for me. I'm helping extend a rather complex (sophisticated??) riemann config. This config seems good enough, mostly thanks to how awesome riemann is, but is faced with ever growing dimensions of growth: raw number of events, their increasing level of detail from an increasing amount of different services and growth in the number of everything (we monitor large cloud clusters).

So we're in a phase now where riemann is being optimised and tuned (I think it's fair to say that most contributors are at my level of experience with riemann - enthusiastic but not expert). I also sometimes think of moving more complex derived metrics out of riemann and into the nodes: so trying to find a sweet spot for where stuff scales better (and again the scale is huge).

I wanted to say thanks and give the background in a timely manner, but it WILL take me some time to properly digest and play with your ideas. I've built a rather nice test riemann test harness around the JVM heap/time impacts of this problem and have seen first hand the explosion in heap caused by our current approach. Our rule was ok when (by :host)(by :keyspace)(by: scope (ie cf names)) involved [0-1000] [0-5] [0-10] but it doesn't scale to [0-??][0-XXXX][0-XXXX]. I guess some cassandra people really do need 1000's of keyspaces and even more column families ;)

One thing I've tried is just simply; (the end goal here is ratio = p1 / p2 for each a/b/c)

(by :a)(by :b)(by :c)(project [p1 p2])(reinject) => (by a: b: c:)(project [p1 p2])(reinject)

pretty sure that's logically the same and it's had a small reduction to heap usage.

I have a few other 'low hanging fruit' things to try but I think I'm limited, so I will study your post more and report back in a few months (lol). As I said there is 'thought pressure' to remove stuff like this out of riemann config and into the node collection layer.

I love riemann (in case that didn't come through); but I find the step from noob => intermediate quite difficult to navigate (I guess it's easy to start a helicopter, but harder to fly it into combat?)

Cheers,
Carl

[Carl Sandland]

can confirm;

- we are definitely 'guarding' using (where) as aggressively possible.

- we are using (reinject) all over the place; there may be some scope to carefully 'copy and paste' only the relevant child streams to replace the reinject ?

I'm pondering the reinjects, I guess it allows for streams to be 'uncoupled' more ?

Cheers,

Carl

[Carl Sandland]

Hello,

Played around with replacing reinject with direct child streams and the payoff was noticeable more in cpu than jvm-heap usage. I guess the payoff depends on how carefully 'routed' or 'partitioned' your head stream is ? Also have some views that replacing one reinject would mean following the same pattern in the other 99 ! It's true i guess that you should have to think about wether this event goes through the head or not each time...

Cheers,
Carl

[Sanel Zukan]

Hi,

Carl Sandland <carl.s...@instaclustr.com> writes:
> Played around with replacing reinject with direct child streams and the
> payoff was noticeable more in cpu than jvm-heap usage. I guess the payoff
> depends on how carefully 'routed' or 'partitioned' your head stream is ?

Yes. Probably you have the reason to use 'reinject', but if streams are
properly ordered, you won't have to. Unless you have some complex
usecase.

> Also have some views that replacing one reinject would mean following the
> same pattern in the other 99 ! It's true i guess that you should have to
> think about wether this event goes through the head or not each time...

It depends how you are ordering the flow. I like approach from [1],
where streams are divided by some logic, but follows linear
route. Example:

(def storage
"Multiple storage locations."
(sdo
influx
cassandra))

(def cpu-checks
"Checks CPU usage."
(where (and (service #"^cpu-check")
(> metric 90))
(by [:host :service]
send-alert)))

(streams
cpu-checks
memory-checks
(precalculate-something storage)
storage)

"precalculate-something" will do some calculations on metric group and
store it in databases. Those "*-checks" doesn't need to store anything,
they will just alert in case something doesn't look right. If I need to
have some check/alert on precalculated values, I'd add:

(precalculate-something a-check storage)

and there is a clear visual distinction how event flows and where it
ends up.

I find this easier to debug, disable (just comment offending line) or to
explain to coworkers that are not familiar with Riemann.

> I also sometimes think of moving more complex derived metrics out of
> riemann and into the nodes:

This is also a good approach.

> One thing I've tried is just simply; (the end goal here is ratio = p1 /
> p2 for each a/b/c)
> (by :a)(by :b)(by :c)(project [p1 p2])(reinject) => (by a: b: c:)(project
> [p1 p2])(reinject)
>
> pretty sure that's logically the same and it's had a small reduction to
> heap usage.

Every 'by' will create a new table with events, that will be present in
memory while Riemann is running, so I'd go with '(by [:a :b :c] ...)`. I
think this is the reason why you see small reduction to heap usage.

> I love riemann (in case that didn't come through); but I find the step
> from noob => intermediate quite difficult to navigate (I guess it's easy to
> start a helicopter, but harder to fly it into combat?)

I think the major challenge with Riemann is trying to see it as a standard
monitoring tool. If you think about it as a data router and everything
else is a side-effect of that (saving, alerting) and always keep in mind
how events flow, things will be easier to follow. At least that worked
for me :)

> Cheers,
> Carl

Best,
Sanel

[1] https://github.com/mcorbin/riemann-configuration-example

[Carl Sandland]

Hi,

As promised a follow up on what worked/didn't work;

1. adding more strict 'partitioning' (routing) of events by service through the main head stream.
2. tried replacing reinjects with direct child streams and the perf was better; unfortunately design arguments about this stopped it for now.
3. replaced (re-find p s) with a (def some-p re-pattern p) then (re-find some-p s) : unsure of impact but it didn't hurt.
4. additional care with type annotations (eg ^String) and logic in 'inner loop' child streams.
5. went back to using direct java string ops in places rather than clojure ones (this seemed to help).
6. played around with (coalesce) a fair bit and am still a bit perplexed by the events that come out of it; it seems more powerful than I understand yet.

We managed to achieve a big improvement with the above; in average and .999 latencies (not always easy to get both!).

The original question is largely the same today as it was for the actual quotient calcs, but I did clean it up/optimise that path in other ways.

My overall gut take; calculating quotients over terms in multiple 'services' is a surprisingly nuanced problem :)

Thanks for the help.

Regards,
Carl