What determines index size

[Vipin Menon]

Hi All,

I see a metric "riemann index size" and I'm trying to understand what it is.

How does this size change? I tried going through the code and saw that it points to the ".size" of the nonblocking HashMap. Suffice to say my clojure knowledge isnt good enough to understand beyond that. I see that in our setup it shows a size of 3 occasionally 1, but now I'm trying to understand what causes the size change. Could somebody kindly explain this please :) 
Much appreciated.

- Thanks

[Toby McLaughlin]

Hi Vipin,

The ".size" of the index is the number of records currently stored. It's a Java method on the NBHM object (which is why it has the dot at the start).

https://www.cs.rice.edu/~javaplt/javadoc/concjunit4.7/org/cliffc/high_scale_lib/NonBlockingHashMap.html#size()

The index stores one record for each unique combination of :host and :service that gets inserted into it, and things are inserted into it by calling the index in your config, often like this:

(let [index (index)]
  (streams
   (where (service #"service of interest")
          index)))

There are several reason you might want to put events (and thus their host/service pairs) in the index. The most commons ones for me are:

1. Indexed events can "expire", so you can detect when a host/service stops sending events.

When events expire, they are removed from the index, and thus the measured size will go down.

https://github.com/riemann/riemann/blob/223b2ef9bc483e52bd9daedc1050118df94e30b2/src/riemann/index.clj#L85

2. Riemann Dash works by querying the index, so if you run that, you can only visualize things that you put in the index via your config.

3. You can query the index yourself with a Riemann client. I don't make much use of this yet, but I have grand plans!

Hope that helps,

Jarpy.

[Fabien Wernli]

Just a clarification on 2. : Riemann Dash doesn't query the index continuously, it connects to the websocket server. It does require the index indeed, just like the tcp server which handles the ad-hoc queries