CheckRate for MemoryMonitor

[JavaDeveloperKairos]

Hi Brian and team,

I had a quick questions with regards to how KairosDB behaves in low memory conditions.

It seems that we had always initialized the MemoryMonitor with checkrate as 20 

https://github.com/kairosdb/kairosdb/blob/master/src/main/java/org/kairosdb/datastore/cassandra/CassandraDatastore.java#L302

> MemoryMonitor mm = new MemoryMonitor(20);

In earlier version of KairosDB (i.e. 0.9.x up until  v1.1.3 ), we then did the explicit dialing down of checkrate to 1  right before the running QueryRunner

v0.9.4 https://github.com/kairosdb/kairosdb/blob/v0.9.4/src/main/java/org/kairosdb/datastore/cassandra/CassandraDatastore.java#L549

v1.1.3 https://github.com/kairosdb/kairosdb/blob/v1.1.3/src/main/java/org/kairosdb/datastore/cassandra/CassandraDatastore.java

//Changing the check rate
mm.setCheckRate(1);

but then removed the above code in v.1.2.0.

Is there any reason for this change? 

The way I understand it was that Memory Monitoring is based on the detecting setCollectionUsageThreshold()settings on a threshold  for old Gen space when the memory is low right after a GC (not otherwise) which tells us that we have got to that point GC was unable to recover memory. So, a checkrate of 1 would mean that it would stop all threads. 

Am I missing something?

[JavaDeveloperKairos]

Hi @Brian or someone who has the idea - please comment on the below question

Basically, we are using an older version and were thinking about what happens if we don't dial down the checkrate to 1.

[Brian Hawkins]

This whole mechanism is based on the assumption that checking memory usage is expensive.

So we bother understand how it works.  The check rate controls how often the memory is actually checked when a call to checkMemoryAndThrowException is called.  So if the check is in a loop that has the potential to add very little memory then the check rate is set higher - or lower depending on how you look at it.  In some places I have the check rate set to 1000, like in the code that adds a data point to the search results.  The cached search results only checks memory after every 1000 data points have been added.

So as in the code snip you shared if the check has the potential of adding a lot of memory between checks I set the checkRate to 1, so every time an actual memory check is made.

Between 1.1.3 and 1.2.0 is that I switch over to using CQL.  One of the many benefits of using CQL is that it pages data to the client instead of sending it all in one big chunk.  So my risk of running out of memory between checks is much lower so I can check less often and thus the setCheckRate(1) was removed in the refactoring of code.

With that said if you are experiencing an OOM at some place where the checks should be changed please let me know and we can update the code in future releases.

Thanks

Brian