KairosDB memory issues when querying.

[swetha kasireddy]

Hi,

I see KairosDB memory issues when I query data at the same time when data is getting inserted. The data getting inserted is around 11k datapoints per second. What is the best way to avoid getting such error?

Status Code: 500
Status: Server Error
Return Value: {"errors":["org.kairosdb.util.MemoryMonitorException: Insufficient memory to perform request"]}

[Loic Coulet]

Hi,
11k/second push is fine with Cassandra even on slow servers.

• What database are you using? please check you're not using H2.
• What version of KairosDB are you using?
• What is the available memory on the server and for KairosDB JVM ?
• How many tags/tags values do you have for the metric you are querying?
  
• And what is the query you are performing? the JSON query would be nice to have to help you.

[swetha kasireddy]

Hi,

We use Cassandra and  KairosDB 0.9.5. The memory available is 8g on both Cassandra and Kairos. I use 2 nodes for Cassandra and 2 nodes for kairos with  a replica of 2.  There is just a single tag. Following is the example of a query that we are using.

{

  "metrics": [

    {

      "tags": {},

      "name": "TEST.exp.var.ALL.FM.view1.COUNT.batch"

    }

  ],

  "cache_time": 0,

  "start_relative": {

    "value": "24",

    "unit": "hours"

[Loic Coulet]

Weird, please make sure you give enough heap size to  java, for example by uncommenting the lines in kairosdb-env.sh that would provide 4G heap size.

If you did almready, did you change settings for KairosDB cache size?

The KairosDB log file may be useful to investigate.

Loic

[swetha kasireddy]

Hi,

Which line are you taking about?  I don't see anything like that in my kairosdb.sh.

Thanks,

Swetha

On Tuesday, October 20, 2015 at 2:40:41 PM UTC-7, swetha kasireddy wrote:

[Loic Coulet]

not the kairosdb.sh but the kairosdb-env file :)

make sure you have no # in front of the lines, and BTZ you may increase to 2 or 4G:

JAVA_OPTS="$JAVA_OPTS -Xmx2G"  # Max heap size
JAVA_OPTS="$JAVA_OPTS -Xms2G"  # Min heap size

Loic

[swetha kasireddy]

I don't see any kairosdn_env.sh there. I just see the following.

drwxrwxrwx 2 app app 4096 Jul 30  2014 .

drwxr-xr-x 8 app app 4096 May 11 23:22 ..

-rwxrwxrwx 1 app app  728 Aug 20  2013 check-log-errors.sh

-rwxrwxrwx 1 app app 1278 Aug 20  2013 create_hbase_table.sh

-rwxrwxrwx 1 app app  829 Jun 18  2014 kairosdb.bat

-rwxrwxrwx 1 app app  889 Aug 21  2013 kairosdb-service.sh

-rwxrwxrwx 1 app app 1617 Jul 30  2014 kairosdb.sh

On Tuesday, October 20, 2015 at 2:40:41 PM UTC-7, swetha kasireddy wrote:

[Erol Merdanović]

Which version are you using and how did you install KairosDB?

[swetha kasireddy]

I have kairosdb-0.9.4-6 and I followed the instructions as given in Getting started as shown below. User  ./kairosdb.sh start to run KairosDB.

https://kairosdb.github.io/docs/build/html/GettingStarted.html   

On Tuesday, October 20, 2015 at 2:40:41 PM UTC-7, swetha kasireddy wrote:

[swetha kasireddy]

Hi Erol,

What is the appropriate setting for  KairosDB cache size?

Thanks,

Swetha

On Tuesday, October 20, 2015 at 2:40:41 PM UTC-7, swetha kasireddy wrote:

[Erol Merdanović]

Honestly, no idea. I think Brian or Loic can give better answer.

[Brian Hawkins]

First off you need to get a newer version of Kairos.  I think every version since the one you have I've added something to make queries use less memory.  The kairosdb-env.sh file may not be in that version either.  It lets you set jdk memory sizes.

Brian