1.3.0 experience

[Dirk Grobler]

Hey,

I've just thought this might be of interest for some of you. We are an IoT company and have been using KDB on Cassandra for 5 years now.

Towards the end of last year our Cassandra cluster with 250GB storage nodes was hitting the 70% usage rate, so a good time to think about an upgrade. 

As we are an IoT company, we don't really need ms granularity of data. Some metrics are updated every minute, the majority are updated every 15 or 60 minutes. It got us really excited, when we heard that second granularity is available in version 1.3 now. Hence we decide to perform an upgrade from 1.2.2 to 1.3.

The upgrade itself went pretty smooth. First we updated the KDB nodes to 1.3 without changing the schema. Was a bit trail and error as the config has changed, but we got there. After checking that all applications run well, we decided to move to the next step of the migration. 
We've setup a new Cassandra cluster also with 250GB nodes. Schema with a second granularity. We set the row width to 2 years, because that's what most of our customers usually see. 
We first run both clusters in parallel and then migrated the data from the old to the new cluster. A few steps have been involved here, but I won't go into all the detail.

After the migration we ran a number of checks to verify that all data had come across. 
And then we looked at the net effect of the migration.

• Disk usage: Old 70%, New 41% (total 250GB)
• End to end query comparison of our most often used metrics, end to end test from our own api:
  • Temperature data per device: 1 months: old 1.8 sec, new 1.8sec
  • Temperature data per device: 1 year: old 8.2sec, new 2.2 sec
  • Temperature data per device: 2 years: old 14.8sec, new 3.28 sec

This is exactly what we have hoped for. We can utilise the same hardware for quite a bit longer and get much better query results. 

Great job, Brian and team. We are certainly very happy users of KDB. Keep up the good work.

-Dirk

[Brian Hawkins]

Hey I appreciate the comments.  This is exactly why I do it.  I love to make useful software that help people become more productive.

Did you change compaction strategies in the new cluster or was C* basically setup the same?

Thanks

Brian