Hardware Sizing Guidelines of Opentsdb?

[hz...@qq.com]

Hi, all. 

Recently I am investigating the performance of opentsdb with different hadward sizing, such as different number of cores and different size of memory. 

And I found that the performance varies if I give too small memory for hbase and opentsdb. If I give small memory for hbase(1G for example), opentsdb will eat up the heap memory under load(maybe 10k points/s) and keep full gc and just hung there, with cpu running 100%.  If I give enough memory for hbase and opentsdb(24G for example), the performance(200k points/s) seems good and the cpu problem just went away.

So I just cannot decide the best hardware deployment if I want to use opentsdb in production environment. I need to know the maximum datapoints can be ingested when the cpu cores and memory limit vary. 

Is there any hardware sizing guidelines of Opentsdb just like the guideline for influxdb(https://docs.influxdata.com/influxdb/v1.1/guides/hardware_sizing/#general-hardware-guidelines-for-a-single-node), expecially the guideline for standalone deployment of hbase and opentsdb? 

[ManOLamancha]

On Wednesday, January 11, 2017 at 6:23:49 PM UTC-8, hz...@qq.com wrote:

Is there any hardware sizing guidelines of Opentsdb just like the guideline for influxdb(https://docs.influxdata.com/influxdb/v1.1/guides/hardware_sizing/#general-hardware-guidelines-for-a-single-node), expecially the guideline for standalone deployment of hbase and opentsdb? 

Hi, I don't have a good sizing guide for stand-alone as most instances are distributed, including our own. For our case we can run a TSD on a VM with 8 cores and 16GB JVM heap to achieve a throughput of 350K writes per second max. The large heap is mostly needed for caching UIDs in memory.

For HBase we run servers on bare-metal with 6 cores (12 threads) and 64GB heap so we have a large memstore and block cache for faster queries.

You can still run both on the same server in a stand-alone mode and try giving as much heap as you can to HBase with a smaller heap for OpenTSDB.