3-nodes Cluster Optimization for HBase / OpenTSDB

[Mykola Ulianytskyi]

Hi

I am planning to create small 3-nodes cluster for OpenTSDB:

- HDFS NameNode: node1

- HDFS SNameNode: node2

- HDFS DataNode: all nodes

- HBase Master: node1

- HBase RegionServer: all nodes

- Zookeepeer: all nodes

- OpenTSDB: all nodes

- Cloudera Manager: node3

Node Spec: 8+ core, 32 or 64 GB RAM, RAID10

Please help me to find recommended Java Heap Sizes for:

- HBase Master

- HBase RegionServer

- OpenTSDB

- HDFS NameNode

- HDFS DataNode

- HDFS Balancer

for two hardware spec variants (32 and 64 GB RAM).

Any additional tweaks are appreciated.

Current OpenTSDB database size: 500 GB

Current OpenTSDB Client Connections: 3000+

OpenTSDB Tweaks:

- compression: lzo

- tsd.core.auto_create_metrics = true

- tsd.storage.fix_duplicates = true

- tsd.core.uid.random_metrics = true

- tsd.storage.hbase.prefetch_meta = true

- tsd.storage.salt.width = 1

- tsd.storage.salt.buckets = 3

Other Tweaks:

- THP: disabled

- EXT4: noatime, nodiratime

- vm.swappiness = 0

- net.core.somaxconn = 1024

- vm.min_free_kbytes = 1048576

- vm.zone_reclaim_mode = 0

- hard/soft nofile limit: 65000

Thank you

[ManOLamancha]

On Wednesday, March 1, 2017 at 2:50:17 AM UTC-8, Mykola Ulianytskyi wrote:

Please help me to find recommended Java Heap Sizes for:

- HBase Master

- HBase RegionServer

- OpenTSDB

- HDFS NameNode

- HDFS DataNode

- HDFS Balancer

for two hardware spec variants (32 and 64 GB RAM).

I'm not too sure about HDFS but the HBase Master doesn't need much heap, probably 2 to 8 GB. DataNodes only need a G or up to 4 if I recall. The TSDs shouldn't need very much unless you have really high cardinality tags. Try around 6 yo 8G there. Then give the rest (with enough for the OS) to the region servers. Set your memstore and block cache sizes fairly high.