Using OpenTSDB to store metrics from Kubernetes / Prometheus

[wvl]

Hi,

I'm experimenting with the following setup:

Google Cloud Container Engine / Kubernetes get their metrics endpoint scraped by Prometheus, which then uses the remote storage feature to push metrics for long term storage into OpenTSDB, with BigTable as the backend.

Sadly, it seems that the default metrics provided by Kubernetes, container far more than 8 tags. I've already tried upping the limit to 10, only to find I need more. From the logs:

WARN  [OpenTSDB I/O Worker #1] PutDataPointRpc: Too many tags: 16 maximum allowed: 10, tags:
{svc_component=apiserver, instance=x.x.x.x_.443, minor=6, goVersion=go1.7.5, buildDate=2017-04-19T20_.22_.08Z, platform=linux/amd64, endpoint=https, gitVersion=v1.6.2, gitCommit=477efc3cbe6a7effca06bd1452fa356e2201e1ee, major=1, service=kubernetes, svc_provider=kubernetes, namespace=default, compiler=gc, job=kubernetes, gitTreeState=clean}:

metric=kubernetes__build__info ts=1494580110 value=1 svc_component=apiserver instance=x.x.x.x_.443 minor=6 goVersion=go1.7.5 buildDate=2017-04-19T20_.22_.08Z platform=linux/amd64 endpoint=https gitVersion=v1.6.2 gitCommit=477efc3cbe6a7effca06bd1452fa356e2201e1ee major=1 service=kubernetes svc_provider=kubernetes namespace=default compiler=gc job=kubernetes gitTreeState=clean

How concerned should I be when raising the tag limit and what does one think about this setup? 

If I am unable to raise the tag count, while keeping OpenTSDB responsive, then it basically will not be usable to monitoring Kubernetes.

[ManOLamancha]

It's fine to have the tag count much higher, we have Kubernetes and similar container platforms reporting up to 30 tags in our TSDB instance. 

BUT the most critical thing for you is to make sure to set the tag value UID size to 5 or 6 bytes as container services will eat up your UIDs quickly (e.g. the build dates and git commits above).