Suggestion on reducing cardinality

[Ramachandra Bhaskar Ayalavarapu]

We have some data points like this

{
  "__name__": "http_request_duration_seconds_bucket",
  "endpoint": "apssd",
  "instance": "10.10.77.218:8080",
  "job": "api-service",
  "le": "0.05",
  "method": "GET",
  "namespace": "foo",
  "pod": "bar",
  "route": "/localhsdssds/v1/project/<unique projectid>/provider/WmRESTProvider/blasdsdsd/fummssd/icon",
  "service": "api-service",
  "status": "4XX"
}

unfortunately though we understand its not best practice to add "route" in the label name which is driving cardinality we cannot do anything as team is not ready to change code. Can i put a first aid any way to reduce the cardinality. (this metric) is too important to drop

some thing like project wise etc by writing rewrite rule. if yes please advice how ?

[Stuart Clark]

You could drop that label in metrics relabling.

-- 
Stuart Clark

[Brian Candler]

Dropping that label would only work if all instances of that metric have the same value for that label - in which case, there wasn't a cardinality problem in the first place.  If there are distinct values, then dropping it will most likely end up with duplicate instances of the same timeseries, which will be rejected.

The only way I can think of working around this is to write some intermediate proxy which collects all the metrics, sums across the appropriate dimension(s), and exposes the results as a new metric.

But really, the answer is to push back.  You are getting junk metrics which cause the TSDB to explode; therefore either the metrics need to be fixed, or you simply stop collecting them.

[Tomoc Field]

If you only need part of the route, for example, the <unique projectid>, you could do a label replacement, then drop the original label.

    metric_relabel_configs:

      - source_labels: [route]
        regex: '.*/v1/project/([^/]+)/.*'
        target_label: "project_id"
        replacement: "$1"

      - action: labeldrop
        regex: ifDescr