How do i handle metrics with variable labels?

[pa...@portworx.com]

We currently use the Prometheus golang client to present metrics to a Prometheus endpoint.

I've recently had an ask from our customers to report these metrics with user-specified labels so they can filter the stats better.

I'm having some serious issues getting this to work. I've seen 2 cases.

1. The larger label map gets created first and a subset will work.

      Example: 

       volume_read_bytes{volumeid="<id>",nodeid="<id>",environment="production",app="mysql",volumename="mysqlvolume"}

       volume_read_bytes{volumeid="<id>",nodeid="<id>",volumename="othervolume"}  

1. metric without labels gets created first and then a user creates a new object with labels and I can't get these to be published.

       volume_read_bytes{volumeid="<id>",nodeid="<id>",volumename="othervolume"} 

       #This volume is now not visible because GetMetricWith(labels)  won't return the existing metric because my labels are not matching.

       // volume_read_bytes{volumeid="<id>",nodeid="<id>",environment="production",app="mysql",volumename="mysqlvolume"} //

Is there a workaround for this?
Delete won't work because labels will forever be not matchy for the bigger label map.

Is there a recommended workflow for this?

My code to set a metric is as follows

func (p *PrometheusSink) setMetricValue(subsystem string, name string, val float64, labels map[string]string) {
    p.mu.Lock()
    defer p.mu.Unlock()
    key := p.normalizeKey(name)
    g, ok := p.gauges[key]

    if !ok {
        lbl_keys := make([]string, 0, len(labels))
        for k := range labels {
            lbl_keys = append(lbl_keys, k)
        }
        g = prometheus.NewGaugeVec(prometheus.GaugeOpts{
            Namespace: "app",
            Subsystem: p.normalizeKey(subsystem),
            Name:      key,
            Help:      key,
        }, lbl_keys)

        p.Registry.Register(g)
        p.gauges[key] = g
    }

    gauge, err := g.GetMetricWith(labels)
    if err == nil {
        gauge.Set(val)
        p.updateGauageTs(g, gauge, labels)
    }
}

[Ben Kochie]

Metrics should never contain target labels. Metrics should only contain labels that identify themselves inside the same exporter.

Adding job/node/environment/etc labels on exporters is an extreme anti-pattern.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c05ff878-20f2-414a-824d-67e586626b27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pa...@portworx.com]

So if I described well-known labels and just pass them as empty by default I would solve this issue correct?

The reason we need this is as follows.

We provided users with persistent storage for their containers/pods (depending on the schedule they use)

They can carve out volumes on a request basis and use them in their containerized applications.

Now, these users want to measure the performance of their application with our product as the underlying storage layer.

It's near impossible for these users to do so if they can't somehow filter the returned metrics down to the requested application/region etcetera.

How would you expect them to do such workflows if at all?