GaugeVec populated at collection time?

[ryszar...@gmail.com]

Hi,
I need something like GaugeVecFunc – a GaugeVec that would be populated at collection time. What would be the easiest way to achieve something like that? I am trying to avoid setting the values in the GaugeVec at the moment when my values change, as it's happening very often, and the intermediate values are not that interesting.

Thanks,

-Ric

[Brian Brazil]

On 26 February 2016 at 17:36, <ryszar...@gmail.com> wrote:

Hi,
I need something like GaugeVecFunc – a GaugeVec that would be populated at collection time. What would be the easiest way to achieve something like that? I am trying to avoid setting the values in the GaugeVec at the moment when my values change, as it's happening very often, and the intermediate values are not that interesting.

How often is it changing? The client library is quite efficient, so if it's less than several hundred times per second it likely isn't worth worrying about and just using the usual interface. 

--

Brian Brazil

www.robustperception.io

[Ryszard Szopa]

I am afraid I am quite firmly in the several hundreds of operations per second. What matters more, however, is that generating the data I would like to put in the GaugeVec is in itself a kinda expensive operation. What is more, things that are now concurrent and independent would end up synchronizing on the metric, which is not really great. I am pretty sure be fine with the amortized cost if it happens once in a while at scraping time, but doing it for every request would require some very careful measuring on my side.

[Brian Brazil]

On 28 February 2016 at 16:56, Ryszard Szopa <ryszar...@gmail.com> wrote:

I am afraid I am quite firmly in the several hundreds of operations per second. What matters more, however, is that generating the data I would like to put in the GaugeVec is in itself a kinda expensive operation. What is more, things that are now concurrent and independent would end up synchronizing on the metric, which is not really great.

We've optimised this so that concurrency isn't a problem for all but the most extreme of use cases (and the Java client can even handle those well, as it's per-thread).

 

I am pretty sure be fine with the amortized cost if it happens once in a while at scraping time, but doing it for every request would require some very careful measuring on my side.

You'd be looking at a custom Collector then. At every scrape you'd send your ConstMetrics down the channel.

Brian

 

On Fri, Feb 26, 2016 at 6:38 PM Brian Brazil <brian....@robustperception.io> wrote:

On 26 February 2016 at 17:36, <ryszar...@gmail.com> wrote:

Hi,
I need something like GaugeVecFunc – a GaugeVec that would be populated at collection time. What would be the easiest way to achieve something like that? I am trying to avoid setting the values in the GaugeVec at the moment when my values change, as it's happening very often, and the intermediate values are not that interesting.

How often is it changing? The client library is quite efficient, so if it's less than several hundred times per second it likely isn't worth worrying about and just using the usual interface. 

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

[Ryszard Szopa]

On Sun, Feb 28, 2016 at 6:05 PM Brian Brazil <brian....@robustperception.io> wrote:

On 28 February 2016 at 16:56, Ryszard Szopa <ryszar...@gmail.com> wrote:

I am afraid I am quite firmly in the several hundreds of operations per second. What matters more, however, is that generating the data I would like to put in the GaugeVec is in itself a kinda expensive operation. What is more, things that are now concurrent and independent would end up synchronizing on the metric, which is not really great.

We've optimised this so that concurrency isn't a problem for all but the most extreme of use cases (and the Java client can even handle those well, as it's per-thread).

As I said, I am concerned about concurrency issues on my side, not Prometheus' :)

 

 

I am pretty sure be fine with the amortized cost if it happens once in a while at scraping time, but doing it for every request would require some very careful measuring on my side.

You'd be looking at a custom Collector then. At every scrape you'd send your ConstMetrics down the channel.

This looks like the way to go. Thanks a lot!

[Björn Rabenstein]

As a concluding remark: The GaugeVec (and other XxxVec) collectors mostly deal with efficiently picking the right individual Gauge for an update of its value (essentially what the With and WithLabelValues methods are doing).

If you want to copy out values from existing state on scrape time, you don’t need to find the individual gauges because you'll never update the value of a `Gauge` in a `GaugeVec`. Instead, all gauges will be created on the fly during scrape time.

You could write a custom collector as Brian suggested, but it’s also pretty easy to use NewGaugeFunc. Assuming your internal system has a concurrency-safe Get(key string) method, code could look like this (somewhere in the initialization part):

    keys := []string{"foo", "bar", "baz"}

    for _, key := range keys {
        prometheus.MustRegister(
            prometheus.NewGaugeFunc(
                prometheus.GaugeOpts{
                    Name:        "mirrored_metric",
                    Help:        "Mirrored from internal system.",
                    ConstLabels: prometheus.Labels{"key": key},
                },
                func() float64 { return internalSystem.Get(key) },
            ),
        )
    }

​

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg  | HRB 110657B

[Ryszard Szopa]

So, this is what I actually did: https://github.com/youtube/doorman/blob/master/go/server/doorman/collector.go, and it seems to do exactly what I wanted. The inspiration was this example: https://godoc.org/github.com/prometheus/client_golang/prometheus#example-Collector--Clustermanager. Note that  c.server.Status takes a lock (so I definitely want to call it only once), and the resource ids are not constant (so I don't think I would be able to do what Björn was suggesting).

[Brian Brazil]

On 29 February 2016 at 16:40, Ryszard Szopa <ryszar...@gmail.com> wrote:

So, this is what I actually did: https://github.com/youtube/doorman/blob/master/go/server/doorman/collector.go, and it seems to do exactly what I wanted. The inspiration was this example: https://godoc.org/github.com/prometheus/client_golang/prometheus#example-Collector--Clustermanager. Note that  c.server.Status takes a lock (so I definitely want to call it only once), and the resource ids are not constant (so I don't think I would be able to do what Björn was suggesting).

The _count suffix is usually used only by a Summary/Histogram, avoid it (similarly _sum and _total). It's also unusual to have the names of aggregates like "sum" in metric names, it's implied already. doorman_server_resources_assigned, doorman_server_resources_requested  and doorman_server_subclient_leases would be clearer.

You should be using ConstMetric rather than keeping the metrics around across scrapes.

I strongly recommend that metrics go in the same file as what they are monitoring. Spreading instrumentation across files just makes things harder to debug and maintain. Instrumentation should be treated as an integral part of your code, as you would logging. Having to register metrics in your main() function is also odd.

Brian

 

On Mon, Feb 29, 2016 at 3:27 PM Björn Rabenstein <bjo...@soundcloud.com> wrote:

As a concluding remark: The GaugeVec (and other XxxVec) collectors mostly deal with efficiently picking the right individual Gauge for an update of its value (essentially what the With and WithLabelValues methods are doing).

If you want to copy out values from existing state on scrape time, you don’t need to find the individual gauges because you'll never update the value of a `Gauge` in a `GaugeVec`. Instead, all gauges will be created on the fly during scrape time.

You could write a custom collector as Brian suggested, but it’s also pretty easy to use NewGaugeFunc. Assuming your internal system has a concurrency-safe Get(key string) method, code could look like this (somewhere in the initialization part):

    keys := []string{"foo", "bar", "baz"}

    for _, key := range keys {
        prometheus.MustRegister(
            prometheus.NewGaugeFunc(
                prometheus.GaugeOpts{
                    Name:        "mirrored_metric",
                    Help:        "Mirrored from internal system.",
                    ConstLabels: prometheus.Labels{"key": key},
                },
                func() float64 { return internalSystem.Get(key) },
            ),
        )
    }

​

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg  | HRB 110657B

--

Brian Brazil

www.robustperception.io

[Ryszard Szopa]

On Mon, Feb 29, 2016 at 6:01 PM Brian Brazil <brian....@robustperception.io> wrote:

The _count suffix is usually used only by a Summary/Histogram, avoid it (similarly _sum and _total). It's also unusual to have the names of aggregates like "sum" in metric names, it's implied already. doorman_server_resources_assigned, doorman_server_resources_requested  and doorman_server_subclient_leases would be clearer.

That's a very good point. I am trying to keep things consistent between different parts of the code and the documentation (so I'll stick to has and wants), but "sum" and "count" do seem unnecessary.

 

You should be using ConstMetric rather than keeping the metrics around across scrapes.

OK, so that would mean creating the ConstMetrics in both Collect and Describe, right? What would be the benefits of that? Is there any example I could look at?

 

I strongly recommend that metrics go in the same file as what they are monitoring. Spreading instrumentation across files just makes things harder to debug and maintain. Instrumentation should be treated as an integral part of your code, as you would logging. Having to register metrics in your main() function is also odd.

I have tests which create multiple servers, and Prometheus would (rightfully) complain about registering duplicate metrics. But, again, you are raising a good point. Exporting NewCollector also doesn't seem so great. So, yet another wrapper function it will be.

Thanks a lot for looking at my code!

-Ric

[Brian Brazil]

On 29 February 2016 at 17:49, Ryszard Szopa <ryszar...@gmail.com> wrote:

On Mon, Feb 29, 2016 at 6:01 PM Brian Brazil <brian....@robustperception.io> wrote:

The _count suffix is usually used only by a Summary/Histogram, avoid it (similarly _sum and _total). It's also unusual to have the names of aggregates like "sum" in metric names, it's implied already. doorman_server_resources_assigned, doorman_server_resources_requested  and doorman_server_subclient_leases would be clearer.

That's a very good point. I am trying to keep things consistent between different parts of the code and the documentation (so I'll stick to has and wants), but "sum" and "count" do seem unnecessary.

 

You should be using ConstMetric rather than keeping the metrics around across scrapes.

OK, so that would mean creating the ConstMetrics in both Collect and Describe, right? What would be the benefits of that? Is there any example I could look at?

You only have to create a Descriptor in Describe, and you can share those. It's better to use ConstMetrics as they work better for custom collectors (doesn't quite make a difference in this case as you're going locking anyway, got the resetting, and are only using gauges).

 

 

I strongly recommend that metrics go in the same file as what they are monitoring. Spreading instrumentation across files just makes things harder to debug and maintain. Instrumentation should be treated as an integral part of your code, as you would logging. Having to register metrics in your main() function is also odd.

I have tests which create multiple servers, and Prometheus would (rightfully) complain about registering duplicate metrics. But, again, you are raising a good point. Exporting NewCollector also doesn't seem so great. So, yet another wrapper function it will be.

I'd suggest doing the registration in the init() function of the file.

Brian

--

Brian Brazil

www.robustperception.io

[Björn Rabenstein]

On 29 February 2016 at 18:52, Brian Brazil <brian....@robustperception.io> wrote:

OK, so that would mean creating the ConstMetrics in both Collect and Describe, right? What would be the benefits of that? Is there any example I could look at?

You only have to create a Descriptor in Describe, and you can share those. It's better to use ConstMetrics as they work better for custom collectors

I’d use one descriptor for each current GaugeVec, but then don’t use GaugeVec but the following:

const (
    namespace = "doorman"
    subsystem = "server"
)

var (
    labelNames = []string{"resource"}
    hasDesc    = prometheus.NewDesc(
        prometheus.BuildFQName(namespace, subsystem, "has"),
        "All capacity assigned to clients for a resource.",
        labelNames, nil,
    )
    wantsDesc = prometheus.NewDesc(
        prometheus.BuildFQName(namespace, subsystem, "wants"),
        "All capacity requested by clients for a resource.",
        labelNames, nil,
    )
    clientsDesc = prometheus.NewDesc(
        prometheus.BuildFQName(namespace, subsystem, "clients"),
        "Number of clients requesting this resource.",
        labelNames, nil,
    )
)

func (c *collector) Describe(ch chan<- *prometheus.Desc) {
    ch <- hasDesc
    ch <- wantsDesc
    ch <- clientsDesc
}

func (c *collector) Collect(ch chan<- prometheus.Metric) {
    status := c.server.Status()
    c.mu.Lock()
    defer c.mu.Unlock()

    for id, res := range status.Resources {
        ch <- prometheus.MustNewConstMetric(
            hasDesc,
            prometheus.GaugeValue,
            res.SumHas,
            id,
        )
        ch <- prometheus.MustNewConstMetric(
            wantsDesc,
            prometheus.GaugeValue,
            res.SumWants,
            id,
        )
        ch <- prometheus.MustNewConstMetric(
            clientsDesc,
            prometheus.GaugeValue,
            res.Count,
            id,
        )
    }
}

With the reworked Go client I’m currently working on, the above will be a bit less clunky, but the essential semantics stays the same, i.e. if you only update metric values during scrape, use metrics created on the fly (with MustNewConstMetric). You don’t have to reset them, and you collect them while you create them.

​

[Ryszard Szopa]

Thanks Björn! This is simpler than what I had, and exactly what I needed.