Difference between Add and Push

[Achintya Kumar]

Hey Guys,
I am using github.com/prometheus/client_golang/prometheus/push in my Golang code to push metrics to a push gateway.
But I wanted to understand the core difference between Add and Push. I am currently using Add to push the metrics but still have some confusions regarding which previous metrics will it replace? Also, I want to use the pusher.Add function to be run after every 10 seconds rather than pushing metrics right away. How can I achieve that? The prometheus/push package doesn't have this kind of functionality.
Thank you.
This message and its attachments are confidential (or legally privileged) information and are meant solely for the addressee of such message. Any unauthorized use of the message / its attachments is strictly prohibited.

[Julius Volz]

Hi Achintya,

On Tue, Jan 10, 2023 at 9:29 AM 'Achintya Kumar' via Prometheus Users <promethe...@googlegroups.com> wrote:

Hey Guys,
I am using github.com/prometheus/client_golang/prometheus/push in my Golang code to push metrics to a push gateway.
But I wanted to understand the core difference between Add and Push. I am currently using Add to push the metrics but still have some confusions regarding which previous metrics will it

Add() will replace only any metrics with the same metric name and labels (so the same time series identity) within a pushed group, if metrics with the same name and labels have been pushed before. In contrast, Push() completely removes all previous metrics in a pushed group and then only adds the newly pushed ones. So you can use Add() instead if you are not sending *all* possible metrics to a group every time, and you don't want the old ones to disappear.

 

replace? Also, I want to use the pusher.Add function to be run after every 10 seconds rather than pushing metrics right away. How can I achieve that? The prometheus/push package doesn't have this kind of functionality.

If you really want that you'll have to implement that yourself by just having a goroutine running in the background that pushes your metrics every 10 seconds. Just be aware of the general limitations of what the Pushgateway should be used for at https://prometheus.io/docs/practices/pushing/. Most of the time it's just used to push metrics once, at the end of a batch job run, but in some cases it can also make sense to push metrics on a more ongoing basis, like if you want to report on the progress of an ongoing batch job while it's running.

Julius

 

Thank you.
This message and its attachments are confidential (or legally privileged) information and are meant solely for the addressee of such message. Any unauthorized use of the message / its attachments is strictly prohibited.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a1df943c-1146-4c39-9983-f077952d1f28n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Achintya Kumar]

Hey Julius,
Thank you for your reply. Understood the difference between Add and Push.
Regarding the goroutine thing, I was trying something like this 

pushGateWayClient := push.New(endpoint, jobName)

ticker := time.NewTicker(10 * time.Second)
   go func() {
       for {
           select {
           case <-ticker.C:
               if err := pushGateWayClient.Add(); err != nil {
                   log.Error("Could not push to push gateway ", zap.Error(err))
               }
           }
       }
   }()

But this isn't working. I can not see the metrics when verifying on grafana. Can you help here?

[Julius Volz]

Are you adding any metrics registry to the Pusher, using the Gatherer() method? https://pkg.go.dev/github.com/prometheus/client_golang/prometheus/push#Pusher.Gatherer

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d86405bd-2fa0-4582-b1e2-7da4ab47bb3bn%40googlegroups.com.

[Achintya Kumar]

No, not using Gatherer() method. Here's my pastebin link for the code. 
Also, I've created different maps to store the previously generated collectors because earlier I was getting the "Duplicate metric collector registration attempted". This was solved using maps.

[Julius Volz]

You *will* need to use the Gatherer() method to register some metrics with your Pusher, otherwise that explains why nothing is being pushed. See this example: https://pkg.go.dev/github.com/prometheus/client_golang/prometheus/push#example-Pusher.Add

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cef48dee-05c4-45d5-91fb-921e51c631b9n%40googlegroups.com.

[Achintya Kumar]

Understood. I used Gatherer() in my code now. But seeing some black spaces in graph. What can be the issue here? These both are counter metrics.

[Julius Volz]

Well that's odd. Let's go through some things:

* The two metrics have gaps at different points in time, so it can't be an issue with Prometheus not being able to scrape the Pushgateway itself

* Other than that, series appearing/disappearing from a scrape endpoint can trigger these kinds of gaps due to staleness handling, but if you're using Add(), no series should disappear, only existing ones should get replaced with new values.

Or are you using Push() somewhere else in your app, or clearing out an entire metrics group in the Pushgateway once in a while?

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e918fe7b-c698-4233-9eab-43fc55b524f8n%40googlegroups.com.