prometheus-pushgateway is flooding with warning messages

[hari.b...@gmail.com]

Configuration
-------------------

We have deployed a pod on k8s where prometheus and prometheus-pushgateway is configured . In another pod prometheus pusher (https://github.com/Showmax/prometheus-pusher) is sending metrics to prometheus-pushgateway.


Observation:
----------------

prometheus-pushgateway is overwhelming with messages like below:

time="2017-08-24T15:56:51Z" level=warning msg="Metric families 'name:\"go_memstats_sys_bytes\" help:\"Number of bytes obtained from system.\" type:GAUGE metric:<label:<name:\"instance\" value:\"host-level-ds-rh9wk\" > label:<name:\"job\" value:\"target_9126\" > gauge:<value:4.9817848e+07 > timestamp_ms:1503590202395 > ' and 'name:\"go_memstats_sys_bytes\" help:\"Number of bytes obtained by system. Sum of all system allocations.\" type:GAUGE metric:<label:<name:\"instance\" value:\"service1-4138284533-4xshv\" > label:<name:\"job\" value:\"target_9102\" > gauge:<value:1.1147512e+07 > timestamp_ms:1503590059331 > metric:<label:<name:\"instance\" value:\"service1-4138284533-cjzpd\" > label:<name:\"job\" value:\"target_9102\" > gauge:<value:1.1147512e+07 > timestamp_ms:1503590059714 > metric:<label:<name:\"instance\" value:\"service1-4138284533-gcqsf\" > label:<name:\"job\" value:\"target_9102\" > gauge:<value:1.1147512e+07 > timestamp_ms:1503590057324 > ' are inconsistent, help and type of the latter will have priority. This is bad. Fix your pushed metrics!" source="diskmetricstore.go:114"


versions:

prometheus: 1.7.0
prometheus-pushgateway:0.3.1

I appreciate if you have some pointers..

[Björn Rabenstein]

Hi Hari,

I have elided a few things from the log line so that you con see more
clearly what's going on:

On 24 August 2017 at 18:31, <hari.b...@gmail.com> wrote:
>
> 'name:\"go_memstats_sys_bytes\" help:\"Number of bytes obtained from system.\" [...] and
> 'name:\"go_memstats_sys_bytes\" help:\"Number of bytes obtained by system. [...]

> are inconsistent, help and type of the latter will have priority.

As you can see, the help string is slightly different for the two
metrics of the same name. The Prometheus data model requires that
metrics of the same name also have the same help string.

To get rid of the warnings, you have to fix your pushers to push the
same help string in both cases.

Note that nothing breaks badly with the current set-up as the latter
help string simply takes priority (as mentioned in the warning
message).

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[hari.b...@gmail.com]

Thanks Björn Rabenstein for helping me to understand the issue. What would be cause of two different types metrics with different help strings  to appear in the prometheus-gateway?  Is pusher is sending two different metrics to gateway?

[Björn Rabenstein]

On 25 August 2017 at 21:57, <hari.b...@gmail.com> wrote:
>
> What would be cause of two different types metrics with different
> help strings to appear in the prometheus-gateway? Is pusher
> is sending two different metrics to gateway?

Most likely two different pushing processes (for example different
binary versions) push slightly different metrics. You have to check
your pushers and see what exactly they are pushing.

[hari.b...@gmail.com]

Thanks for your inputs gave me some hints:

The problem is with metric name "go_memstats_sys_bytes" was published by a container telegraf and same metric was pushed StatsD go exporter. As you mentioned slightly different help values for same metric coming from two different sources.

1) Since this is default go runtime metric, I did ensure OS/go lang version for telegraf and statsD exporters are same.
2) Are there any way to turn off the default go_* metrics from StatsD Exporter.

Once again thanks for your help.
Hari

[Björn Rabenstein]

On 28 August 2017 at 23:52, <hari.b...@gmail.com> wrote:
>
> The problem is with metric name "go_memstats_sys_bytes" was published by a
> container telegraf and same metric was pushed StatsD go exporter. As you
> mentioned slightly different help values for same metric coming from two
> different sources.

For the record: Those metrics are not meant to be pushed to the
Pushgateway. I warmly recommend to read
https://prometheus.io/docs/practices/pushing/ .

Perhaps you are doing something for which you should better use a push
proxy? Look at https://github.com/RobustPerception/PushProx for
example.

> 1) Since this is default go runtime metric, I did ensure OS/go lang version
> for telegraf and statsD exporters are same.

The metric is coming from the Prometheus Go client library (
https://godoc.org/github.com/prometheus/client_golang/prometheus ). I
assume the StatsD exparter and Telegraf are each using different
versions of that library.

> 2) Are there any way to turn off the default go_* metrics from StatsD
> Exporter.

Yes. See https://godoc.org/github.com/prometheus/client_golang/prometheus#hdr-Advanced_Uses_of_the_Registry
(in particular last paragraph in that section).

It's probably easier to compile both binaries with the same version of
the Prometheus Go client. (And again, it's probably wrong in the first
place to blanket-push all metrics to the Pushgateway.)

[hari.b...@gmail.com]

Thanks for your tips/help. by disabling go_* metrics at go client library in the statsd exporter resolved the issue.