using pushgateway metric with other metrics returns empty result

[master magra]

Hi All,

I've started looking into using pushgateway metrics.

I tried a variation of the curl example found in the pushgateway git readme:

cat <<EOF | curl --data-binary @- http://my-prometheus-host:9091/metrics/job/some_job/instance/some_instance

# TYPE another_metric gauge

# HELP another_metric Just an example.

my_metric 6324321

EOF

I can see the metric appear and a basic query returns the expected value.

I can also apply basic arithmetic to it and get the expected result e.g.

my_metric + 100 returns 6324421 

However, if i try to use it in arithmetic with other existing gauge metrics it always returns an empty result e.g.

my_metric / existing_gauge_metric returns empty query result.

Any ideas what is going on?

Thanks!

[Chris Siebenmann]

It's likely that these two metrics have different labels. Normally,
the labels must match in arithmetic expressions in order to get any
results. If you want to ignore some labels (or only match on some
labels), you need to use ignoring() or on(), as in:

my_metric / on(...) existing_gauge_metric

Both of these are discussed in the "One-to-one vector matches" section
of the PromQL "Operators" documentation,

https://prometheus.io/docs/prometheus/latest/querying/operators/#one-to-one-vector-matches

- cks