Metrics with timestamps to Pushgateway

[Bas Harenslak]

Hi,

I don't understand the Pushgateway behaviour. I have an ephemeral job pushing metrics to the Pushgateway every 3 seconds. The Prometheus server scrapes every 15 seconds. My sample metric in Prometheus server will have the value of the last pushed sample in the Pushgateway (a described in the documentation). However, I want to see all pushed metrics. The documentation says "Essentially, if you push more often than every 5min, you could attach the time of pushing as a timestamp." Therefore, as the documentation suggests, I add a timestamp:

mymetric [value] [timestamp]

However, the metric still contains only the last pushed value. I could set my job to push metrics every 15 seconds but that way I would lose a lot of detailed metrics. Please explain how I can get this to work.

I'm testing with this script:

i=0
while sleep 3
do
 VALUE=$i
 ((i+=1))
 TS=`date +%s`000
 echo "mymetric $VALUE $TS"
 cat <<EOF | curl --data-binary @- http://192.168.99.100:9091/metrics/job/mytestjob/mylabel/myvalue
# TYPE mymetric counter
# HELP mymetric Just an example
mymetric $VALUE $TS
EOF
done

Every 3 seconds, it pushes a counter value to my Pushgateway and increments by 1. The $TS value is in milliseconds (added 000 to 'date +%s' since OSX doesn't support '%N' in date). The Prometheus server is set up to scrape every 15 seconds, and only reads the last pushed value at that time:

Why is Prometheus only reading the last pushed value from the Pushgateway?

[Ben Kochie]

That is the normal behavior of Prometheus.  The pushgatway is not a queue or buffer of some kind.  It only holds the last value sent to it so that Prometheus can collect that value at random.  If you want to push values every 3 seconds, you need to configure Prometheus with a scrape interval of 3 seconds or faster.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.