Metrics from PUSH Consumer - Relabeled Metrics? Check "Up" state?

[Alexander Wilke]

Hello,

I have some Enterprise Firewalls which unfortunately only can PUSH metrics to remote_write API of Prometheus. Vendor has No Plans to offer PULL.

Which possibilities do I have in Prometheus to Change some metrics at the time they arrive on the API? I want to add some Custom Labels based in existing Label information.

In Addition ist there any possibility to Check If a devices is still sending metrics?

Because Prometheus is Not pulling it can Not Check the "Up" state i think but is it possible to query If there are metrics in Prometheus with a specific Label Set and Not older than e.g. 10minutes?

And how can I Check If the Prometheus Server can ingest all the metrics which arrive on the API? For Pull i have some dashboards which Show sampels per Seconds and so on.

I am a little Bit confused because from my understanding remote write is in General used to write Data from one Prometheus to another Destination so the configuation and Tuning Parameters seem Not to fit for incoming Traffic.

Maybe you can Help me.

[Alexander Wilke]

Hello,

I am still looking for a solution to identify if a device which uses "PUSH" method is not sending data anmore for e.g. 10 minutes.

The amount of devices is limited at around 20-30.

any idea how to check if there are not any new metrics from it? All metrics from these devices sharing the same labels like "host_name" so this is a label I could verify and they all share same metrics like "push_device_uptime(host_name: deviceName)"

[Brian Candler]

> I am still looking for a solution to identify if a device which uses "PUSH" method is not sending data anmore for e.g. 10 minutes.

Push an additional metric which is "last push time", and check when that value is more than 10 minutes earlier than the current time.

If you already have a metric like "push_device_uptime", which I presume is monotonically increasing, then you can check for when this stops increasing:

expr: push_device_uptime <= push_device_uptime offset 10m

[Alexander Wilke]

Will I run into issues with "staleness" if there aren't any metrics anymore for (more) than 5 minutes?

Or perhaps can I use this "staleness" indicator in some way?

[Chris Siebenmann]

> Will I run into issues with "staleness" if there aren't any metrics anymore
> for (more) than 5 minutes?
> Or perhaps can I use this "staleness" indicator in some way?

Perhaps this is a use for absent() or absent_over_time(), if you know
specific metrics that should always be present from the push sources
(and you know the push sources).

It might be possible to craft something clever with 'offset' and
'unless' to filter out metrics that are still present, eg:

pushed_metric offset 10m unless pushed_metric

I think this will give you every pushed_metric series that was present
ten minutes ago and isn't now (because it's stale, since it hasn't been
pushed recently enough). This is less clear than an explicit absent(),
but means you don't have to statically know the job/instance/etc labels
for all push sources that should be there.

- cks

[Alexander Wilke]

Hello,

absent_over_time() Looks good. It should allow me to alert If metrics is Not available for 10min. This will not alert If system reboots.

If i use a query with Offset I think it would alert only AeS Long as the Offset still hast a value, eg Offset 10m i will get an Alarm For 10m but after that the "now" is the Same as the Offset.

Will give it a try.

Thanks!!