Query regarding remote write

[Sushil pratap singh]

Hi ,

I wanted to know , is there a way to identify the remote write endpoint being down . So that it can be known through some kind of alerting.

Thanks

Sushil

[Alex Duzsardi]

I think you could create a gauge metric which sends 1 as the metric value and then test if the value is still 1, otherwise the service is down.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8c91038d-10b2-47de-85df-3670d0792523n%40googlegroups.com.

--

Alexandru Duzsardi,

DevOps Engineer

Skype: alexinno83

GPG/PGP Key: https://keybase.io/aduzsardi/pgp_keys.asc

GitLab: https://gitlab.com/aduzsardi

GitHub: https://github.com/aduzsardi

LinkedIn: https://www.linkedin.com/in/aduzsardi

E-mail: alex.d...@infinitpartners.com

InFinIT Partners,

Address: Str. Macinului Nr. 17, Cluj-Napoca, Romania 

Web:  www.infinitpartners.com

[Sushil pratap singh]

Hi Alex , 

isn't  there anything by default in prometheus itself to know whether the remote write link has gone down. 

if we keep it as a gauge metric we have to push the metric again and again by pinging from the node prometheus server is running and it will be an overhead

Thanks

Sushil

[Alex Duzsardi]

Don't know about push gateway and remote write metrics , if you had an actual target configured through prometheus configuration file reachable on the network , then prometheus keeps an up metric for those per job & instance label pairs, see here https://prometheus.io/docs/concepts/jobs_instances/

[Brian Candler]

Have a look at the prometheus_remote_storage_* metrics, for example

prometheus_remote_storage_queue_highest_sent_timestamp_seconds

prometheus_remote_storage_pending_samples

prometheus_remote_storage_sent_bytes_total

I have only looked at these briefly, but I think

time() - prometheus_remote_storage_queue_highest_sent_timestamp_seconds

may give you an indication of the backlog.