Prometheus remote write drop.

[Sohaib Omar]

Hey all, 

I am using Prometheus remote write to write metrics into remote storage(InfluxDB). I am trying to benchmark the Prometheus remote_writer, for that purpose is there any way to achieve below:

1. Calculate the ingestion rate/second into the remote storage.
2.  the drop rate of series at the remote ingestion time. for example, expected remote writes vs actual remote writes.

Thanks

[Brian Candler]

Try:

curl localhost:9090/metrics | grep prometheus_remote_storage

You will find a bunch of metrics there:

prometheus_remote_storage_dropped_samples_total

prometheus_remote_storage_enqueue_retries_total

prometheus_remote_storage_failed_samples_total

prometheus_remote_storage_highest_timestamp_in_seconds

prometheus_remote_storage_pending_samples

prometheus_remote_storage_queue_highest_sent_timestamp_seconds

prometheus_remote_storage_retried_samples_total

prometheus_remote_storage_samples_in_total

prometheus_remote_storage_sent_batch_duration_seconds_bucket

prometheus_remote_storage_sent_batch_duration_seconds_count

prometheus_remote_storage_sent_batch_duration_seconds_sum

prometheus_remote_storage_sent_bytes_total

prometheus_remote_storage_shard_capacity

prometheus_remote_storage_shards

prometheus_remote_storage_shards_desired

prometheus_remote_storage_shards_max

prometheus_remote_storage_shards_min

prometheus_remote_storage_string_interner_zero_reference_releases_total

prometheus_remote_storage_succeeded_samples_total

[Sohaib Omar]

if  rate(prometheus_remote_storage_samples_in_total[10m])  and rate(prometheus_remote_storage_succeeded_samples_total[10m]) does not match, does this means that some of the series fail to be writtent into remote storage.

For example in my case the above queries result is.
 rate(prometheus_remote_storage_samples_in_total[10m]) = 1965.2

rate(prometheus_remote_storage_succeeded_samples_total[10m]) = 267.2

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec8a996f-2f98-42c0-bb4a-9c067828a4d9%40googlegroups.com.