Exporting Metrics from Opentelemetry to Prometheus via remote write

[John Perrin]

Hello

We are exporting metrics from an OpenTelemetry collector using the prometheusremotewrite exporter to Prometheus started with the  --enable-feature=remote-write-receiver

The metrics are the results of k6 http requests eg response time which are passed into the Opentelemetry collector and then exported to Prometheus and we want to ingest all the data points into Prometheus. We then use Grafana to aggregate and visualise the metrics.

It generally works fine except when a metric has the same name and value within the same time interval inside Prometheus. In this case Prometheus rejects the metric as a duplicate when Opentelemetry tries to export it and the metric is dropped.  

In this use case my question is why doesn't Prometheus use the metric timestamp which has nanosecond precision. If it did then it would be unlikely that 2 metrics would be duplicates. 

The other option we have had some success with is to add a random number tag to the metric but this doesn't always work and is not ideal.

I'm not clear on how Prometheus creates it's internal timestamps in this case and wondered if there is some way to configure it to use the metric timestamp.

[Brian Candler]

It sounds like you're having problems with a third-party software which you refer to as "an OpenTelemetry collector"

As a starting point, can you provide a link to that project and/or its documentation, and show how you have configured it?

Can you also show the exact error message you see? And what version of Prometheus you are running?

> my question is why doesn't Prometheus use the metric timestamp which has nanosecond precision

Prometheus *does* preserve these timestamps with remote write. So at this point, it sounds like a problem with the third-party software.

(Aside: Prometheus generally ignores the timestamps when doing standard scraping over HTTP, but the timestamp is then the time of the scrape, so they are always different)

[Brian Candler]

FWIW, Prometheus v2.47.0 (just released) includes an experimental OpenTelemetry (OTLP) Ingestion endpoint.

https://github.com/prometheus/prometheus/pull/12643/files