Exporting custom metrics to remote storage from python code.

[Yotam Loewenbach]

Hey community, does anyone got experience with exporting custom metrics from python code straight to remote storage endpoint (m3db or other), without Prometheus scraping it first and then export it via remote write? Is there a way to achieve it with prometheus client library?

[Stuart Clark]

If you are wanting to send things straight to one of the remote write
destinations you'd need to use whatever client library they provide,
making sure you format the metrics in whichever format they require
(which is likely custom to each destination). As that's nothing to do
with Prometheus, the client library wouldn't really help.

If you are talking about ingesting metrics into a Prometheus server
without scraping, then that's not possible (bar the experimental
backfill work using OpenMetrics) as it is not the design that Prometheus
uses.

However if you are looking to be using the Python Prometheus client
library already what is the issue with scraping your application in the
normal way?

--
Stuart Clark

[Yotam Loewenbach]

Hey, Stuart thanks for your answer.

I will look into my remote storage client libraries.

I guess I'm just trying to reduce the complexity because all I really need is to send my custom metrics to remote storage, and I don't want to use Prometheus in the middle.

The metrics pipeline I'm trying to build is:

MyApp ->  remote storage, instead of:

MyApp -> prometheus -> remote storage

[Stuart Clark]

On 11/02/2021 14:06, Yotam Loewenbach wrote:
> Hey, Stuart thanks for your answer.
> I will look into my remote storage client libraries.
> I guess I'm just trying to reduce the complexity because all I really
> need is to send my custom metrics to remote storage, and I don't want
> to use Prometheus in the middle.
> The metrics pipeline I'm trying to build is:
> MyApp ->  remote storage, instead of:
> MyApp -> prometheus -> remote storage
>

Fair enough. So you probably want to be looking at the docs for your
remote storage system/asking them questions as it would be totally
bypassing Prometheus.

While such a solution might be simpler in one way (metrics don't go via
a Prometheus server) it is more complex in another - if the remote
storage system isn't specifically designed for metrics (i.e. is a more
generic database system) you would likely find that their client
libraries are just basic "write this value" or "run this query" type
actions. You could look at the graphite bridge code and create something
similar with your chosen storage client library.

--
Stuart Clark

[Yotam Loewenbach]

Thanks a lot for the advice, Stuart.

After some research, I found that I can use Opentelemtry to build the metrics pipeline I need.

They recently developed a remote write exporter for their python SDK.

I'm attaching a link to the project in case you or anyone else is interested:

https://github.com/open-telemetry/opentelemetry-python-contrib/tree/release/0.17b0/exporter/opentelemetry-exporter-prometheus-remote-write

Thanks for your help.

Yotam loewenbach