/metrics endpoint not showing metrics scraped from applications

[Rounak Salim]

The /metrics endpoint only shows me the metrics for the Prometheus server and none of the metrics scraped by Prometheus from other applications are shown.

How can I get all my metrics to be shown in the /metrics endpoint?

This is causing problems getting our metrics consumed by DataDog agents since they are only able to pick up the Prometheus server metrics from the /metrics endpoint and I couldn't get it to work with any other endpoint on their config file. 

Followed this documentation to get it setup with the Prometheus integration: https://www.datadoghq.com/blog/monitor-prometheus-metrics/

[Christian Hoffmann]

Hi,

On 8/27/20 3:16 AM, 'Rounak Salim' via Prometheus Users wrote:
> The /metrics endpoint only shows me the metrics for the Prometheus
> server and none of the metrics scraped by Prometheus from other
> applications are shown.
>
> How can I get all my metrics to be shown in the /metrics endpoint?

I don't think you can. /metrics is supposed to be about the application
(Prometheus) itself, not about data from other servers.

The /federate endpoint would probably do what you want:
https://prometheus.io/docs/prometheus/latest/federation/

However, keep in mind that it's not intended to export arbitrary large
chunks of data.

> This is causing problems getting our metrics consumed by DataDog agents
> since they are only able to pick up the Prometheus server metrics from
> the /metrics endpoint and I couldn't get it to work with any other
> endpoint on their config file. 
> Followed this documentation to get it setup with the Prometheus
> integration: https://www.datadoghq.com/blog/monitor-prometheus-metrics/

I don't know datadog but it seems like it is intended to scrape the
targets itself without any Prometheus server inbetween?

Kind regards,
Christian

[Rounak Salim]

It does indeed seem like the DataDog integration is only meant to retrieve information about the Prometheus server itself and not other metrics that are being scraped by Prometheus.

I'm unfamiliar with federation but it seems like it's mostly used for pulling data from multiple Prometheus instances into a single instance. Do all the scraped metrics show up on the /federate URL if federation is setup?

Thanks for your help!

[Christian Hoffmann]

Hi,

On 8/28/20 10:08 PM, 'Rounak Salim' via Prometheus Users wrote:
> I'm unfamiliar with federation but it seems like it's mostly used for
> pulling data from multiple Prometheus instances into a single instance.
> Do all the scraped metrics show up on the /federate URL if federation is
> setup?

The federate endpoint can access all metrics from the Prometheus
instance. It's supposed to be used to grab all aggregated metrics, for
example (see the docs and the job:.* matcher). In theory, it can also
export all metrics.

However, as soon as the data volume rises you may run into resource
problems on Prometheus, the network or DataDog. The API simply isn't
intended to be used like this.

Maybe DataDog can use be used as a remote_write target instead?

Kind regards,
Christian