Prometheus Datadog integration

[adi garg]

Hello experts,

I need to pass the metrics collected by Prometheus to datadog, I know that we need to make changes in datadog's conf file present in openmetrics.d directory. But as Prometheus only shows metrics about it's own server on /metrics. How can we send them to datadog?

Thnks,

Aditya Garg

[Ben Kochie]

Try the federation endpoint.

https://prometheus.io/docs/prometheus/latest/federation/

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5f88afae-2716-4464-806a-69d8c67f1471%40googlegroups.com.

[adi garg]

Isn't there any way to do this just by using datadog conf file?

On Friday, February 28, 2020 at 4:59:59 PM UTC+5:30, Ben Kochie wrote:

Try the federation endpoint.

https://prometheus.io/docs/prometheus/latest/federation/

On Fri, Feb 28, 2020 at 11:56 AM adi garg <addy1...@gmail.com> wrote:

Hello experts,

I need to pass the metrics collected by Prometheus to datadog, I know that we need to make changes in datadog's conf file present in openmetrics.d directory. But as Prometheus only shows metrics about it's own server on /metrics. How can we send them to datadog?

Thnks,

Aditya Garg

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[adi garg]

Hey Ben, I tried reading about federation point, but I am not able to connect how it can be helpful for my problem. can you please elaborate?

[Andrey Kezikov]

When you requesting for /metrics endpoint - you reaching inner exporter on the node, what gives you information about prometheus itself as it can query itself for own status to save into history.

Ben proposing you to use federation endpoint to retrieve all collected metrics from the Prometheus, you can try it with GET request to Prometheus, replacing in your URL part '/metrics' with federation endpoint and query: '/federate?match[]={__name__=~"up"}'

It will return you all metrics for your query. You can use more broad regexp, but it will put huge load on prometheus itself and possible "out-of-time" responses, so would be better to request reduce request scopes to exactly what you need to see in datadog

[adi garg]

  1 # my global config$

  2 global:$

  3   scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.$

  4   evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.$

  5   # scrape_timeout is set to the global default (10s).$

  6 $

  7 # Alertmanager configuration$

  8 alerting:$

  9   alertmanagers:$

 10       - static_configs:$

 11          - targets:$

 12             - localhost:9093$

 13 $

 14 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.$

 15 rule_files:$

 16   # - "first_rules.yml"$

 17     - 'alerts.yml'$

 18   # - "second_rules.yml"$

 19 $

 20 # A scrape configuration containing exactly one endpoint to scrape:$

 21 # Here it's Prometheus itself.$

 22 scrape_configs:$

 23   # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.$

 24     - job_name: 'federate'$

 25       metrics_path: '/federate'$

 26       params:~$

 27         'match[]':$

 28             - '{job="prom"}'$

 29       static_configs:$

 30        - targets: ['localhost:9090']$

 31     - job_name: 'prom'$

 32       metrics_path: '/'$

 33       static_configs:$

 34         - targets: ['localhost:12156']$

My expectation:- I will get all the metrics with the job name 'prom' on my localhost:9090/federate, but I didn't get anything there. I know I am doing something wrong. Can someone please guide me?

Moreover, I do have some doubts:

1) Can we change the route name other than 'federate' or is it fixed?

2) The federation endpoint collects all metrics just for the recent timestamp and from the local system of the Prometheus server. right?

[Brian Candler]

Maybe you've misunderstood scraping and federation.

Prometheus scrapes are "pulls".  When a prometheus scrape job runs, it makes an outbound HTTP connection to a remote server (an "exporter") and reads back metrics from it.  So in your example here:

 31     - job_name: 'prom'$

 32       metrics_path: '/'$

 33       static_configs:$

 34         - targets: ['localhost:12156']$

prometheus periodically connects to localhost port 12156, and whatever is listening on that port is a prometheus "exporter" which responds with metrics.  You can also talk to it like this:

curl localhost:12156

and that server will respond with lines of the form

metricname{label="value",...} value

These are what prometheus ingests into its timeseries database.  No data is sent *from* prometheus to the exporter.

Federation is a special case of scraping: it's when one prometheus server scrapes the /federate endpoint of another prometheus server, to read out metrics which that server has already collected.   (For example: a central aggregation server is scraping the /federate endpoint of remote servers in other data centres).

Therefore, you would only configure your own prometheus server to scrape a /federate endpoint when you need to scrape metrics out of another, remote prometheus server. 

So this makes no sense:

 24     - job_name: 'federate'$

 25       metrics_path: '/federate'$

 26       params:~$

 27         'match[]':$

 28             - '{job="prom"}'$

 29       static_configs:$

 30        - targets: ['localhost:9090']$

Here you're trying to scrape your own /federate endpoint, which means collecting data from yourself and writing it back to yourself.

The original question was: how do you get data from prometheus to datadog?  I don't know: it depends on what interface(s) datadog provides for collecting/ingesting data, and datadog is a commercial cloud-hosted service.  (Perhaps their technical support can help?)

However you said originally: "I know that we need to make changes in datadog's conf file present in openmetrics.d directory. But as Prometheus only shows metrics about it's own server on /metrics. How can we send them to datadog?"

If datadog's agent is able to scrape an exporter, then point it at your /federate?match=... endpoint  rather than your /metrics endpoint.

[adi garg]

Hey Brian, Actually to my knowledge, Datadog can also pull metrics from the endpoints. Thus, I want to give it an endpoint which has all the filtered metrics I want to send to datadog. That's why I use /federate endpoint taking data from my own Prometheus server. Is there any easy way to send metrics to datadog without using a federate endpoint?

[Brian Candler]

You can't get prometheus to connect to a remote system to "push" metrics [except using the remote_write functionality, which is a specific protocol the remote system needs to support]

However, other servers can scrape or "pull" metrics from prometheus, by scraping the /federate endpoint.  That is: you can configure datadog's agent to scrape prometheus' /federate endpoint.

The first thing to do is, make sure you scrape the /federate endpoint manually using curl. Try something like this:

curl '127.0.0.1:9090/federate?match%5b%5d=%7bjob%3d"node"%7d'

Yes it's ugly, because of URL escaping.  That example uses query {job="node"}, escaped as follows:

= -> %3d

[ -> %5b

] -> %5d

{ -> %7b

} -> %7d

If course, if you don't have any metrics labeled with job="node" then you'll have to amend this as appropriate.  To scrape absolutely everything (which probably isn't a good idea), use:

curl 'prometheus:9090/federate?match%5b%5d=%7b__name__%3d%7e".*"%7d'

When that's working, turn your attention to datadog to get it to do the same scraping.  According to this post it should be possible.  I haven't tested it (since I don't use datadog), but you can try:

instances:
  # The prometheus endpoint to query from
  - prometheus_url: http://targethost:9090/federate?match%5b%5d=%7bjob%3d"node"%7d
    namespace: "my_prometheus"
    metrics:
      - "*"

This may need adjusting depending on whether datadog does its own escaping of URLs.  If you can't make it work, then since you're using a commercial service, I suggest you talk to their technical support - that's what you're paying for after all.

What you showed before was configuring prometheus to scrape its own /federate endpoint, and that doesn't make sense.

[adi garg]

Thanks, Brian, for the detailed answer. I am referring to the official doc by datadog(https://www.datadoghq.com/blog/monitor-prometheus-metrics/), they have only mentioned about getting the metrics from /metrics endpoint in their example, and there is nothing else(like federation points, etc). Is the doc incomplete or am I missing something because they have given nothing about what to do when we need to have the target metrics also?

One more doubt -> /metrics endpoint only shows metrics of Prometheus server or the metrics of the whole system on which Prometheus is running.

[Brian Candler]

Note: with curl, the -g flag disables curl's own "globbing" of special characters, so this simplifies to:

curl -g '127.0.0.1:9090/federate?match[]={job="node"}'

[Brian Candler]

On Tuesday, 3 March 2020 08:52:15 UTC, adi garg wrote:

One more doubt -> /metrics endpoint only shows metrics of Prometheus server or the metrics of the whole system on which Prometheus is running.

The /metrics endpoint, on the prometheus server itself, shows only internal metrics about the prometheus application.

If you want to collect information about the system on which Prometheus is running, then you need to install node_exporter on that system, and configure prometheus to scrape it.