Docker Metrics cannot be scraped.

[Isabel Noronha]

Hi,

I am unable to scrape docker metrics using Prometheus.

I tried metric addr: "127.0.0.1:9323" in daemon.json file

Note Prometheus is running in a container.

Any help would be appreciated.

Thank you,

Isabel

[Julius Volz]

According to https://docs.docker.com/config/daemon/prometheus/ you also need the '"experimental": true' setting in your daemon.json, do you have that as well?

If yes, could you provide more information on the problem? Can you reach http://127.0.0.1:9323/metrics manually from that host? Note that if you specify 127.0.0.1 as the IP to listen on, you will not be able to scrape the endpoint from another host. If you want to make the metrics endpoint reachable from other hosts, you'll likely want just ":9323" or "<machine-ip>:9323".

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/227a4767-aed5-497e-9cff-f80f9d9cb884%40googlegroups.com.

[Julius Volz]

Oh, and since you mentioned that Prometheus is running in a container: each container will have its own network namespace (at least by default), so that it will have its own idea of "localhost" / "127.0.0.1", so it will not be able to connect to the host machine's 9323 under that address. You'll need to provide the host machine's host name or IP and ensure that is reachable.

[Isabel Noronha]

Yes experimental is set to true.

On Wednesday, April 15, 2020 at 6:41:24 PM UTC+5:30, Julius Volz wrote:

According to https://docs.docker.com/config/daemon/prometheus/ you also need the '"experimental": true' setting in your daemon.json, do you have that as well?

If yes, could you provide more information on the problem? Can you reach http://127.0.0.1:9323/metrics manually from that host? Note that if you specify 127.0.0.1 as the IP to listen on, you will not be able to scrape the endpoint from another host. If you want to make the metrics endpoint reachable from other hosts, you'll likely want just ":9323" or "<machine-ip>:9323".

On Wed, Apr 15, 2020 at 12:41 PM Isabel Noronha <isabeln...@gmail.com> wrote:

Hi,

I am unable to scrape docker metrics using Prometheus.

I tried metric addr: "127.0.0.1:9323" in daemon.json file

Note Prometheus is running in a container.

Any help would be appreciated.

Thank you,

Isabel

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Isabel Noronha]

Yeah I have tried most of these combinations.

So let me explain it more in detail.

Running docker on a server with linux fedora 30.

docker daemon.json file

{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
}

Both are set.

So in this file I tried couple of different values for metrics-addr like the server IP x.x.x.x:9323 or 127.0.01:9323 or 0.0.0.0:9323.

In prometheus.yml

- job_name: 'docker'
         # metrics_path defaults to '/metrics'
         # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9323']

In targets:[x.x.x.x:9323] 

Note if x.x.x.x:9323 id present in daemon.json then same target IP is in prometheus.yml.

This is done for 127.0.0.1:9323 and 0.0.0.0:9323.

And I try to reach the x.x.x.x:9323 from another VM.
I have configured 3 targets on prometheus and are scraping metrics.

Only docker is down.

Thank you 

On Wednesday, April 15, 2020 at 6:44:09 PM UTC+5:30, Julius Volz wrote:

Oh, and since you mentioned that Prometheus is running in a container: each container will have its own network namespace (at least by default), so that it will have its own idea of "localhost" / "127.0.0.1", so it will not be able to connect to the host machine's 9323 under that address. You'll need to provide the host machine's host name or IP and ensure that is reachable.

On Wed, Apr 15, 2020 at 3:11 PM Julius Volz <juliu...@gmail.com> wrote:

According to https://docs.docker.com/config/daemon/prometheus/ you also need the '"experimental": true' setting in your daemon.json, do you have that as well?

If yes, could you provide more information on the problem? Can you reach http://127.0.0.1:9323/metrics manually from that host? Note that if you specify 127.0.0.1 as the IP to listen on, you will not be able to scrape the endpoint from another host. If you want to make the metrics endpoint reachable from other hosts, you'll likely want just ":9323" or "<machine-ip>:9323".

On Wed, Apr 15, 2020 at 12:41 PM Isabel Noronha <isabeln...@gmail.com> wrote:

Hi,

I am unable to scrape docker metrics using Prometheus.

I tried metric addr: "127.0.0.1:9323" in daemon.json file

Note Prometheus is running in a container.

Any help would be appreciated.

Thank you,

Isabel

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Stuart Clark]

On 16/04/2020 05:56, Isabel Noronha wrote:

Yeah I have tried most of these combinations.

So let me explain it more in detail.

Running docker on a server with linux fedora 30.

docker daemon.json file

{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
}

Both are set.

So in this file I tried couple of different values for metrics-addr like the server IP x.x.x.x:9323 or 127.0.01:9323 or 0.0.0.0:9323.

In prometheus.yml

- job_name: 'docker'
         # metrics_path defaults to '/metrics'
         # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9323']

In targets:[x.x.x.x:9323] 

Note if x.x.x.x:9323 id present in daemon.json then same target IP is in prometheus.yml.

This is done for 127.0.0.1:9323 and 0.0.0.0:9323.

And I try to reach the x.x.x.x:9323 from another VM.
I have configured 3 targets on prometheus and are scraping metrics.

Only docker is down.

As Julius mentioned as you are using Prometheus in a container you can't use "localhost" to reference the underlying host - localhost is the container itself. You need to use a different IP address to reference the host, such as the docker0 or ethernet IP (assuming Docker is configured to listen on those IPs, for example using 0.0.0.0:9323.

Thank you 

On Wednesday, April 15, 2020 at 6:44:09 PM UTC+5:30, Julius Volz wrote:

Oh, and since you mentioned that Prometheus is running in a container: each container will have its own network namespace (at least by default), so that it will have its own idea of "localhost" / "127.0.0.1", so it will not be able to connect to the host machine's 9323 under that address. You'll need to provide the host machine's host name or IP and ensure that is reachable.

On Wed, Apr 15, 2020 at 3:11 PM Julius Volz <juliu...@gmail.com> wrote:

According to https://docs.docker.com/config/daemon/prometheus/ you also need the '"experimental": true' setting in your daemon.json, do you have that as well?

If yes, could you provide more information on the problem? Can you reach http://127.0.0.1:9323/metrics manually from that host? Note that if you specify 127.0.0.1 as the IP to listen on, you will not be able to scrape the endpoint from another host. If you want to make the metrics endpoint reachable from other hosts, you'll likely want just ":9323" or "<machine-ip>:9323".

On Wed, Apr 15, 2020 at 12:41 PM Isabel Noronha <isabeln...@gmail.com> wrote:

Hi,

I am unable to scrape docker metrics using Prometheus.

I tried metric addr: "127.0.0.1:9323" in daemon.json file

Note Prometheus is running in a container.

Any help would be appreciated.

Thank you,

Isabel

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/227a4767-aed5-497e-9cff-f80f9d9cb884%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f6ad22c3-772a-4bec-b469-cda5869bdb4c%40googlegroups.com.

-- 
Stuart Clark