duplicate metrics with prometheus+grafana and docker-compose

[Anne Schumann]

Hi there,

I am new with Prometheus and I am facing issues with duplicate entries visualised with grafana. Since I am new, I don't know whether this is  a bug (have seen several issues on GitHub) or an incorrect config. Please provide me with a couple of hints.

### Host operating system: ubuntu

### Are you running node_exporter in Docker?

Yes, despite the warning, I am using prometheus with docker-compose. node-exporter is configured as a scraping job. I want to monitor a bunch of docker containers that run on my host.

### What did you do that produced an error?

Queried CPU usage for my containers after restarting my containers on the machine.

### What did you expect to see?

One value per container.

### What did you see instead?

Two metrics per container. 

Can anybody help?

Cheers,

Anne

[Brian Candler]

Please show the metrics, complete with their labels.

It might just be that the metrics you're looking at are per CPU core, and you have a dual-core CPU.  This will be clear if the metrics and labels are shown.

[Matthias Rampke]

Node exporter does not provide per container metrics, it looks at the whole node. This is sometimes difficult from inside a container, thus the warning.

For metrics about containers, check out cAdvisor: https://github.com/google/cadvisor

In Prometheus it is common to have metrics broken out by labels. For example, the CPU metrics in node exporter are all per CPU core, and we expect you to aggregate them into whichever form you need.

Can you share your query and what the output is, especially what labels it has?

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bdcf0544-d4b5-405c-befb-fcfb5599cd00n%40googlegroups.com.

[Anne Schumann]

Hi, thanks for the quick replies. It might very well be that the issue is with multiple CPU cores. Now, I am really very new to Prometheus – I just took this from a colleague who built it, hence my unspecific question –, and I am irritated by the output. For instance, my query for CPU usage is container_exporter_cpu_usage_percent{container=~"<containername"}. Or for container_exporter_memory_usage{container=~"<containername>"}. The metrics are always duplicated, mostly with the same values (but not always, so it might really be the CPUs?), although I have checked that for each service only one container is running on my host.

I am now thinking about removing these panels entirely from Grafana since I have others services (glance, for instance) which seem to provide plausible metrics.

Best,

Anne

[Brian Candler]

Using the PromQL browser in prometheus (or promtool on the command line), please can you run the query

container_exporter_cpu_usage_percent{container="somecontainer"}
and show all the results returned.  That is, so we can see the problem, together with the *actual* labels of the result.

If you do that, you can show whether the metrics really are duplicated, or in fact they are separate metrics with distinct label sets.  And in the latter case we can propose some solutions, i.e. how to aggregate the metrics in a useful way.

[Anne Schumann]

Hi , sorry for my late reply. So what I did: I stopped and then restarted a bunch of containers on my system using docker-compose. Prometheus lives in another container that I start with docker-compose as well. What I get querying,for my containers is two values per container although I have verified that on my system, only one container per containername is running. So each output is labeled with a container name and for each container on my system I have two identically named metrics.

Querying for "somecontainer" obviously gives me no data, but I maybe I misunderstood the question?

Best,

Anne

You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/6ojUQZSeJ9g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cefd7a2c-bf06-49aa-a821-e59fe1f1b82bn%40googlegroups.com.

[Brian Candler]

And if you won't show the metrics (again), then I can't help you.  a.k.a. "show me"