Getting host name on cAdvisor container_last_seen results

3,037 views
Skip to first unread message

dave08

unread,
Nov 22, 2017, 11:28:26 AM11/22/17
to Prometheus Developers
Anyone know how to get a list of containers running on each node (with host name..) w/ promql and grafana? I tried: `container_last_seen{id=~"/docker.*"} * ON(instance) GROUP_LEFT(host) host`, but it's empty...

Tom Wilkie

unread,
Nov 22, 2017, 3:21:46 PM11/22/17
to dave08, Prometheus Developers
Does Prometheus know where the containers are running?  On Kubernetes, this information can be scraped out of kube-state-metrics.  Here is an example:

  container_last_seen 
* on(namespace, pod_name) group_left(node) 
  label_replace(kube_pod_info, "pod_name", "$1", "pod", "(.*)")

Unfortunately cAdvisor uses the pod_name label name and kube-state-metrics uses pod, so a label_replace is needed.

Cheers

Tom


On Wed, Nov 22, 2017 at 4:28 PM dave08 <dav...@gmail.com> wrote:
Anyone know how to get a list of containers running on each node (with host name..) w/ promql and grafana? I tried: `container_last_seen{id=~"/docker.*"} * ON(instance) GROUP_LEFT(host) host`, but it's empty...

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/02355a44-bb00-4530-a724-f5a191ab9286%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dave08

unread,
Nov 23, 2017, 8:23:02 AM11/23/17
to Prometheus Developers
Yes, the host metric contains two labels, instance and host, I need to match by instance and get host value into the container_last_seen metric, but I'm getting no results... what could it be?

Thanks for answering!


On Wednesday, November 22, 2017 at 10:21:46 PM UTC+2, Tom Wilkie wrote:
Does Prometheus know where the containers are running?  On Kubernetes, this information can be scraped out of kube-state-metrics.  Here is an example:

  container_last_seen 
* on(namespace, pod_name) group_left(node) 
  label_replace(kube_pod_info, "pod_name", "$1", "pod", "(.*)")

Unfortunately cAdvisor uses the pod_name label name and kube-state-metrics uses pod, so a label_replace is needed.

Cheers

Tom


On Wed, Nov 22, 2017 at 4:28 PM dave08 <dav...@gmail.com> wrote:
Anyone know how to get a list of containers running on each node (with host name..) w/ promql and grafana? I tried: `container_last_seen{id=~"/docker.*"} * ON(instance) GROUP_LEFT(host) host`, but it's empty...

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

Tom Wilkie

unread,
Nov 23, 2017, 8:34:03 AM11/23/17
to dave08, Prometheus Developers
Can you send the output of the two queries `container_last_seen{id=~"/docker.*"}` and `host`?

Tanks

Tom

On Thu, Nov 23, 2017 at 1:23 PM dave08 <dav...@gmail.com> wrote:
Yes, the host metric contains two labels, instance and host, I need to match by instance and get host value into the container_last_seen metric, but I'm getting no results... what could it be?

Thanks for answering!


On Wednesday, November 22, 2017 at 10:21:46 PM UTC+2, Tom Wilkie wrote:
Does Prometheus know where the containers are running?  On Kubernetes, this information can be scraped out of kube-state-metrics.  Here is an example:

  container_last_seen 
* on(namespace, pod_name) group_left(node) 
  label_replace(kube_pod_info, "pod_name", "$1", "pod", "(.*)")

Unfortunately cAdvisor uses the pod_name label name and kube-state-metrics uses pod, so a label_replace is needed.

Cheers

Tom


On Wed, Nov 22, 2017 at 4:28 PM dave08 <dav...@gmail.com> wrote:
Anyone know how to get a list of containers running on each node (with host name..) w/ promql and grafana? I tried: `container_last_seen{id=~"/docker.*"} * ON(instance) GROUP_LEFT(host) host`, but it's empty...

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d995d071-688a-44c1-9a78-2c590ed610b7%40googlegroups.com.

dave08

unread,
Nov 23, 2017, 9:32:39 AM11/23/17
to Prometheus Developers
This is partial of container_last_seen{id=~"/docker/.*"}

container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="il2oqyf3gj0c5o4p5t2hx1t2w",container_label_com_docker_swarm_service_name="elk_elasticsearch",container_label_com_docker_swarm_task_id="aoisvqqfhxqwey2s2rr6lz2d9",container_label_com_docker_swarm_task_name="elk_elasticsearch.1.aoisvqqfhxqwey2s2rr6lz2d9",id="/docker/6dec39b02d17a1e3920e0d8d929979a0a3cb693f37fa1ed369f54d7510fbfa4b",image="elasticsearch:5.4.2-alpine@sha256:8a3c6a42bdc8a775f563f19dfccebb82f902533933d57c3a30a0ec8193ef92a5",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="elk_elasticsearch.1.aoisvqqfhxqwey2s2rr6lz2d9"}1511447432
container_last_seen{container_label_com_docker_stack_namespace="core",container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="9bekxfqdsbdqavnd2ktdixqoa",container_label_com_docker_swarm_service_name="core_pgadmin",container_label_com_docker_swarm_task_id="r8remmvcvicfwpqrb8tirjuui",container_label_com_docker_swarm_task_name="core_pgadmin.1.r8remmvcvicfwpqrb8tirjuui",container_label_org_label_schema_license="PostgreSQL",container_label_org_label_schema_name="pgAdmin4",container_label_org_label_schema_url="https://www.pgadmin.org",container_label_org_label_schema_vcs_url="https://github.com/fenglc/dockercloud-pgAdmin4",container_label_org_label_schema_version="2.0",id="/docker/3d8b446c48701e8de153e6ffe2104ceba5989ddad07199c262f9fbf03554b461",image="fenglc/pgadmin4:2.0@sha256:8c233ab3cf6beffc93d99a46bd423eebfd6f424ba41da08db6e23168fc7b6565",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="core_pgadmin.1.r8remmvcvicfwpqrb8tirjuui"}1511447433
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="0baqm7fvv5ffd7n9zdw584ltb",container_label_com_docker_swarm_service_name="elk_logspout",container_label_com_docker_swarm_task_id="hogk30tqppia1jwdhqxg9gie2",container_label_com_docker_swarm_task_name="elk_logspout.oviokdkepn2k4oi776t7uk7m1.hogk30tqppia1jwdhqxg9gie2",id="/docker/934a0b07d4fef4419394cab89ca699b9cb92a5f250da85c1aae81f90033162ed",image="gliderlabs/logspout:latest@sha256:9a807db8e9e484920c3228770da76b8af68e8704d8ca0965e68bf8cefc93cee1",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="elk_logspout.oviokdkepn2k4oi776t7uk7m1.hogk30tqppia1jwdhqxg9gie2"}1511447432
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="qpz7jav2chu05hv152x9fjydd",container_label_com_docker_swarm_service_name="monitor-exporters_node-exporter",container_label_com_docker_swarm_task_id="0tkrxwvdh1qwvrwd5w65zsfdg",container_label_com_docker_swarm_task_name="monitor-exporters_node-exporter.oviokdkepn2k4oi776t7uk7m1.0tkrxwvdh1qwvrwd5w65zsfdg",container_label_org_label_schema_build_date="2017-05-18T23:59:50Z",container_label_org_label_schema_name="node-exporter",container_label_org_label_schema_schema_version="1.0",container_label_org_label_schema_url="https://github.com/basi/docker-node-exporter/blob/master/README.md",container_label_org_label_schema_usage="/src/README.md",container_label_org_label_schema_vcs_branch="HEAD",container_label_org_label_schema_vcs_ref="118ed1b61870dbb8c36d0fe5e23aee23abbb62e6",container_label_org_label_schema_vcs_url="https://github.com/bvis/docker-node-exporter.git",container_label_org_label_schema_vendor="Basi",container_label_org_label_schema_version="v1.14.0",id="/docker/7c14c969ca5641275c26b27cab3d3ad5811696449769f1c87bcde77b90a521e5",image="basi/node-exporter:v1.14.0@sha256:9517cfc33b1a90d071910112606b48946755fc3d13627dc07e61061ac18c7cfc",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_node-exporter.oviokdkepn2k4oi776t7uk7m1.0tkrxwvdh1qwvrwd5w65zsfdg"}1511447432
container_last_seen{container_label_com_docker_stack_namespace="monitor",container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="zpggk5z0s1xdqm142sd20b832",container_label_com_docker_swarm_service_name="monitor_grafana",container_label_com_docker_swarm_task_id="xcvr9xr7iibks65yhpab1azw4",container_label_com_docker_swarm_task_name="monitor_grafana.1.xcvr9xr7iibks65yhpab1azw4",id="/docker/593e9d9d1103ae36012700dff7292f451368120de6ded274c1fca604f27c6968",image="grafana/grafana:4.5.2@sha256:8aeff30747a27e8ab822ef8259c8dd8c96be21d65ae27cc51e82379ab1f95839",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="monitor_grafana.1.xcvr9xr7iibks65yhpab1azw4"}1511447432
container_last_seen{container_label_com_docker_stack_namespace="core",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="qzkolyrg7v96fltnznm4qu37b",container_label_com_docker_swarm_service_name="core_db",container_label_com_docker_swarm_task_id="puscvzfg19t07mgzdf9kr8co5",container_label_com_docker_swarm_task_name="core_db.1.puscvzfg19t07mgzdf9kr8co5",id="/docker/e5951dd42c874e1e061aaf90aea8d53a05e113526c98e1282c8394c54493a068",image="postgres:9.6@sha256:87457f03e727da04b9955c0c5351fc02cac3e082f716b8edd9ac5070db11b74c",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="core_db.1.puscvzfg19t07mgzdf9kr8co5"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="ym8jmjy5nr4jxlzz6zdepp54w",container_label_com_docker_swarm_service_name="elk_logstash",container_label_com_docker_swarm_task_id="yd0jm1y9bmu0sgpgb6rv9978v",container_label_com_docker_swarm_task_name="elk_logstash.1.yd0jm1y9bmu0sgpgb6rv9978v",id="/docker/66d25ef3b13cbd7874507436146dc005c34b53b0bc97517aad3cf1078cbadc1f",image="localhost:5000/logstash:latest",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="elk_logstash.1.yd0jm1y9bmu0sgpgb6rv9978v"}1511447431
container_last_seen{id="/docker/cae4481ba9e1c80e35c03dbbd1f87142eb19c25dc0515d71d2ee3236e3594159",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="distracted_nightingale"}1511447431
container_last_seen{id="/docker/4c5fdddf9829585713fbe026db4cfa5661d47597369fac7494e6d682422f9b3f",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="cocky_mirzakhani"}1511447430
container_last_seen{container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="sn0uqcfbqx45k4r9bz1l77bca",container_label_com_docker_swarm_service_name="old_redis",container_label_com_docker_swarm_task_id="yxg5v7oi68lrf2fd1udql6b16",container_label_com_docker_swarm_task_name="old_redis.1.yxg5v7oi68lrf2fd1udql6b16",id="/docker/13fb890a62b92923b834bcc67f12924b83e497ac7d5f5019d1f352792a7cd026",image="redis:4.0.2-alpine@sha256:504cf109b4922e7cb8db839feb1a1ea6d434782e8be9822fbc0ef0a05ac64844",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="old_redis.1.yxg5v7oi68lrf2fd1udql6b16"}1511447431
container_last_seen{id="/docker/53d292395327031d439d10321564e01700cc689637be960dd3c037e9f1146a72",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="fervent_benz"}1511447431
container_last_seen{container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="eofz8s8i77ffv0omcc493fqp3",container_label_com_docker_swarm_service_name="mysql",container_label_com_docker_swarm_task_id="ruh8mayd5anpts5ftu2xns13v",container_label_com_docker_swarm_task_name="mysql.1.ruh8mayd5anpts5ftu2xns13v",id="/docker/4e72c3fb22cb6d8a3d02f7cea176fdcbcd2cc45cf6538c0c4147aa73b8478d73",image="percona:5.7@sha256:77286d22bce30ce17a16e932c4b4ac1f7f81aa7aee92c1ef3fc006a500ff9113",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="mysql.1.ruh8mayd5anpts5ftu2xns13v"}1511447433
container_last_seen{container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="s1e68evxilnc28hk1288rmo4r",container_label_com_docker_swarm_service_name="old_admin",container_label_com_docker_swarm_task_id="a4jarm4nf1xcq3mxwz5bygdt5",container_label_com_docker_swarm_task_name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5",container_label_io_webdevops_layout="8",container_label_io_webdevops_version="1.5.0",container_label_maintainer="in...@webdevops.io",container_label_vendor="WebDevOps.io",id="/docker/c148af9f9bab69628a8a0bb591aab90f83d45fbbd65a0e66ccd74956582cf816",image="localhost:5000/admin-old:34@sha256:0c96ba0850a061c6d8e4702ccb03990f70c05b1323383ebb21094cd9dbaa3778",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5"}1511447432
container_last_seen{id="/docker/2b4d71ff4497036dc2797b92190f2e5396ed16d450667a2db74f3f9cfa0e684d",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="reverent_wright"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="ci-prod",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="4z5f1nnanbxesr5pjtesq4vyz",container_label_com_docker_swarm_service_name="ci-prod_registry",container_label_com_docker_swarm_task_id="xv68m0hfn5ktwzd80hxgdfefw",container_label_com_docker_swarm_task_name="ci-prod_registry.1.xv68m0hfn5ktwzd80hxgdfefw",id="/docker/f5838d88398b265b927cd56b23f621686ea810137668cb636b9747ff036d291c",image="registry:2.6.1@sha256:5eaafa2318aa0c4c52f95077c2a68bed0b13f6d2b464835723d4de1484052299",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="ci-prod_registry.1.xv68m0hfn5ktwzd80hxgdfefw"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="proxy",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="nwqnel71yezn82goamvniy1eq",container_label_com_docker_swarm_service_name="proxy_swarm-listener",container_label_com_docker_swarm_task_id="i5e7o2up0lsk7e1dnwbodbksw",container_label_com_docker_swarm_task_name="proxy_swarm-listener.1.i5e7o2up0lsk7e1dnwbodbksw",id="/docker/efebdf1a53aaf327d2884469d55b8f02dc4ae7561dcc275e4c212df42d6dd770",image="vfarcic/docker-flow-swarm-listener:17.11.11-23@sha256:ac27bcfba99963d118734f4e74885decf57f08eb92b7fba78164bcdf0466a209",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="proxy_swarm-listener.1.i5e7o2up0lsk7e1dnwbodbksw"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="0baqm7fvv5ffd7n9zdw584ltb",container_label_com_docker_swarm_service_name="elk_logspout",container_label_com_docker_swarm_task_id="wr354npxbktpv10ph6itjuqve",container_label_com_docker_swarm_task_name="elk_logspout.47o07qastg7thfsmgtaf66l0w.wr354npxbktpv10ph6itjuqve",id="/docker/a98ca3a9fcbdb730a2069c65509ca401eb7de16bd6ef6d657514f2abcd011522",image="gliderlabs/logspout:latest@sha256:9a807db8e9e484920c3228770da76b8af68e8704d8ca0965e68bf8cefc93cee1",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="elk_logspout.47o07qastg7thfsmgtaf66l0w.wr354npxbktpv10ph6itjuqve"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="monitor",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="382zl4p8snnet5z4j3lan7hx3",container_label_com_docker_swarm_service_name="monitor_prometheus",container_label_com_docker_swarm_task_id="ewjb1440hovex3pedu9ni7o0y",container_label_com_docker_swarm_task_name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y",container_label_maintainer="The Prometheus Authors <prometheus...@googlegroups.com>",id="/docker/a20b81ac0a25185f399129e32c76735bcb65d5fb6c7fbc283832ac4d2f06214b",image="vfarcic/docker-flow-monitor:17.11.12-27@sha256:82b7f70492a65658fd36c9b5095839aed8cf0380ed65d11a104495126e7517cf",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y"}1511447430
container_last_seen{id="/docker/6d56033a189ee74549c37b075ae0865cb990fa96e30cd80e402bc5d7e457fc11",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="romantic_hermann"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="0baqm7fvv5ffd7n9zdw584ltb",container_label_com_docker_swarm_service_name="elk_logspout",container_label_com_docker_swarm_task_id="cldqo6jedacb18bh61ai8wn0k",container_label_com_docker_swarm_task_name="elk_logspout.ovhu53o07e3ofsv8lnpubxl48.cldqo6jedacb18bh61ai8wn0k",id="/docker/ef7fc0e55aad040920e0fd0a7d7f077ec4d74a8ba14dde2a9ab679332122f570",image="gliderlabs/logspout:latest@sha256:9a807db8e9e484920c3228770da76b8af68e8704d8ca0965e68bf8cefc93cee1",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="elk_logspout.ovhu53o07e3ofsv8lnpubxl48.cldqo6jedacb18bh61ai8wn0k"}1511447433
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="s6fa816uqnv50po1fcu88dhzb",container_label_com_docker_swarm_service_id="0baqm7fvv5ffd7n9zdw584ltb",container_label_com_docker_swarm_service_name="elk_logspout",container_label_com_docker_swarm_task_id="817mx6h00lk4ebjon6blfw3ne",container_label_com_docker_swarm_task_name="elk_logspout.s6fa816uqnv50po1fcu88dhzb.817mx6h00lk4ebjon6blfw3ne",id="/docker/4346e180f362d235f02e30fa63bc76204ee5d620689d9811c607eda07d9f943e",image="gliderlabs/logspout:latest@sha256:9a807db8e9e484920c3228770da76b8af68e8704d8ca0965e68bf8cefc93cee1",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="elk_logspout.s6fa816uqnv50po1fcu88dhzb.817mx6h00lk4ebjon6blfw3ne"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="whhvhklvpwihck7d2heov3vjm",container_label_com_docker_swarm_service_name="monitor-exporters_ha-proxy",container_label_com_docker_swarm_task_id="xlakoiphap7xknof9qdhks2l6",container_label_com_docker_swarm_task_name="monitor-exporters_ha-proxy.1.xlakoiphap7xknof9qdhks2l6",id="/docker/3f56d05985fc036295e112cf0b843b62e3fc2e3ca3f6fa2b7dc42ffedcc9aa59",image="quay.io/prometheus/haproxy-exporter:latest@sha256:a1dbb39efb6c01794a754fc18665c44e2deb6d2062522147c09b24fac27d0fb6",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_ha-proxy.1.xlakoiphap7xknof9qdhks2l6"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="qpz7jav2chu05hv152x9fjydd",container_label_com_docker_swarm_service_name="monitor-exporters_node-exporter",container_label_com_docker_swarm_task_id="kqwht4tig5k1sh5ifufj8trbq",container_label_com_docker_swarm_task_name="monitor-exporters_node-exporter.ovhu53o07e3ofsv8lnpubxl48.kqwht4tig5k1sh5ifufj8trbq",container_label_org_label_schema_build_date="2017-05-18T23:59:50Z",container_label_org_label_schema_name="node-exporter",container_label_org_label_schema_schema_version="1.0",container_label_org_label_schema_url="https://github.com/basi/docker-node-exporter/blob/master/README.md",container_label_org_label_schema_usage="/src/README.md",container_label_org_label_schema_vcs_branch="HEAD",container_label_org_label_schema_vcs_ref="118ed1b61870dbb8c36d0fe5e23aee23abbb62e6",container_label_org_label_schema_vcs_url="https://github.com/bvis/docker-node-exporter.git",container_label_org_label_schema_vendor="Basi",container_label_org_label_schema_version="v1.14.0",id="/docker/587438608a058c3174063a797afb2f7ed85f7c40e4c45615be973cdf4b6a5d11",image="basi/node-exporter:v1.14.0@sha256:9517cfc33b1a90d071910112606b48946755fc3d13627dc07e61061ac18c7cfc",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_node-exporter.ovhu53o07e3ofsv8lnpubxl48.kqwht4tig5k1sh5ifufj8trbq"}1511447433
container_last_seen{id="/docker/0076131eb5df06769152a79bdc4d6027794b45e06d651c58faebcb426f2ed179",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="xenodochial_leakey"}1511447431
container_last_seen{id="/docker/5608cc02122e1d7a7185c800cfba762045436a497fd4cb19908e4bd0269abd3c",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="upbeat_hoover"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="rogh448mfcd8fngswimso2j22",container_label_com_docker_swarm_service_name="elk_kibana",container_label_com_docker_swarm_task_id="mgz018hg06t0eqi4ughglygt0",container_label_com_docker_swarm_task_name="elk_kibana.1.mgz018hg06t0eqi4ughglygt0",id="/docker/c4157741295eba435c9adc2050d367085107fe88fe057531e4b3da15278ee531",image="kibana:5.4.2@sha256:f34bf12833679261c83a3f49f5fe97fb9c4a0aa0ec98d92fba503235c74fadea",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="elk_kibana.1.mgz018hg06t0eqi4ughglygt0"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="v0phsqgcypg72bdwwvhpa1fal",container_label_com_docker_swarm_service_name="monitor-exporters_cadvisor",container_label_com_docker_swarm_task_id="i6tvomo8fyln21fbidh2bvr5s",container_label_com_docker_swarm_task_name="monitor-exporters_cadvisor.47o07qastg7thfsmgtaf66l0w.i6tvomo8fyln21fbidh2bvr5s",id="/docker/b2f31e0f93003e4eab4ed87304cda53eb017e043dc35073b888093b440f7fd0c",image="google/cadvisor:latest@sha256:ed53f9f93bb52d64fa8f95daece42d95d18d1e48a8ec9237bd5944ed634348a1",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_cadvisor.47o07qastg7thfsmgtaf66l0w.i6tvomo8fyln21fbidh2bvr5s"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="v0phsqgcypg72bdwwvhpa1fal",container_label_com_docker_swarm_service_name="monitor-exporters_cadvisor",container_label_com_docker_swarm_task_id="ougz5mea0e79gl40tjj92d4eh",container_label_com_docker_swarm_task_name="monitor-exporters_cadvisor.oviokdkepn2k4oi776t7uk7m1.ougz5mea0e79gl40tjj92d4eh",id="/docker/3eac41c97110f87653aca37ca838bbb5ffdc5c86fe73486d11fcf4d8b0a4857f",image="google/cadvisor:latest@sha256:ed53f9f93bb52d64fa8f95daece42d95d18d1e48a8ec9237bd5944ed634348a1",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_cadvisor.oviokdkepn2k4oi776t7uk7m1.ougz5mea0e79gl40tjj92d4eh"}1511447433
container_last_seen{container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="63781x7lcw8sme5e9qq95e6k5",container_label_com_docker_swarm_service_name="import-percona",container_label_com_docker_swarm_task_id="kdi0zbxjk0146g6jdiio3cbtd",container_label_com_docker_swarm_task_name="import-percona.1.kdi0zbxjk0146g6jdiio3cbtd",id="/docker/1b715d378997132f9ceca260c17c2bc33613e0b904ba78bb33337660bc37f987",image="percona:5.7@sha256:77286d22bce30ce17a16e932c4b4ac1f7f81aa7aee92c1ef3fc006a500ff9113",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="import-percona.1.kdi0zbxjk0146g6jdiio3cbtd"}1511447433
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="s6fa816uqnv50po1fcu88dhzb",container_label_com_docker_swarm_service_id="v0phsqgcypg72bdwwvhpa1fal",container_label_com_docker_swarm_service_name="monitor-exporters_cadvisor",container_label_com_docker_swarm_task_id="mi51zr3tyl3btjurtipcufc1l",container_label_com_docker_swarm_task_name="monitor-exporters_cadvisor.s6fa816uqnv50po1fcu88dhzb.mi51zr3tyl3btjurtipcufc1l",id="/docker/3fd427759c9a9c3c567422acede9f816adb4329c38b2f2088364aad799a2212d",image="google/cadvisor:latest@sha256:ed53f9f93bb52d64fa8f95daece42d95d18d1e48a8ec9237bd5944ed634348a1",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_cadvisor.s6fa816uqnv50po1fcu88dhzb.mi51zr3tyl3btjurtipcufc1l"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="s6fa816uqnv50po1fcu88dhzb",container_label_com_docker_swarm_service_id="qpz7jav2chu05hv152x9fjydd",container_label_com_docker_swarm_service_name="monitor-exporters_node-exporter",container_label_com_docker_swarm_task_id="xebm887lnrmxsits3njxkukvb",container_label_com_docker_swarm_task_name="monitor-exporters_node-exporter.s6fa816uqnv50po1fcu88dhzb.xebm887lnrmxsits3njxkukvb",container_label_org_label_schema_build_date="2017-05-18T23:59:50Z",container_label_org_label_schema_name="node-exporter",container_label_org_label_schema_schema_version="1.0",container_label_org_label_schema_url="https://github.com/basi/docker-node-exporter/blob/master/README.md",container_label_org_label_schema_usage="/src/README.md",container_label_org_label_schema_vcs_branch="HEAD",container_label_org_label_schema_vcs_ref="118ed1b61870dbb8c36d0fe5e23aee23abbb62e6",container_label_org_label_schema_vcs_url="https://github.com/bvis/docker-node-exporter.git",container_label_org_label_schema_vendor="Basi",container_label_org_label_schema_version="v1.14.0",id="/docker/b05850c26e18241715d4df7d84aac92fc3090b80e066123f279b215c2edb516e",image="basi/node-exporter:v1.14.0@sha256:9517cfc33b1a90d071910112606b48946755fc3d13627dc07e61061ac18c7cfc",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_node-exporter.s6fa816uqnv50po1fcu88dhzb.xebm887lnrmxsits3njxkukvb"}1511447430
container_last_seen{container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="vyd8a5erxu4z14940ztpei10j",container_label_com_docker_swarm_service_name="mysqld-exporter",container_label_com_docker_swarm_task_id="lai5g14jarqjxk7cexjkoi3ve",container_label_com_docker_swarm_task_name="mysqld-exporter.1.lai5g14jarqjxk7cexjkoi3ve",id="/docker/f50191d813c4ee5fed5d851dda7a7f3446dbe0918d0d3548cd9ffed2fa7e0375",image="prom/mysqld-exporter:latest@sha256:a1eda24a95f09a817f2cf39a7fa3d506df88e76ebdc08c0293744ebaa546e3ab",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="mysqld-exporter.1.lai5g14jarqjxk7cexjkoi3ve"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="core",container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="5id1baamip53qdxmg4os5asqf",container_label_com_docker_swarm_service_name="core_odoo",container_label_com_docker_swarm_task_id="ulhyxvzih1e9rh9po6jbn3j3c",container_label_com_docker_swarm_task_name="core_odoo.1.ulhyxvzih1e9rh9po6jbn3j3c",id="/docker/dee5559c40a9dcd2b561c639dfd5c88aeb1ccc5bf7e4822ce80d5e014cfa7513",image="localhost:5000/odoo:40@sha256:fa15e0e39fa267a1991d025c795cc402b5e4d9c949e65410e99236b68dd3cd1d",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="core_odoo.1.ulhyxvzih1e9rh9po6jbn3j3c"}1511447433
container_last_seen{container_label_com_docker_stack_namespace="proxy",container_label_com_docker_swarm_node_id="oviokdkepn2k4oi776t7uk7m1",container_label_com_docker_swarm_service_id="z87tfvnynw9r0vfenmi89rao7",container_label_com_docker_swarm_service_name="proxy_proxy",container_label_com_docker_swarm_task_id="hmhepoakfw8d631bpah0xfn5w",container_label_com_docker_swarm_task_name="proxy_proxy.1.hmhepoakfw8d631bpah0xfn5w",id="/docker/d6ced4731ab45b991c79c78300f52059bee4b59df2d2b87622cfb5996b4fdb30",image="vfarcic/docker-flow-proxy:17.11.11-56@sha256:a6a22ee3cbfa5ce0e996ff162069f3f0ccc31b21f750a00da355bf3cc758366a",instance="10.0.2.26:8080",job="monitor-exporters_cadvisor",name="proxy_proxy.1.hmhepoakfw8d631bpah0xfn5w"}1511447433
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="v0phsqgcypg72bdwwvhpa1fal",container_label_com_docker_swarm_service_name="monitor-exporters_cadvisor",container_label_com_docker_swarm_task_id="bxopq382md0ojxcpsfoll0c9x",container_label_com_docker_swarm_task_name="monitor-exporters_cadvisor.ovhu53o07e3ofsv8lnpubxl48.bxopq382md0ojxcpsfoll0c9x",id="/docker/00042de2bca67d64c023dfce2868a103f3bf9ae7a3664005bde5d09bdded9810",image="google/cadvisor:latest@sha256:ed53f9f93bb52d64fa8f95daece42d95d18d1e48a8ec9237bd5944ed634348a1",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_cadvisor.ovhu53o07e3ofsv8lnpubxl48.bxopq382md0ojxcpsfoll0c9x"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="ci-prod",container_label_com_docker_swarm_node_id="s6fa816uqnv50po1fcu88dhzb",container_label_com_docker_swarm_service_id="zlwczhf5ixmhirtquuloa5pdl",container_label_com_docker_swarm_service_name="ci-prod_agent",container_label_com_docker_swarm_task_id="v80xfp7c6d3riqpw4pngp5n4y",container_label_com_docker_swarm_task_name="ci-prod_agent.s6fa816uqnv50po1fcu88dhzb.v80xfp7c6d3riqpw4pngp5n4y",container_label_org_label_schema_build_date="unknown",container_label_org_label_schema_description="Jenkins agent based on the Swarm plugin",container_label_org_label_schema_name="jenkins-swarm-agent",container_label_org_label_schema_schema_version="1.0",container_label_org_label_schema_url="https://github.com/vfarcic/docker-jenkins-slave-dind/blob/master/README.md",container_label_org_label_schema_usage="/src/README.md",container_label_org_label_schema_vcs_branch="unknown",container_label_org_label_schema_vcs_ref="unknown",container_label_org_label_schema_vcs_url="unknown",container_label_org_label_schema_vendor="vfarcic",container_label_org_label_schema_version="0.2.0",id="/docker/25176ca3819e7b27dd993f225731dc1b63e0d8704530cc28dd56d772843e5a5a",image="vfarcic/jenkins-swarm-agent:latest@sha256:6b3ff8ec249ef602471f25dea74b50621656a2d9a0c2750d2dc472a311b3b45a",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="ci-prod_agent.s6fa816uqnv50po1fcu88dhzb.v80xfp7c6d3riqpw4pngp5n4y"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="qpz7jav2chu05hv152x9fjydd",container_label_com_docker_swarm_service_name="monitor-exporters_node-exporter",container_label_com_docker_swarm_task_id="vae4358kvcaqcc56qx7srk55f",container_label_com_docker_swarm_task_name="monitor-exporters_node-exporter.47o07qastg7thfsmgtaf66l0w.vae4358kvcaqcc56qx7srk55f",container_label_org_label_schema_build_date="2017-05-18T23:59:50Z",container_label_org_label_schema_name="node-exporter",container_label_org_label_schema_schema_version="1.0",container_label_org_label_schema_url="https://github.com/basi/docker-node-exporter/blob/master/README.md",container_label_org_label_schema_usage="/src/README.md",container_label_org_label_schema_vcs_branch="HEAD",container_label_org_label_schema_vcs_ref="118ed1b61870dbb8c36d0fe5e23aee23abbb62e6",container_label_org_label_schema_vcs_url="https://github.com/bvis/docker-node-exporter.git",container_label_org_label_schema_vendor="Basi",container_label_org_label_schema_version="v1.14.0",id="/docker/a5a5fcfe6ffac31037cc6f746daef9554f825df069781e2cf8e8bd11beeab670",image="basi/node-exporter:v1.14.0@sha256:9517cfc33b1a90d071910112606b48946755fc3d13627dc07e61061ac18c7cfc",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="monitor-exporters_node-exporter.47o07qastg7thfsmgtaf66l0w.vae4358kvcaqcc56qx7srk55f"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="proxy",container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="z87tfvnynw9r0vfenmi89rao7",container_label_com_docker_swarm_service_name="proxy_proxy",container_label_com_docker_swarm_task_id="snfbpridwajlh071tgx3eyi5c",container_label_com_docker_swarm_task_name="proxy_proxy.2.snfbpridwajlh071tgx3eyi5c",id="/docker/4a33393acee4747591f07bd284f5dc7d385adcec366b1b7afead407c3a73ec7d",image="vfarcic/docker-flow-proxy:17.11.11-56@sha256:a6a22ee3cbfa5ce0e996ff162069f3f0ccc31b21f750a00da355bf3cc758366a",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="proxy_proxy.2.snfbpridwajlh071tgx3eyi5c"}

 This is host


On Thursday, November 23, 2017 at 3:34:03 PM UTC+2, Tom Wilkie wrote:
Can you send the output of the two queries `container_last_seen{id=~"/docker.*"}` and `host`?

Tanks

Tom

On Thu, Nov 23, 2017 at 1:23 PM dave08 <dav...@gmail.com> wrote:
Yes, the host metric contains two labels, instance and host, I need to match by instance and get host value into the container_last_seen metric, but I'm getting no results... what could it be?

Thanks for answering!


On Wednesday, November 22, 2017 at 10:21:46 PM UTC+2, Tom Wilkie wrote:
Does Prometheus know where the containers are running?  On Kubernetes, this information can be scraped out of kube-state-metrics.  Here is an example:

  container_last_seen 
* on(namespace, pod_name) group_left(node) 
  label_replace(kube_pod_info, "pod_name", "$1", "pod", "(.*)")

Unfortunately cAdvisor uses the pod_name label name and kube-state-metrics uses pod, so a label_replace is needed.

Cheers

Tom


On Wed, Nov 22, 2017 at 4:28 PM dave08 <dav...@gmail.com> wrote:
Anyone know how to get a list of containers running on each node (with host name..) w/ promql and grafana? I tried: `container_last_seen{id=~"/docker.*"} * ON(instance) GROUP_LEFT(host) host`, but it's empty...

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

dave08

unread,
Nov 23, 2017, 9:33:59 AM11/23/17
to Prometheus Developers
This is host

host{host="swarm-03",instance="10.0.2.22:9100",job="monitor-exporters_node-exporter"}1
host{host="swarm-01",instance="10.0.2.20:9100",job="monitor-exporters_node-exporter"}1
host{host="swarm-00",instance="10.0.2.21:9100",job="monitor-exporters_node-exporter"}1
host{host="swarm-02",instance="10.0.2.24:9100",job="monitor-exporters_node-exporter"}
container_last_seen{container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="s1e68evxilnc28hk1288rmo4r",container_label_com_docker_swarm_service_name="old_admin",container_label_com_docker_swarm_task_id="a4jarm4nf1xcq3mxwz5bygdt5",container_label_com_docker_swarm_task_name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5",container_label_io_webdevops_layout="8",container_label_io_webdevops_version="1.5.0",container_label_maintainer="inf...@webdevops.io",container_label_vendor="WebDevOps.io",id="/docker/c148af9f9bab69628a8a0bb591aab90f83d45fbbd65a0e66ccd74956582cf816",image="localhost:5000/admin-old:34@sha256:0c96ba0850a061c6d8e4702ccb03990f70c05b1323383ebb21094cd9dbaa3778",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5"}1511447432
container_last_seen{id="/docker/2b4d71ff4497036dc2797b92190f2e5396ed16d450667a2db74f3f9cfa0e684d",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="reverent_wright"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="ci-prod",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="4z5f1nnanbxesr5pjtesq4vyz",container_label_com_docker_swarm_service_name="ci-prod_registry",container_label_com_docker_swarm_task_id="xv68m0hfn5ktwzd80hxgdfefw",container_label_com_docker_swarm_task_name="ci-prod_registry.1.xv68m0hfn5ktwzd80hxgdfefw",id="/docker/f5838d88398b265b927cd56b23f621686ea810137668cb636b9747ff036d291c",image="registry:2.6.1@sha256:5eaafa2318aa0c4c52f95077c2a68bed0b13f6d2b464835723d4de1484052299",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="ci-prod_registry.1.xv68m0hfn5ktwzd80hxgdfefw"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="proxy",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="nwqnel71yezn82goamvniy1eq",container_label_com_docker_swarm_service_name="proxy_swarm-listener",container_label_com_docker_swarm_task_id="i5e7o2up0lsk7e1dnwbodbksw",container_label_com_docker_swarm_task_name="proxy_swarm-listener.1.i5e7o2up0lsk7e1dnwbodbksw",id="/docker/efebdf1a53aaf327d2884469d55b8f02dc4ae7561dcc275e4c212df42d6dd770",image="vfarcic/docker-flow-swarm-listener:17.11.11-23@sha256:ac27bcfba99963d118734f4e74885decf57f08eb92b7fba78164bcdf0466a209",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="proxy_swarm-listener.1.i5e7o2up0lsk7e1dnwbodbksw"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="0baqm7fvv5ffd7n9zdw584ltb",container_label_com_docker_swarm_service_name="elk_logspout",container_label_com_docker_swarm_task_id="wr354npxbktpv10ph6itjuqve",container_label_com_docker_swarm_task_name="elk_logspout.47o07qastg7thfsmgtaf66l0w.wr354npxbktpv10ph6itjuqve",id="/docker/a98ca3a9fcbdb730a2069c65509ca401eb7de16bd6ef6d657514f2abcd011522",image="gliderlabs/logspout:latest@sha256:9a807db8e9e484920c3228770da76b8af68e8704d8ca0965e68bf8cefc93cee1",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="elk_logspout.47o07qastg7thfsmgtaf66l0w.wr354npxbktpv10ph6itjuqve"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="monitor",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="382zl4p8snnet5z4j3lan7hx3",container_label_com_docker_swarm_service_name="monitor_prometheus",container_label_com_docker_swarm_task_id="ewjb1440hovex3pedu9ni7o0y",container_label_com_docker_swarm_task_name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y",container_label_maintainer="The Prometheus Authors <prometheus-developers@googlegroups.com>",id="/docker/a20b81ac0a25185f399129e32c76735bcb65d5fb6c7fbc283832ac4d2f06214b",image="vfarcic/docker-flow-monitor:17.11.12-27@sha256:82b7f70492a65658fd36c9b5095839aed8cf0380ed65d11a104495126e7517cf",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="monitor-exporters",container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="v0phsqgcypg72bdwwvhpa1fal",container_label_com_docker_swarm_service_name="monitor-exporters_cadvisor&quo

Tom Wilkie

unread,
Nov 23, 2017, 9:53:28 AM11/23/17
to dave08, Prometheus Developers
Hi Dave - the problem is the instance label values don't match - container_last_seen is giving you eg instance="10.0.2.5:8080", and host is giving you eg instance="10.0.2.22:9100".

I would try and use relabelling to make these more uniform, or use a label_replace in the query like I did.

Thanks

Tom

container_last_seen{container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="s1e68evxilnc28hk1288rmo4r",container_label_com_docker_swarm_service_name="old_admin",container_label_com_docker_swarm_task_id="a4jarm4nf1xcq3mxwz5bygdt5",container_label_com_docker_swarm_task_name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5",container_label_io_webdevops_layout="8",container_label_io_webdevops_version="1.5.0",container_label_maintainer="in...@webdevops.io",container_label_vendor="WebDevOps.io",id="/docker/c148af9f9bab69628a8a0bb591aab90f83d45fbbd65a0e66ccd74956582cf816",image="localhost:5000/admin-old:34@sha256:0c96ba0850a061c6d8e4702ccb03990f70c05b1323383ebb21094cd9dbaa3778",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5"}1511447432
container_last_seen{id="/docker/2b4d71ff4497036dc2797b92190f2e5396ed16d450667a2db74f3f9cfa0e684d",image="odoo-csv-import",instance="10.0.2.13:8080",job="monitor-exporters_cadvisor",name="reverent_wright"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="ci-prod",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="4z5f1nnanbxesr5pjtesq4vyz",container_label_com_docker_swarm_service_name="ci-prod_registry",container_label_com_docker_swarm_task_id="xv68m0hfn5ktwzd80hxgdfefw",container_label_com_docker_swarm_task_name="ci-prod_registry.1.xv68m0hfn5ktwzd80hxgdfefw",id="/docker/f5838d88398b265b927cd56b23f621686ea810137668cb636b9747ff036d291c",image="registry:2.6.1@sha256:5eaafa2318aa0c4c52f95077c2a68bed0b13f6d2b464835723d4de1484052299",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="ci-prod_registry.1.xv68m0hfn5ktwzd80hxgdfefw"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="proxy",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="nwqnel71yezn82goamvniy1eq",container_label_com_docker_swarm_service_name="proxy_swarm-listener",container_label_com_docker_swarm_task_id="i5e7o2up0lsk7e1dnwbodbksw",container_label_com_docker_swarm_task_name="proxy_swarm-listener.1.i5e7o2up0lsk7e1dnwbodbksw",id="/docker/efebdf1a53aaf327d2884469d55b8f02dc4ae7561dcc275e4c212df42d6dd770",image="vfarcic/docker-flow-swarm-listener:17.11.11-23@sha256:ac27bcfba99963d118734f4e74885decf57f08eb92b7fba78164bcdf0466a209",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="proxy_swarm-listener.1.i5e7o2up0lsk7e1dnwbodbksw"}1511447431
container_last_seen{container_label_com_docker_stack_namespace="elk",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="0baqm7fvv5ffd7n9zdw584ltb",container_label_com_docker_swarm_service_name="elk_logspout",container_label_com_docker_swarm_task_id="wr354npxbktpv10ph6itjuqve",container_label_com_docker_swarm_task_name="elk_logspout.47o07qastg7thfsmgtaf66l0w.wr354npxbktpv10ph6itjuqve",id="/docker/a98ca3a9fcbdb730a2069c65509ca401eb7de16bd6ef6d657514f2abcd011522",image="gliderlabs/logspout:latest@sha256:9a807db8e9e484920c3228770da76b8af68e8704d8ca0965e68bf8cefc93cee1",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="elk_logspout.47o07qastg7thfsmgtaf66l0w.wr354npxbktpv10ph6itjuqve"}1511447430
container_last_seen{container_label_com_docker_stack_namespace="monitor",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="382zl4p8snnet5z4j3lan7hx3",container_label_com_docker_swarm_service_name="monitor_prometheus",container_label_com_docker_swarm_task_id="ewjb1440hovex3pedu9ni7o0y",container_label_com_docker_swarm_task_name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y",container_label_maintainer="The Prometheus Authors <prometheus...@googlegroups.com>",id="/docker/a20b81ac0a25185f399129e32c76735bcb65d5fb6c7fbc283832ac4d2f06214b",image="vfarcic/docker-flow-monitor:17.11.12-27@sha256:82b7f70492a65658fd36c9b5095839aed8cf0380ed65d11a104495126e7517cf",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y"}1511447430

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/3caae929-a8d8-4eb1-ba41-e1ef37047132%40googlegroups.com.

dave08

unread,
Nov 23, 2017, 11:30:31 AM11/23/17
to Prometheus Developers
Thanks alot! Now I see the error! I guess the IPs of the instance label in containers_last_seen are not the ips of the hosts themselves... I guess that's not the case in Kubernetes...
container_last_seen{container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="s1e68evxilnc28hk1288rmo4r",container_label_com_docker_swarm_service_name="old_admin",container_label_com_docker_swarm_task_id="a4jarm4nf1xcq3mxwz5bygdt5",container_label_com_docker_swarm_task_name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5",container_label_io_webdevops_layout="8",container_label_io_webdevops_version="1.5.0",container_label_maintainer="inf...@webdevops.io",container_label_vendor="WebDevOps.io",id="/docker/c148af9f9bab69628a8a0bb591aab90f83d45fbbd65a0e66ccd74956582cf816",image="localhost:5000/admin-old:34@sha256:0c96ba0850a061c6d8e4702ccb03990f70c05b1323383ebb21094cd9dbaa3778",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5"}1511447432
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

Tom Wilkie

unread,
Nov 23, 2017, 12:04:10 PM11/23/17
to dave08, Prometheus Developers
In my case I set the instance label to be the pod name using relabelling, and use the kube_pod_info to map from pod name to node name.

Thanks

Tom

container_last_seen{container_label_com_docker_swarm_node_id="ovhu53o07e3ofsv8lnpubxl48",container_label_com_docker_swarm_service_id="s1e68evxilnc28hk1288rmo4r",container_label_com_docker_swarm_service_name="old_admin",container_label_com_docker_swarm_task_id="a4jarm4nf1xcq3mxwz5bygdt5",container_label_com_docker_swarm_task_name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5",container_label_io_webdevops_layout="8",container_label_io_webdevops_version="1.5.0",container_label_maintainer="in...@webdevops.io",container_label_vendor="WebDevOps.io",id="/docker/c148af9f9bab69628a8a0bb591aab90f83d45fbbd65a0e66ccd74956582cf816",image="localhost:5000/admin-old:34@sha256:0c96ba0850a061c6d8e4702ccb03990f70c05b1323383ebb21094cd9dbaa3778",instance="10.0.2.5:8080",job="monitor-exporters_cadvisor",name="old_admin.1.a4jarm4nf1xcq3mxwz5bygdt5"}1511447432
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/360efaea-d375-4044-b0c9-94cbecca31e4%40googlegroups.com.

Umang Goel

unread,
Jul 15, 2020, 1:48:24 AM7/15/20
to Prometheus Developers
Hello Dave,

How did you get these hosts as i see you are using docker swarm, even i am using docker swarm with dns service discovery, but i am not able to get these hosts in the labels. Can you please explain how to use this.
This is host

container_last_seen{container_label_com_docker_stack_namespace="monitor",container_label_com_docker_swarm_node_id="47o07qastg7thfsmgtaf66l0w",container_label_com_docker_swarm_service_id="382zl4p8snnet5z4j3lan7hx3",container_label_com_docker_swarm_service_name="monitor_prometheus",container_label_com_docker_swarm_task_id="ewjb1440hovex3pedu9ni7o0y",container_label_com_docker_swarm_task_name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y",container_label_maintainer="The Prometheus Authors <prometheus...@googlegroups.com>",id="/docker/a20b81ac0a25185f399129e32c76735bcb65d5fb6c7fbc283832ac4d2f06214b",image="vfarcic/docker-flow-monitor:17.11.12-27@sha256:82b7f70492a65658fd36c9b5095839aed8cf0380ed65d11a104495126e7517cf",instance="10.0.2.25:8080",job="monitor-exporters_cadvisor",name="monitor_prometheus.1.ewjb1440hovex3pedu9ni7o0y"}1511447430
Reply all
Reply to author
Forward
0 new messages