Prometheus isn’t scrapping cadvisor and node-exporter

[Mohd Rashid]

Hi All,

Need help

When i click on prometheus URL on target page it showing below message.

cadvisor (0/1 up) and node-exporter (0/1 up) are showing in Prometheus URL

Here is the my filename.yml file

version: '3.2' services: prometheus: image: prom/prometheus:latest # container_name: monitoring_prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--web.console.libraries=/usr/share/prometheus/console_libraries' - '--web.console.templates=/usr/share/prometheus/consoles' volumes: - /Prometheus/alert.rules:/etc/prometheus/alert.rules - /Prometheus/container.yml:/etc/prometheus/container.yml - /Prometheus/diskusage.yml:/etc/prometheus/diskusage.yml - ./prometheus.yml:/etc/prometheus/prometheus.yml ports: - 9090:9090 node-exporter: image: prom/node-exporter:latest # container_name: monitoring_node_exporter volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro ports: - 9091:9091 command: - '--path.procfs=/host/proc' - '--path.sysfs=/host/sys' - '--path.rootfs=/host' - '--collector.filesystem.ignored-mount-points="^(/rootfs|/host|)/(sys|proc|dev|host|etc)($|/)"' - '--collector.filesystem.ignored-fs-types="^(sys|proc|auto|cgroup|devpts|ns|au|fuse\.lxc|mqueue)(fs|)$"' cadvisor: image: google/cadvisor:latest # container_name: monitoring_cadvisor ports: - 9092:9092 volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro grafana: image: grafana/grafana:latest # image: grafana-custom:latest # container_name: monitoring_grafana volumes: - grafana-data:/var/lib/grafana # - ./data/grafana:/var/lib/grafana # - ./grafana/provisioning:/etc/grafana/provisioning # - ./defaults.ini:/usr/share/grafana/conf/defaults.ini ports: - 3000:3000 environment: - "GF_SMTP_ENABLED=true" - "GF_SMTP_HOST=smtp.gmail.com:465" - "GF_SMTP_USER=te...@gmail.com" - "GF_SMTP_PASSWORD=xxxxxx" alertmanager: image: prom/alertmanager # privileged: true volumes: - ./alertmanager.yml:/alertmanager.yml command: - '--config.file=/alertmanager.yml' ports: - '9093:9093' karma-alert-dashboard: # image: cloudflare/unsee image: lmierzwa/karma:latest environment: # - "ALERTMANAGER_URIS=default:http://alertmanager:9093" - "ALERTMANAGER_URI=http://alertmanager:9093" ports: - '9094:9094' volumes: grafana-data:

I have deploy 5 docker service through docker stack -c filename.yml test, All services and docker container are UP.

First 3 are connected together and are working fine

Grafana = http://SERVERIP:3000/ - 3000

Prometheus = http://SERVERIP:9090/ - 9090
AlertManager = http://SERVERIP:9093/ - 9093

Last 3 are Not connected together and are Not working fine because of port issue in docker container and in docker service also

Karma = http://SERVERIP:9094/ - 8080

Node Exporter= http://SERVERIP:9100/ - 9100
CAdvisor = http://SERVERIP:8080/ - 8080

Here is the Screenshot

image1248×385 88.4 KB

When i check ports number 3 services are showing different in terminal but in my filename.yml everything is correct can anyone let me what is the issue and where i’m wrong

When i check logs of 3 docker container

1. docker logs cadvisor

Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory

1. node-exporter is not showing any logs.
2. Here is the logs of lmierzwa/karma

level=info msg="Version: v0.86-83-gf5e5269f" level=info msg="Configured Alertmanager source" name=default proxy=false readonly=false uri=http://alertmanager:9093 level=info msg="Initial Alertmanager collection" level=info msg="Pulling latest alerts and silences from Alertmanager" level=info msg="Collecting alerts and silences" alertmanager=default level=info msg="GET request" timeout=40 uri=http://alertmanager:9093/metrics level=info msg="Upstream version" alertmanager=default version=0.22.2 level=info msg="Got silences" alertmanager=default duration=0.000204419 silences=0 level=info msg="Detecting ticket links in silences" alertmanager=default silences=0 level=info msg="Collected alert groups" alertmanager=default duration=0.000201856 groups=0 level=info msg="Deduplicating alert groups" alertmanager=default groups=0 level=info msg="Processing deduplicated alert groups" alertmanager=default groups=0 level=info msg="Merging autocomplete hints" alertmanager=default hints=0 level=info msg="Collection completed" level=info msg="Done, starting HTTP server" level=info msg="Starting HTTP server" address=[::]:8080 level=info msg="Pulling latest alerts and silences from Alertmanager" level=info msg="Collecting alerts and silences" alertmanager=default level=info msg="GET request" timeout=40 uri=http://alertmanager:9093/metrics level=info msg="Upstream version" alertmanager=default version=0.22.2 level=info msg="Got silences" alertmanager=default duration=0.000155633 silences=0