Prometheus dashboard loads however in target section promtheus isn't UP.

[Isabel Noronha]

Hey,

I have been working on prometheus for the past 3 months.

This is the first time where I have come across such a problem where prometheus is running inside a container.

However, when I check the dashboard it says"context deadline exceeded".It is not even able to discover node_exporter.

Can anyone help me with a possible cause?

Docker version 19.03.5

Docker-compose.yml

version: '3.7'

volumes:

    prometheus_data: {}

    grafana_data: {}

networks:

  front-tier:

  back-tier:

services:

  prometheus:

    image: prom/prometheus:v2.18.0

    volumes:

      - ./prometheus/:/etc/prometheus/

      - prometheus_data:/prometheus

    command:

      - '--config.file=/etc/prometheus/prometheus.yml'

      - '--web.listen-address=:9010'

      - '--storage.tsdb.path=/prometheus'

      - '--web.console.libraries=/usr/share/prometheus/console_libraries'

      - '--web.console.templates=/usr/share/prometheus/consoles'

    ports:

      - 9010:9010

    links:

      - cadvisor:cadvisor

      - alertmanager:alertmanager

    depends_on:

      - cadvisor

    networks:

      - back-tier

    restart: always

#    deploy:

#      placement:

#        constraints:

#          - node.hostname == ${HOSTNAME}

  node-exporter:

    image: prom/node-exporter

    volumes:

      - /proc:/host/proc:ro

      - /sys:/host/sys:ro

      - /:/rootfs:ro

    command:

      - '--path.procfs=/host/proc'

      - '--path.sysfs=/host/sys'

      - --collector.filesystem.ignored-mount-points

      - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"

    ports:

      - 9100:9100

    networks:

      - back-tier

    restart: always

    deploy:

      mode: global

  alertmanager:

    image: prom/alertmanager

    ports:

      - 9093:9093

    volumes:

      - ./alertmanager/:/etc/alertmanager/

    networks:

      - back-tier

    restart: always

    command:

      - '--config.file=/etc/alertmanager/config.yml'

      - '--storage.path=/alertmanager'

#    deploy:

#      placement:

#        constraints:

#          - node.hostname == ${HOSTNAME}

  cadvisor:

    image: google/cadvisor

    volumes:

      - /:/rootfs:ro

      - /var/run:/var/run:rw

      - /sys:/sys:ro

      - /var/lib/docker/:/var/lib/docker:ro

    ports:

      - 8080:8080

    networks:

      - back-tier

    restart: always

    deploy:

      mode: global

#  grafana:

#    image: grafana/grafana:6.7.0

#    user: "472"

#    depends_on:

#      - prometheus

#    ports:

#      - 3000:3000

#    volumes:

#      - grafana_data:/var/lib/grafana

#      - ./grafana/provisioning/:/etc/grafana/provisioning/

#    env_file:

#      - ./grafana/config.monitoring

#    networks:

#      - back-tier

#      - front-tier

#    restart: always

prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    monitor: my-project
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093
    scheme: http
    timeout: 10s
    api_version: v1
rule_files:
- /etc/prometheus/alert.rules
scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 40s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - X.X.X.X:9010
- job_name: cadvisor
  honor_timestamps: true
  scrape_interval: 40s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - X.X.X.X:8080
- job_name: node-exporter
  honor_timestamps: true
  scrape_interval: 40s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - X.X.X.X:9100

[Christian Hoffmann]

Hi,

On 6/5/20 11:45 AM, Isabel Noronha wrote:
> I have been working on prometheus for the past 3 months.
> This is the first time where I have come across such a problem where
> prometheus is running inside a container.
> However, when I check the dashboard it says"context deadline
> exceeded".It is not even able to discover node_exporter.
>
> Can anyone help me with a possible cause?

This is often caused by network problems, e.g. firewalls dropping
connection attempts.

Try verifying that the address in your service discovery can really
reach the target (i.e.: not a loopback address, correct ip
address/resolvable dns name, no firewalls inbetween).

Kind regards,
Christian

[Isabel Noronha]

Thanks a lot !!

I completely forgot that the system was rebooted yesterday and I lost track of firewall.

After disabling firewall and flushing iptables it works now.