EC2 service discovery does discover my target nodes but unable to find them prometheus database

[amey...@gmail.com]

Hi Team,

I have a weird problem, EC2 service discovery does discover my target nodes (node exporter and cadvisor) by unable to find them prometheus database when I query the database. What could be the problem is there a way to debug this?

Thanks,

Amey

[amey...@gmail.com]

Here is my configuration file. I don't think its something to do with my configuration. Just tried with debug level didn't help much. 

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ['localhost:9090']
# EC2 service discovery for node metrics
  - job_name: 'ec2_sd_node_exporter_task'
    scrape_interval: 40s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    ec2_sd_configs:
    - region: eu-west-1
      profile: instance-profile
      #access_key: xxxxxxxxxxxxxxxxxxxx
      #secret_key: xxxxxxxxxxxxxxxxxxxx
      refresh_interval: 1m
      port: 9100
    relabel_configs:
    # Drop the instances which are stopped
    - source_labels: [__meta_ec2_instance_state]
      regex: stopped
      action: drop
    # Only monitor instances with a Name starting with "cgs-itt*"
    - source_labels: [__meta_ec2_tag_Name]
      regex: cgs-itt.*
      action: keep
    - source_labels: [__meta_ec2_tag_Name]
      target_label: Name
    - source_labels: [__meta_ec2_private_ip]
      regex: (.*)
      replacement: ${1}:9100
      action: replace
      target_label: __address__

# EC2 service discovery for container metrics
  - job_name: 'ec2_sd_container_task'
    scrape_interval: 40s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    ec2_sd_configs:
    - region: eu-west-1
      profile: instance-profile
      refresh_interval: 1m
      port: 8572
    relabel_configs:
    - source_labels: [__meta_ec2_instance_state]
      regex: stopped
      action: drop
    - source_labels: [__meta_ec2_tag_Name]
      regex: cgs-itt.*
      action: keep
    - source_labels: [__meta_ec2_tag_Name]
      target_label: Name
    - source_labels: [__meta_ec2_private_ip]
      regex: (.*)
      replacement: ${1}:8572
      action: replace
      target_label: __address__

[amey...@gmail.com]

Hey guys,

Looks like I found the issue. The node exporter and advisor are running on host (not in docker containers). What are the privileged flags that needs to be passed during the node exporter startup ? I have tried passing these but no luck. --path.procfs /proc --path.sysfs /sys --collector.runit.servicedir="/etc/service

Can someone advise me how to run the node exporters and cadvisor on host with proper parameters (centos 7). 

Thanks,

Amey