Prometheus won't start after changing config yaml.

[Gurdip sira]

Hi,

I have prometheus installed on a windows server. Whenever I try to change the config file, like to add another target to be scraped, the service never starts again.

In event logs I just get an error saying service coukdn't be started. I know the yaml file is valid.

Anyone got any ideas?

[Brian Candler]

You've made an error in your config file.  It can be valid YAML (i.e. a YAML editor says it's OK) but invalid semantically for Prometheus.

Show the actual error message from prometheus when you try to start it, and the actual yaml file before and after (exact copy-paste, with exact formatting)

[Gurdip sira]

Hi,

This is the error:

Unable to restart the application: "C:\prometheus-2.21.0.windows-amd64.tar\prometheus-2.21.0.windows-amd64\prometheus-2.21.0.windows-amd64\prometheus.exe" exited immediately after it was started.

This message was logged from an unregistered version of AlwaysUp. Please see https://www.CoreTechnologies.com/products/AlwaysUp/ for registration information.

Standard yaml file:

# my global config

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

  # - "first_rules.yml"

  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  - job_name: 'prometheus'

    static_configs:

    - targets: ['10.0.0.95:3182']

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.

    static_configs:

    - targets: ['localhost:9090']

What i want to do is add a new target, so I do:

# my global config

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration

alerting:

  alertmanagers:

  - static_configs:

    - targets: 127.0.0.1:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

  # - "first_rules.yml"

  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.

    static_configs:

    - targets: ['10.0.0.95:9182/metrics' ]

    

  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.

    static_configs:

    - targets: ['localhost:9090' ]

The only thing I have noticed is the prometheus server can't seem to reach that /metrics endpoint, but would that cause this issue?

[Brian Candler]

No, you need to look at the logs *from prometheus itself* as it starts up, not from your service manager (since it seems your service manager doesn't capture the output of the service as it starts up).

Run prometheus in the foreground, with whatever flags your service manager normally passes to it.

Under Linux you would run it directly from the shell: e.g.

/opt/prometheus/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data/

I don't do Windows, so adapt as necessary.