How Prometheus reload config works

[黄宇龙]

Hi Prometheus professionals,

I have some questions about reloading config at runtime.

1. Is it wise to do config reload frequently? eg. remote write target changes

2. When a config is reloaded, what will be restarted? 

3. Will remote write queues being recreated if a new config is added?  Will there any metric lost ?

Thanks.

Yulong

[Jiacai Liu]

Reloader's implementation is here:
-
https://github.com/prometheus/prometheus/blob/dd5a46aa47665d7b9dcb46d9c5e45152e9786222/cmd/prometheus/main.go#L506

As you can see, remote_adater/scrape/notify... will be
reloaded. It's not wise to do reload frequently, since it will
reinitialize those components. FYI, I reload once a day.

AKAIK, remote_adater will retry failed write, you can do some test
to check whether if will lose metrics duration reload.

[黄宇龙]

Thanks @jicai

After reading the codes, I feel less worried about reloading the config.

prometheus/write.go at dd5a46aa47665d7b9dcb46d9c5e45152e9786222 · prometheus/prometheus (github.com)  

prometheus/manager.go at dd5a46aa47665d7b9dcb46d9c5e45152e9786222 · prometheus/prometheus (github.com) 

In theory, reloading configs only impacts those components whose config has been changed. 

In my situation, the remote targets are added or removed frequently. 

But if reloading won't affect the existing unchanged target, it should be fine.

[Julien Pivotto]

Prometheus reloads should only reload what has changed. Recent releases also show the time reload takes for each component in the logs.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4361d7d5-651b-497c-afeb-91c8270c0bfdn%40googlegroups.com.