File_sd_config sync

[akshay sharma]

Hi, 

I have two instances of Prometheus running with one service. 

with both instances, one sidecar is running which is updating the config file(file_sd_config) with the target information based on the request(POST, CURL). 

so while doing multiple post requests, the request is reaching through service to any of the instances in a round-robin fashion. 

so both the instance has config file but with different target information.. so is there any way we can sync the config file between the two instances? 

as it is giving issues while removing and updating the existing target. using curl request. 

so please help, Is there any way we can sync the config file?

[Brian Candler]

More generally you're asking "how do I write an application which accepts updates to an inventory and makes the same updated data available to multiple containers?"

There are many possible solutions, but ultimately you need a single source of truth which the other containers refer to or receive a copy of.

Having a single file that you share using NFS is one way. Exposing it using HTTP and having the other containers copy it is another way. Pushing out the file using rsync or scp is another way. Putting the data in a shared SQL database is another. Posting the target list via a publish-subscribe server like NATS or RabbitMQ is another. You could publish the list using Kubernetes native mechanisms like ConfigMap (but note that has a maximum size of 1 MiB)

Given that this is Prometheus we're talking about, I suggest you look at the http_sd_config instead of file_sd_config.

- Make one instance of your container which accepts the target updates using HTTP POST (you already have this)

- Modify it so that it exposes the current full list of targets via HTTP GET

- Configure all the Prometheus servers with http_sd_config to fetch this list of targets periodically

In other words, you have one "inventory" service which all the other "Prometheus" services talk to. This is reasonably robust: if the inventory service goes down, the Prometheus servers will continue to scrape using the last known set of targets.

Aside: if you were using Netbox as your central inventory, then you could use netbox-plugin-prometheus-sd to get this functionality.

[Brian Candler]

Yet another way is to send your POST updates to every container separately: that is, use the pod IP address rather than the service IP address as the target for your "curl", and repeat it for every container.

However you then need to deal with the issues of:

- how do you keep track of which container IPs to POST to? (Maybe you use the Kubernetes API and query for specific pod labels)

- what happens if a container is unavailable when you POST an update to it? Do you remember it's missing and retry later?

- what happens when a new container starts, or an existing container restarts? How does it get its initial inventory?