Automatic update in service discovery for prometheus

[pratyu...@gmail.com]

If my node instances (IP address) gets dynamically changed then how can it be automatically updated in service discovery so that prometheus can reload the changed instances on specified refresh interval.

How can we do that (automatically updating changed instances in service discovery) in consul_sd_config, file_sd_config and ec2_sd_config?

Which service discovery (consul, file, ec2) would be easier to handle in that case?

[Stuart Clark]

It really depends what you are doing for service discovery currently. Do
you use Consul or EC2 instances at the moment?

IP addresses don't generally change once a node is running. Consul would
generally be configured to register services on node (or service) startup.

As soon as Consul is updated Prometheus will see the new node (as well
as stopping scraping any removed nodes). The same is true for EC2 - new
instances will be reflected straight away in the scrape list.

The file SD is slightly different. While any change to that file will be
immediately seen by Prometheus it is up to you to change the file.
Popular methods include cron jobs, sidecar services or configuration
management tools. How quickly things change is then up how often those
additional tools run and discover differences. This is also dependent on
where the information is coming from & how often that is refreshed.

[pratyu...@gmail.com]

Hey Stuart,

I want to know that suppose I have a node running at any specific instance. I used the agent service register to add it to the consul. Now if the IP gets dynamically changed so then how can consul be updated automatically. And as soon as Consul is updated Prometheus will see the same node with a new address.

Can it be updated automatically ? or I have to change the service then do Consul reload and after that Prometheus will see the changed node address.

Thanks

[Stuart Clark]

On 2020-06-25 13:09, pratyu...@gmail.com wrote:
> Hey Stuart,
>
> I want to know that suppose I have a node running at any specific
> instance. I used the agent service register to add it to the consul.
> Now if the IP gets dynamically changed so then how can consul be
> updated automatically. And as soon as Consul is updated Prometheus
> will see the same node with a new address.
> Can it be updated automatically ? or I have to change the service then
> do Consul reload and after that Prometheus will see the changed node
> address.
>

IP addresses don't generally change while a machine is running, so it
would be enough to register in Consul on startup.

However, if machine's IP address does change you'd need to detect that
and trigger something to update Consul.

--
Stuart Clark

[pratyu...@gmail.com]

So if I am using AWS instance which provides dynamic IP address. My server of that project is stopped and restarted for any reason then how can we detect that and trigger something to update consul ?

[Stuart Clark]

On 2020-06-25 16:03, pratyu...@gmail.com wrote:
> So if I am using AWS instance which provides dynamic IP address. My
> server of that project is stopped and restarted for any reason then
> how can we detect that and trigger something to update consul ?
>

You mentioned registering with Consul on machine startup, so if a server
is stopped or restarted the Consul registration will be updated each
time the server boots.

--
Stuart Clark