confused about consul discovery

[Barry Kaplan]

From the example:

consul_sd_configs:
  - server:   '127.0.0.1:5361'
    services: ['auth', 'api', 'load-balancer', 'postgres']

What's not clear to me is how does prometheus know how to scrape these services? I would assume that, eg, auth is defined as a consul service to perform authorization. What path would prometheus use to get the metrics? 

For postgres wouldn't there need to be an exporter? How does prometheus know about that exporter based on that consul service def?

[Barry Kaplan]

I think this example is what I have been looking for:

https://github.com/docker-infra/prometheus-consul/blob/master/prometheus.conf.tmpl

We currently use datadog for most things that prometheus would do. Our system is very dynamic and nodes/services get added/removed all the time. Its the responsibility of the node/service (via ansible) to define their monitoring by extending the datadog agent config. Nothing need be done on the datadog side for new a node/service to get data captured. (Of course configuring visualizations/alerts is a manual process, but only when a new "job" kind is defined.)

We configuring logging similarly. A consul service advertises the rabbitmq instance that fronts logstash/ES. Each service that want's to log need only configure itself locally, there is no need to back to central ELK cluster to make config changes. 

I should also note that our ops monitoring system is shared across various environments (production, staging, on-prem-A, on-prem-B, etc.) So maintaining a global config would be quite painful.

My goal is do something similar with prometheus. I don't want to have to maintain a single global config that needs to be open to all other parts of the system. This seems much easier to achieve with an agent based system. But using consul-template to maintain the prometheus should be able to work just as well.

I don't see any posts here about using consul-template for this -- just that docker project from the link above. Is anybody using consul-template to maintain the prometheus config?

[Fabian Reinartz]

Regarding your first mail: Prometheus knows how to scrape these services based on the configured metrics_path for the surrounding scrape 

config (default "/metrics"). Consul is queried for all instances of the listed services. It then scrapes the returned instances on the

configured metrics endpoint. 

You have to configure things in a way that all nodes returned by your consul_sd_configs actually serve metrics on that 

endpoint. You can filter nodes via relabeling based on consul tags, for example.

And yes, for something like RabbitMQ or Postgres you need to run an exporter. There's no magic to know about those - you have to add them 

to your service discovery and configure Prometheus appropriately, i.e. add a rabbitmq-exp service that contains all your exporter instances.

Regarding your second mail: The repository you linked works with pre-0.14.0 configurations which are no longer valid. Back then, there

was no way to directly configure Prometheus with Consul and one had to restart Prometheus on every config change. By now you can

configure Prometheus to use Consul directly and reload the configuration at runtime via SIGHUP.

The fact that the tool you found rewrites and restarts based on Consul data doesn't change the fact the config is still global if you are using

a single Prometheus for everything. As far as I can see the watched job names are also hard coded - which leaves you with another global

config you have to maintain.

While this was certainly a neat tool back then, it's functionality is now basically built into Prometheus.

We plan (and have tested) to scrape Consul services without having to configure their name in Prometheus. You could then just attach the

"monitor" tag to every instance you register in your Consul and Prometheus would automatically scrape it on the /metrics endpoint (or

whichever else you configured). Is that closer to what you want to achieve?

[Barry Kaplan]

On Saturday, July 4, 2015 at 2:02:37 PM UTC+5:30, Fabian Reinartz wrote:

Prometheus knows how to scrape these services based on the configured metrics_path for the surrounding scrape config (default "/metrics").

Ah, I missed that. I was starting from the blog http://prometheus.io/blog/2015/06/01/advanced-service-discovery/ and that did not mention metrics_path other than in the context of DNS. I see it now in the config docs.

And yes, for something like RabbitMQ or Postgres you need to run an exporter. 

So in this case the ServiceAddress of rabbit would be the host to find the exporter? How is the exporter port defined when it discovered indirectly via a rabbit sevice? I assume it can't always be on a fixed port since we may need to run multiple exporters per host. And in our system, we run most things on mesos/marathon, so all ports are dynamically allocated.

 

The repository you linked works with pre-0.14.0 configurations which are no longer valid.

Yes, I realized that. My main point for referencing that was that it allowed me not to have to list explicit services in the prometheus config -- I could use all the power of consul-template to define jobs. Service discovery by tag would a big plus, and I'm guessing it may handle our all our use cases. 

 

As far as I can see the watched job names are also hard coded - which leaves you with another global

config you have to maintain.

That is just for that example. It need not be that way. With consul-template I can select services based on any of its properties, including tags. It also allows for extracting other job parameters from consul/kv. I do this for haproxy config: Kinds (== jobs) of services are defined in consul/kv tree, along with url path prefixes, health check info, timeouts, etc. These define the frontend and form form an outer loop for backends. The backend inner loop selects those services instances and declares the backends. Very flexible and quite easy to define. 

But again, with the ability to define multiple jobs, each with their metrics_path, etc and select via consul tag that should be quite flexible without having to add another layer of tooling.

 

Thanks much Fabian! Your reply was very helpful.

[xilarra...@qdqmedia.com]

> We plan (and have tested) to scrape Consul services without having to configure their name in Prometheus. You could then just attach the
> "monitor" tag to every instance you register in your Consul and Prometheus would automatically scrape it on the /metrics endpoint (or
> whichever else you configured). Is that closer to what you want to achieve?
>

Hi! looking at this, now you can left empty the service list and will scrape all the services (including consul default one :/), the question is, are any plans to implement the keyword tag specification so prometheus knows which ones are valid services to scrape?

Thank you!

[Brian Brazil]

You can use the drop and keep relabel action to filter based on consul tags.

Brian

 

Thank you!

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[xilarra...@qdqmedia.com]

I missed that! Works like a charm

Thank you!