namespaces vs. label for microservices

[cpogo...@gmail.com]

We're having a big farm of microservices, which are monitored with prometheus. I still have a concern though regarding the naming convention described here:

https://prometheus.io/docs/practices/naming/

Let's say I have a 2 microservices: blogpost and auth.

So using the rule "For metrics specific to an application, the prefix is usually the application name itself." - i create 2 metrics:

* blogpost:posts_total
* auth:users_total

Now for each of them I need a metrics http_request_duration_seconds, which is recommended as having no prefix. Thus I label it with appliction names.

* http_request_duration_seconds{app="blogpost"}
* http_request_duration_seconds{app="blogpost"}

As a result I have a totally messed up bunch of metrics partially with, partially without labels/namespaces related to the same services. And locating them in Grafana becomes quite a challenge.

Wouldn't it be easier to just use namespaces ALWAYS?

P.S.: I would be happy to do so, but as long as this stays a recommendation, it's hard to get things through like this: https://github.com/siimon/prom-client/issues/51

[Brian Brazil]

On 30 November 2016 at 20:20, <cpogo...@gmail.com> wrote:

We're having a big farm of microservices, which are monitored with prometheus. I still have a concern though regarding the naming convention described here:

https://prometheus.io/docs/practices/naming/

Let's say I have a 2 microservices: blogpost and auth.

So using the rule "For metrics specific to an application, the prefix is usually the application name itself." - i create 2 metrics:

* blogpost:posts_total
* auth:users_total

Now for each of them I need a metrics http_request_duration_seconds, which is recommended as having no prefix. Thus I label it with appliction names.

* http_request_duration_seconds{app="blogpost"}
* http_request_duration_seconds{app="blogpost"}

In this case, the labels would be target labels coming from service discovery/relabelling. Thus these labels would also be on the app-specific metrics.

As a result I have a totally messed up bunch of metrics partially with, partially without labels/namespaces related to the same services. And locating them in Grafana becomes quite a challenge.

Wouldn't it be easier to just use namespaces ALWAYS?

What do you mean by namespaces here?

Brian

 

P.S.: I would be happy to do so, but as long as this stays a recommendation, it's hard to get things through like this: https://github.com/siimon/prom-client/issues/51

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/03db2ae3-f4ca-4ae0-adc6-d130cc3f98fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[cpogo...@gmail.com]

Brian, thanks for the quick reply!

>
> What do you mean by namespaces here?
>

It's my interpretation of "The metric name should have a (single-word) application prefix relevant to the domain the metric belongs to. The prefix is sometimes referred to as namespace by client libraries."

In my example namespaces are "blogpost:" and "auth:". What I find really handy is that in Grafana you just start typing "blogpost:" and it will suggest you all the metrics starting with it, thus all metrics read from the app as long as convention is kept.

> In this case, the labels would be target labels coming from service discovery/relabelling. Thus these labels would also be on the app-specific metrics.

Yes. I even have that implemented. But I lose the nice feature of Grafana mentioned above. Finding my own 10 metrics among 1000 of metrics from all microservices becomes a challenge.

If a label was the right way to go to separate microservices, may be it could be a reason for Grafana to make a filter by label when looking for a metric. But i struggle to find this kind of recommendation on the naming convention page either.

[Brian Brazil]

On 30 November 2016 at 21:01, <cpogo...@gmail.com> wrote:

Brian, thanks for the quick reply!

>
> What do you mean by namespaces here?
>
It's my interpretation of "The metric name should have a (single-word) application prefix relevant to the domain the metric belongs to. The prefix is sometimes referred to as namespace by client libraries."

In my example namespaces are "blogpost:" and "auth:". What I find really handy is that in Grafana you just start typing "blogpost:" and it will suggest you all the metrics starting with it, thus all metrics read from the app as long as convention is kept.

That's not the Prometheus convention, and usually a very small proportion of metrics from an application would be from the application itself against its libraries and dependencies.

 

> In this case, the labels would be target labels coming from service discovery/relabelling. Thus these labels would also be on the app-specific metrics.

Yes. I even have that implemented. But I lose the nice feature of Grafana mentioned above. Finding my own 10 metrics among 1000 of metrics from all  microservices becomes a challenge.

If a label was the right way to go to separate microservices, may be it could be a reason for Grafana to make a filter by label when looking for a metric. But i struggle to find this kind of recommendation on the naming convention page either.

That'd be a feature request for Grafana.  What you suggest is a naming anti-pattern in Prometheus.

--

Brian Brazil

www.robustperception.io

[Matthias Rampke]

To elaborate a bit on what Brian is saying:

"metrics specific to the application itself" refers to them being specific to the business logic. For example, a metric about the number of blog posts could be "blogpost_count". For HTTP metrics, the "domain" is HTTP, so no matter which application they come from you should use "http_request_…".

Through the labels added during discovery / ingestion you can match this all up.

In Grafana, you can then use the templating functionality and the special "label_values" query form to automatically discover all appropriate labels and select them interactively / via the URL. Obviously, such a generic dashboard would not include the business logic specific metrics mentioned above. When building an application specific dashboard, use the same templating variables (so you can just clone the graphs) but set them to fixed values where appropriate.

/MR

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLpPY8zyOx9k79dGEGurWvOBOSYUx-FowQF-OumM6S7UOA%40mail.gmail.com.