Pushgateway metric name examples

[mhernand...@gmail.com]

Can someone post pushgateway metric name examples?

It would be great if these metrics follow the best naming convention.

I looked at the prometheus docs on naming convention for for pushgateway it's a little different because Idk if I should have the metric name include the job or only differentiate the jobs by label and keep the metric name the same (for ex. timing a function).

[Brian Brazil]

The usual naming rules apply, the start of a metric should be the name of the library it is coming from. In the case of batch jobs that's often also the name of the batch job as the library for most metrics will be the main function of the batch job, the same as you'd have in the main function of any other application. In all cases the jobs will also be differentiated by label. 

--

Brian Brazil

www.robustperception.io

[Miguel Hernandez]

Considering a celery job, would this be a good example?:

celery_job_duration_seconds{job="job_name1"}

celery_job_duration_seconds{job="job_name2"}

etc.

[Brian Brazil]

On 17 July 2018 at 17:38, Miguel Hernandez <mhernand...@gmail.com> wrote:

Considering a celery job, would this be a good example?:

celery_job_duration_seconds{job="job_name1"}

celery_job_duration_seconds{job="job_name2"}

That's okay if the metric is coming from celery itself.

Brian

 

etc.

On Tue, Jul 17, 2018 at 12:39 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 17 July 2018 at 00:33, <mhernand...@gmail.com> wrote:

Can someone post pushgateway metric name examples?

It would be great if these metrics follow the best naming convention.

I looked at the prometheus docs on naming convention for for pushgateway it's a little different because Idk if I should have the metric name include the job or only differentiate the jobs by label and keep the metric name the same (for ex. timing a function).

The usual naming rules apply, the start of a metric should be the name of the library it is coming from. In the case of batch jobs that's often also the name of the batch job as the library for most metrics will be the main function of the batch job, the same as you'd have in the main function of any other application. In all cases the jobs will also be differentiated by label. 

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

[Miguel Hernandez]

The metric is coming from client instrumented code within a celery task (python function), then pushed to the gateway with a job name as a label identifier.  Since the job duration is handled by client code instrumentation, would this mean that the metric is NOT coming from celery itself?  If so, would the aforementioned metric name example still be okay?

[Brian Brazil]

On 17 July 2018 at 20:51, Miguel Hernandez <mhernand...@gmail.com> wrote:

The metric is coming from client instrumented code within a celery task (python function), then pushed to the gateway with a job name as a label identifier.  Since the job duration is handled by client code instrumentation, would this mean that the metric is NOT coming from celery itself? 

Yes.

 

If so, would the aforementioned metric name example still be okay?

No, it's not okay. It should follow the naming of whatever the client code is.

Brian

 

On Tue, Jul 17, 2018 at 9:55 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 17 July 2018 at 17:38, Miguel Hernandez <mhernand...@gmail.com> wrote:

Considering a celery job, would this be a good example?:

celery_job_duration_seconds{job="job_name1"}

celery_job_duration_seconds{job="job_name2"}

That's okay if the metric is coming from celery itself.

Brian

 

etc.

On Tue, Jul 17, 2018 at 12:39 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 17 July 2018 at 00:33, <mhernand...@gmail.com> wrote:

Can someone post pushgateway metric name examples?

It would be great if these metrics follow the best naming convention.

I looked at the prometheus docs on naming convention for for pushgateway it's a little different because Idk if I should have the metric name include the job or only differentiate the jobs by label and keep the metric name the same (for ex. timing a function).

The usual naming rules apply, the start of a metric should be the name of the library it is coming from. In the case of batch jobs that's often also the name of the batch job as the library for most metrics will be the main function of the batch job, the same as you'd have in the main function of any other application. In all cases the jobs will also be differentiated by label. 

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

[Miguel Hernandez]

Here's a concrete example following from our discussion.
# prom_metrics.py file
CELERY_REGISTRY = CollectorRegistry()

g = Gauge(
    'celery_job_duration_seconds', 'Time it takes to complete a celery job.', registry=CELERY_REGISTRY
)

For example, I am initializing a gauge here with the metric name and when I push to gateway below, differentiate the metric based on the job label.

# still in prom_metrics.py file
def push_stats(job_name):
    """
    Call this function at the end of each Celery task;
    if you have batch tasks, have another function to push those to the batch registry
    """
    push_to_gateway('localhost:9091', job=job_name, registry=CELERY_REGISTRY)
    return

As you can see above, the push_stats function is called from within an arbitrary celery task, passing in a job name to help identify that celery task.

from prom_metrics import g, push_stats

# Different file containing celery task
@shared_task
def do_something():
    job_name = 'some_module.do_something'
    with g.time():
        do_something_else()
    push_stats(job_name)

     push_stab_na

In this case, what would you recommend the metric name be when pushing to the pushgateway?  

I feel that changing the metric name to include the celery task name (i.e. do_something_duration_seconds) would be redundant since the job label would contain that information either way.

Let me know what you think.

Btw, thank you for all your help so far.

[Miguel Hernandez]

UPDATE TO PREVIOUS EMAIL, ADDED the prometheus_client imports

Here's a concrete example following from our discussion.
# prom_metrics.py file

from prometheus_client import Gauge, CollectorRegistry, push_to_gateway
CELERY_REGISTRY = CollectorRegistry()

[Miguel Hernandez]

And I'm not sure what you mean by "It should follow the naming of whatever the client code is."  Concrete examples would be very helpful.

[Brian Brazil]

Usually you'd base it off the path and name of the file containing the metric.

 

I feel that changing the metric name to include the celery task name (i.e. do_something_duration_seconds) would be redundant since the job label would contain that information either way.

Redundancy like that is not uncommon for batch jobs, as the metrics almost always come from main functions and batch jobs tend to be singletons.

Brian

 

Let me know what you think.

Btw, thank you for all your help so far.

--

Brian Brazil

www.robustperception.io

[Miguel Hernandez]

So how about this:  path_to_file_name_job_duration_seconds{job="do_something"}

[Brian Brazil]

On 18 July 2018 at 04:42, Miguel Hernandez <mhernand...@gmail.com> wrote:

So how about this:  path_to_file_name_job_duration_seconds{job="do_something"}

Something like that would work, though exactly which bits of the path it makes sense to include varies.

--

Brian Brazil

www.robustperception.io

[Miguel Hernandez]

Well, there's a folder called "celery" in our project which contains the celery tasks that we want to get metrics from.  For instance, let's assume that the project I'm working on is called "omedeus."  What do you think about this:  omedeus_celery_file_name_job_duration_seconds{job="do_something"} where file_name is the name of the file that contains the celery task python function.

[Brian Brazil]

On 18 July 2018 at 06:34, Miguel Hernandez <mhernand...@gmail.com> wrote:

Well, there's a folder called "celery" in our project which contains the celery tasks that we want to get metrics from.  For instance, let's assume that the project I'm working on is called "omedeus."  What do you think about this:  omedeus_celery_file_name_job_duration_seconds{job="do_something"} where file_name is the name of the file that contains the celery task python function.

That sounds about right. The "celery" does still feel a bit weird, as that's a bit like putting kuberentes in the metric name of every application that happens to run on kuberentes - but on the other hand it does appear to be your structure.

 Brian

--

Brian Brazil

www.robustperception.io

[Miguel Hernandez]

Yes, it is a part of our structure.  I see what you mean about celery feeling weird.  However, we want to be able to identify that the metric is from within a celery task/job so that it will be easier to find when setting up a 'Celery task/job stats' graph on Grafana.

What would you say is best?

[Brian Brazil]

On 18 July 2018 at 22:11, Miguel Hernandez <mhernand...@gmail.com> wrote:

Yes, it is a part of our structure.  I see what you mean about celery feeling weird.  However, we want to be able to identify that the metric is from within a celery task/job so that it will be easier to find when setting up a 'Celery task/job stats' graph on Grafana.

What would you say is best?

If you want to identify whether a metric comes from a specific type of task/job, that's the role of a target label. Metric names tell you which line of code produced a metric, target labels tell you which process it was scraped from.

Brian

--

Brian Brazil

www.robustperception.io