Python Multiprocess

[Esau Rodriguez]

Hi all, 

I'm not sure if I'm missing something but I'm seeing a behaviour with the python client and multiprocess using gunicorn and flask I'm not sure if I'm missing something or there's a bug there.

When I hit the endpoint producing the prometheus text to be scrapped I'm seeing 2 versions for the same metrics with different help texts. I would expect to see only one metric (the multiprocess one).

I thought I had something wrong in my setup so I tried it with e pretty simple project that I found here https://github.com/amitsaha/python-prometheus-demo/tree/master/flask_app_prometheus_multiprocessing (not my code).

I hit a random url and then the `/metrics` endpoint

You can see in the raw response down here we have 2 entries for each metric, with different `types` and `help` texts. In this example there really wasn't any processes but in the real example in prod we have several processes and we see the prometheus scraper `picks` a different value depending on the order of the response.

Am I missing something or is there a bug there? 

The raw response was:

<pre>

% curl --location --request GET 'http://localhost:5000/metrics'

# HELP request_latency_seconds Multiprocess metric

# TYPE request_latency_seconds histogram

request_latency_seconds_sum{app_name="webapp",endpoint="/metrics"} 0.00040912628173828125

request_latency_seconds_sum{app_name="webapp",endpoint="/"} 0.0001652240753173828

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.005"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.01"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.025"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.05"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.075"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.1"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.25"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.75"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="1.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="2.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="5.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="7.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="10.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="+Inf"} 1.0

request_latency_seconds_count{app_name="webapp",endpoint="/metrics"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.005"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.01"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.025"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.05"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.075"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.1"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.25"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.75"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="1.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="2.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="5.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="7.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="10.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="+Inf"} 1.0

request_latency_seconds_count{app_name="webapp",endpoint="/"} 1.0

# HELP request_count_total Multiprocess metric

# TYPE request_count_total counter

request_count_total{app_name="webapp",endpoint="/metrics",http_status="200",method="GET"} 1.0

request_count_total{app_name="webapp",endpoint="/",http_status="404",method="GET"} 1.0

# HELP request_count_total App Request Count

# TYPE request_count_total counter

request_count_total{app_name="webapp",endpoint="/metrics",http_status="200",method="GET"} 1.0

request_count_total{app_name="webapp",endpoint="/",http_status="404",method="GET"} 1.0

# HELP request_count_created App Request Count

# TYPE request_count_created gauge

request_count_created{app_name="webapp",endpoint="/metrics",http_status="200",method="GET"} 1.617798968564061e+09

request_count_created{app_name="webapp",endpoint="/",http_status="404",method="GET"} 1.61779898142748e+09

# HELP request_latency_seconds Request latency

# TYPE request_latency_seconds histogram

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.005"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.01"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.025"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.05"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.075"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.1"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.25"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="0.75"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="1.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="2.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="5.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="7.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="10.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/metrics",le="+Inf"} 1.0

request_latency_seconds_count{app_name="webapp",endpoint="/metrics"} 1.0

request_latency_seconds_sum{app_name="webapp",endpoint="/metrics"} 0.00040912628173828125

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.005"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.01"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.025"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.05"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.075"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.1"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.25"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="0.75"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="1.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="2.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="5.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="7.5"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="10.0"} 1.0

request_latency_seconds_bucket{app_name="webapp",endpoint="/",le="+Inf"} 1.0

request_latency_seconds_count{app_name="webapp",endpoint="/"} 1.0

request_latency_seconds_sum{app_name="webapp",endpoint="/"} 0.0001652240753173828

# HELP request_latency_seconds_created Request latency

# TYPE request_latency_seconds_created gauge

request_latency_seconds_created{app_name="webapp",endpoint="/metrics"} 1.617798968520208e+09

request_latency_seconds_created{app_name="webapp",endpoint="/"} 1.617798981426993e+09

</pre>

Kind regards,

Esau.

[Chris Marchbanks]

Hello,

It appears that there is a subtle bug/misunderstanding in the code that is linked, though that is possibly due to the multiprocess documentation not being clear enough. When the code specifies a registry for each metric (example), it is causing both a process-local metric, and the multi process metrics to be registered, so depending on which process handles the request you will get different responses for the metric with "Request latency" in the HELP text. In the multiprocess documentation this is what is meant by "Registries can not be used as normal, all instantiated metrics are exported". If you remove the registry=registry lines in the example you will see just the multiprocess output as expected.

You could also move the registry and MultiProcessCollector code into the request handler to make it clear that the registry used by the MultiProcessCollector should not have anything registered to it, as seen in the example in the multiprocess documentation I linked above.

Let me know if that was unclear or you have more questions,

Chris

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/dd6d0ea6-ef26-4aaf-b926-9490a109c15cn%40googlegroups.com.

[Esau Rodriguez]

Hi Chris,

thanks a lot for your response. I definitively didn't understand that line of the documentation. 

I just tested the solution on the simple test and it works as expected.

Thanks a lot,

Esau.

[Matthias Rampke]

Would you mind making a PR to improve the documentation? As the expert it is easy to write documentation that is technically correct but not helpful to an unsuspecting user; you are now in the best position to make this clear to the next one reading it.

Best,

MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d33a0134-0033-434e-a268-8ccaf9984b3en%40googlegroups.com.

[Esau Rodriguez]

Hi MR,

I definitively will try to make a PR for this as soon as I can.

Esau.