Context and Timeouts in client_golang

[Joe Adams]

I would like to propose adding context to the client_golang promhttp.Handler functions, through a new set of functions in the promhttp package. I believe that there was some discussion around this, but not for a long time.

The goal of this change is for exporters other metric providers to have an opportunity to understand cancellation. The cancellation could be through the http.Request.Context() being cancelled when the http request is closed from the client, but Prometheus also provides a header "X-Prometheus-Scrape-Timeout-Seconds" on scrape requests. Today, if an exporter has an expensive collector, there is no way to know that the results will be thrown away and that collection can be stopped in the event that the scrape from Prometheus has timed out.[1]

I have only begun to dig into the work that would be required to propagate the context the whole way down through a Registry and I can already tell that it would not be trivial. The general idea would be to add context versions of promhttp.Handler() and the supporting functions that would pull the context from the http.Request and optionally create a child context with the timeout/deadline from the X-Prometheus-Scrape-Timeout-Seconds header. Downstream, the prometheus.Registry would also need to understand context. I think this may only need an additional GatherCtx() func, but the downstream prometheus.Collector interface would probably need a context-aware counterpart.

I believe that this is all possible without any breaking changes, but I have not researched enough to know for sure. I want to put this out to the community and maintainers to get some feedback before spending too much time trying to make these changes.

Joe Adams

@sysadmind

1. https://github.com/prometheus-community/postgres_exporter/pull/558

[Bjoern Rabenstein]

Hi Joe,

I think you got the below quite right in general. The most relevant
prior discussion I know of is starting at this GH comment (hidden in
the issue about not using channels in the Collector interface):
https://github.com/prometheus/client_golang/issues/228#issuecomment-475214970

Fundamentally, I think the addition of contexts is straight forward
and indeed very desirable, for all the reasons you listed, and maybe
even more.

The reason why it hasn't happened yet is that it requires changes to
so many parts of the whole library. So we end up with a lot of very
similar almost-duplicates of functions/methods, which pollutes the
namespace, confuses the user, and is generally a design smell (in
particular in interfaces), all because we have to keep backwards
compatibility for this incredibly widely used library.

So the plan was to do it all in the glorious v2 rewrite (for which I
piled up a lot of those changes in design or additional features that
would require weird wrinkles to not make them breaking changes, see
the v2 milestone:
https://github.com/prometheus/client_golang/milestone/2 ).

However, for various reason, the progress on the v2 rewrite
stalled. We have new maintainers now (@bwplotka and @kakkoyon on GH),
and it's their call how to proceed here. My gut feeling is that they
tend to rather add more features to v1, even if it requires wrinkles,
than expedite a v2 rewrite. Perhaps they will follow up
here. Otherwise, try to get in touch with them is nome other
way. Maybe file a feature or start a discussion in GH.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/25a3ef44-6f02-492f-8f6c-28383cd0d6d8n%40googlegroups.com.


--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in