Changing consensus on HTTP headers

[Julien Pivotto]

Hello,

We once again got a request last week regarding overwriting HTTP header.

This is a frequent request, even if it does not seem to affect a lot of
users.

However, I have crafted a pull request that changes that consensus and
makes HTTP headers configurable in the common HTTP client, with some
reserved headers.

In this pull request, I set 3 ways to set HTTP headers:

- With a map of strings->string, so HTTP headers can conveniently be
read back from the HTTP API (usecase: remote write tenant).
- With a map of strings->secret, so you can specify custom API key
fields and they won't be visible in the UI's.
- With a map of strings->filenames, so headers can be read from files.

The reserved headers come from what we implemented in Prometheus remote
write. I was the one putting that list on, and I think it's a reasonable
list. It excludes headers that Prometheus will generally write itself
and headers that would change the properties of the connection.

What does the community & team members think about this?

--
Julien Pivotto
@roidelapluie

[Bjoern Rabenstein]

On 28.11.22 11:29, Julien Pivotto wrote:
>
> However, I have crafted a pull request that changes that consensus and
> makes HTTP headers configurable in the common HTTP client, with some
> reserved headers.

For findability: https://github.com/prometheus/common/pull/416

> What does the community & team members think about this?

Personally, I have no strong opinion on this.

However, since we apparently created a consensus previously to not do
this, could you perhaps remind everyone what the reasoning behind that
consensus was?

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Julien Pivotto]

On 06 Dec 17:54, Bjoern Rabenstein wrote:
> On 28.11.22 11:29, Julien Pivotto wrote:
> >
> > However, I have crafted a pull request that changes that consensus and
> > makes HTTP headers configurable in the common HTTP client, with some
> > reserved headers.
>
> For findability: https://github.com/prometheus/common/pull/416
>
> > What does the community & team members think about this?
>
> Personally, I have no strong opinion on this.
>
> However, since we apparently created a consensus previously to not do
> this, could you perhaps remind everyone what the reasoning behind that
> consensus was?

https://github.com/prometheus/prometheus/issues/1724

Quoting Brian in 2016:
> The question here is how complex do we want to allow scraping protocol
> to be, and how complex a knot are we willing to let users tie themselves
> in via the core configuration? Are we okay with making it easy for a
> scrape not to be quickly testable via a browser? At some point we have
> to tell users to use a proxy server to handle the more obscure use
> cases, rather than drawing their complexity into Prometheus.
>
> As far as I'm aware the use case here relates to a custom auth solution
> with a non-recommended network setup. It's not unlikely that the next
> request in this vein would be to make these relabelable, and as this is
> an auth-related request, per discussion on #1176 we're not going to do
> that. I think we'd need a stronger use case to justify adding this
> complexity.

I do think that Brian's comments on authorization and security are still
valid, and I don't plan to add headers support to relabeling - such as I
don't plan to add relabeling for basic auth and other autorisation
methods.

--
Julien Pivotto
@roidelapluie

[Bjoern Rabenstein]

On 06.12.22 23:15, Julien Pivotto wrote:
>
> https://github.com/prometheus/prometheus/issues/1724
>
> Quoting Brian in 2016:
> > The question here is how complex do we want to allow scraping protocol
> > to be, and how complex a knot are we willing to let users tie themselves
> > in via the core configuration? Are we okay with making it easy for a
> > scrape not to be quickly testable via a browser? At some point we have
> > to tell users to use a proxy server to handle the more obscure use
> > cases, rather than drawing their complexity into Prometheus.
> >
> > As far as I'm aware the use case here relates to a custom auth solution
> > with a non-recommended network setup. It's not unlikely that the next
> > request in this vein would be to make these relabelable, and as this is
> > an auth-related request, per discussion on #1176 we're not going to do
> > that. I think we'd need a stronger use case to justify adding this
> > complexity.
>
> I do think that Brian's comments on authorization and security are still
> valid, and I don't plan to add headers support to relabeling - such as I
> don't plan to add relabeling for basic auth and other autorisation
> methods.

Thank you very much. Yes, this all makes sense. I.e. no plans for
support via relabeling, but allow users to do their special thing in
special cases via the config, even if that also opens up the
possibility to build a foot gun. (BTW, I'm a fan of clearly
documenting the dragons, so don't just add the config option, but put
a warning sign next it describing the typical pitfalls, like creating
metric endpoints that are inaccessible to browsers.)

[Matthias Rampke]

In general, what is a foot gun to me can be a rocket shoe to you, so I am in favor of providing them to those who require them, with clear labeling of the dangers.

Specifically in this case, it has become more common ("beyond corp", "zero trust") to use HTTPS over the public internet, combined with additional layers of authentication. Always requiring users to layer even more proxies on top is a drag on them and on Prometheus, so I am all in favor of this.

I had to read up on the semantics of specifying a header multiple times, is this something to call out in the documentation?

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/Y4/bTp4k4mlqSHWg%40mail.rabenste.in.