move https/ package to client_golang

[Julien Pivotto]

Dear developers,

It has crossed my mind that we are planning to move the node_exporter
https package to github.com/prometheus/common/ in the near future.

I think that for the benefit of the whole community, it might be better
to move it to our "public code" repository, github.com/prometheus/client_golang.

We know that e.g. cortex is already interesting to reuse this, and I
think that basically any Golang-based exporter would benefit from this.
A lot of them will probably depend on that code in the future.

If we do so, it would still be possible to mark it as EXPERIMENTAL at
the beginning, but we would recognize that it can be used by the whole
community.

Note: I am also volunteering to be the 'assigned maintainer' of that
code, whether it comes in the client_golang or in common.

WDYT?

--
Julien Pivotto
@roidelapluie

[Julien Pivotto]

Note: I am not challenging the moment we will move it out of
node_exporter - we will do it just as planned before.

--
Julien Pivotto
@roidelapluie

[Brian Brazil]

There's one technical problem here, client_golang currently supports back to Go 1.9. The https code as written requires Go 1.14 for simplicity, which we can get away with in common as we control all the binaries including it and common as internal code. Bjeorn as maintainer of client_golang would have to chime in on this aspect.

 

--

Brian Brazil

www.robustperception.io

[Julien Pivotto]

We could make that two different modules. Would require tags that start
with 'https': https/v0.1.0 + different go.mod. Since it is completely
different from the rest, that would be an option too. Would also allow
us to propagate security issues without updating the full client_golang.

--
Julien Pivotto
@roidelapluie

[Bjoern Rabenstein]

On 24.05.20 21:17, Julien Pivotto wrote:
>
> I think that for the benefit of the whole community, it might be better
> to move it to our "public code" repository, github.com/prometheus/client_golang.

I don't think that github.com/prometheus/client_golang is our "public
code" repository. It's the Go instrumentation client, which happens to
share its repository with the experimental and not properly maintained
HTTP API client. The latter is essentially an accident, a result of a
habit of certain core contributors back then to "just do" things
without discussing them in depth with their peers. Go Modules weren't
a thing back then, otherwise I would have resisted more strongly.

With Go Modules in the game, we should either avoid putting multiple
more or less independent libraries into the same repo, or we need to
embrace multi-module repos after all. I haven't developed an educated
opinion about the latter yet. I just noticed a number of fairly strong
statements that multi-module repos are supposed to be very painful.

Having said all that, github.com/prometheus/common is of course just
as problematic as a location. It contains many independent libraries
in the same repo. It would essentially be impossible to have
meaningful post-1.0 semver in that repo without turning it into a
multi-module repo. Plus, the common repo is (by now) explicitly marked
as an internal one.

The most straight forward solution right now is to create a new repo
"github.com/prometheus/tls" or similar. (And I would then advocate to
move the HTTP API client out of client_golang into its own repo,
should we ever see that it is steering towards a v1.0 release.)

If people feel strongly about the proliferation of repos in the
Prometheus GH org, I'm willing to educate myself about multi-module
repositories so that I can say if I will support or resist a move to a
multi-module client_golang repo (if that's possible in a non-breaking
fashion at all - if not, we should probably plan to make the common
repo multi-module).

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Julien Pivotto]

I would be in favor of creating a new repository. As I said, it is
important to be able to release that quickly and independently, and also
trying to follow go semver "for real" in this repo.

> If people feel strongly about the proliferation of repos in the
> Prometheus GH org, I'm willing to educate myself about multi-module
> repositories so that I can say if I will support or resist a move to a
> multi-module client_golang repo (if that's possible in a non-breaking
> fashion at all - if not, we should probably plan to make the common
> repo multi-module).

This is code that is going to be used and share across many
repositories, I think if we are willing to put it in a separate repo we
should do it.

> --
> Björn Rabenstein
> [PGP-ID] 0x851C3DA17D748D03
> [email] bjo...@rabenste.in
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200526191532.GQ2326%40jahnn.

--
Julien Pivotto
@roidelapluie

[Ben Kochie]

I was thinking about building an "exporter kit" repo that would include some helpful functions to reduce the amount of boilerplate needed to write exporters.

Perhaps we could star this project, have the https code moved there?

github.com/prometheus/exporter/https

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200526192156.GA1130273%40oxygen.

[Julien Pivotto]

Like

github.com/prometheus/toolkit/tls

Yes that would be an option. As an independent piece of code it could still have its own tags and go.mod, to allow smooth updates.

As it does not produce binaries that should be easy.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoh49%2BrvMpFaM8fwTEpQS%2BFyHHtuNhS_C%2B%3DKOwe7SkDUw%40mail.gmail.com.

[Brian Brazil]

On Wed, 27 May 2020 at 07:05, Ben Kochie <sup...@gmail.com> wrote:

I was thinking about building an "exporter kit" repo that would include some helpful functions to reduce the amount of boilerplate needed to write exporters.

I've thought such a thing would be useful for a long time, though my presumption was always that it would end up in client_golang as it's not too far from instrumentation.

In general I'm not a big fan of widespread proliferation of repos, particularly if it's lots of tiny repos. Even in the previous cases where we managed to get the layering largely right, it still was quite a pain in terms of overhead and release management if the repos were being actively developed. A single toolkit-y repo I could live with, I'd be concerned if we were talking repos beyond that.

Brian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABbyFmoh49%2BrvMpFaM8fwTEpQS%2BFyHHtuNhS_C%2B%3DKOwe7SkDUw%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Stuart Clark]

On 27/05/2020 07:50, Brian Brazil wrote:

On Wed, 27 May 2020 at 07:05, Ben Kochie <sup...@gmail.com> wrote:

I was thinking about building an "exporter kit" repo that would include some helpful functions to reduce the amount of boilerplate needed to write exporters.

I've thought such a thing would be useful for a long time, though my presumption was always that it would end up in client_golang as it's not too far from instrumentation.

In general I'm not a big fan of widespread proliferation of repos, particularly if it's lots of tiny repos. Even in the previous cases where we managed to get the layering largely right, it still was quite a pain in terms of overhead and release management if the repos were being actively developed. A single toolkit-y repo I could live with, I'd be concerned if we were talking repos beyond that.

How does the release management/overhead differ between several single purpose repos and a single repo containing independent things in different directories?

-- 
Stuart Clark

[Brian Brazil]

I don't think we've really had that particular situation come up yet, but I imagine there'd be similar challenges either way - likely with the multiple-repo case being a bit trickier to debug through.

 

--

Brian Brazil

www.robustperception.io

[Julien Pivotto]

We would need to adapt our Makefiles; and stop vendoring
if we go for multi repo in one.

The benefits for the users is to be able to only pull the https/package.

But now that I think about it, of we make a toolkit repo, it will
probably ALSO include the http server logic, so the exporters would NOT
need to use http directly.

At the end, I then think we then can do a single toolkit repo.

Next to the toolkit repo (in prometheus org) we could also create a
template repo [1] github.com/prometheus-community/exemple_exporter
which reuses that and that people can just "fork" [2].

[1] https://help.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-template-repository
[2] https://help.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-from-a-template

>
> --
> Brian Brazil
> www.robustperception.io

>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLrCUHOYnd3u%2B%2B8KNexFr0qdrT_0e69eeC8ZMz--r4WymQ%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[Julien Pivotto]

On 27 May 07:50, Brian Brazil wrote:
> On Wed, 27 May 2020 at 07:05, Ben Kochie <sup...@gmail.com> wrote:
>
> > I was thinking about building an "exporter kit" repo that would include
> > some helpful functions to reduce the amount of boilerplate needed to write
> > exporters.
> >
>
> I've thought such a thing would be useful for a long time, though my
> presumption was always that it would end up in client_golang as it's not
> too far from instrumentation.
>
> In general I'm not a big fan of widespread proliferation of repos,
> particularly if it's lots of tiny repos. Even in the previous cases where
> we managed to get the layering largely right, it still was quite a pain in
> terms of overhead and release management if the repos were being actively
> developed. A single toolkit-y repo I could live with, I'd be concerned if
> we were talking repos beyond that.
>
> Brian

Do we have consensus on:

- A new public repository in the Prometheus organisation
- That repository will contain go code that will be used by Prometheus,
PGW, AM and Official exporters (including but not limited to tls)
- That repository will follow go semver and be public, therefore
be reusable by unofficial exporters too

Should we name it:
- github.com/prometheus/exporter
- github.com/prometheus/toolkit

regards,

--
Julien Pivotto
@roidelapluie

[Tobias Schmidt]

 What is the scope of a prometheus/toolkit repo in contrast to the existing prometheus/common repo which already includes go pkgs used in other projects? We also already have prometheus/promu "Prometheus Utility Tool".

Would it be an option to just put the new package in prometheus/common?

regards,

--
Julien Pivotto
@roidelapluie

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200602212055.GA1260858%40oxygen.

[Julien Pivotto]

Common is internal code that is generally not intend to share or support to the outside world (with the exception of expfmt).

Plus we are speaking about more than TLS but possible also nice landing
page, config pretty print, logging/CLI options etc.

>
> regards,
> >
> > --
> > Julien Pivotto
> > @roidelapluie
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to prometheus-devel...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-developers/20200602212055.GA1260858%40oxygen
> > .
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAChBsdAevqfKLJMZXPsb%3DkrO%2Be_uEeZ9vKhyR0Z8YVF_hHTLKA%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[Tobias Schmidt]

Thanks for the info and apologies for the noise. Now I also found Björn's paragraph on this matter. I wouldn't immediately understand the scope of a toolkit repo next to the existing common and promu ones.

I'd vote to make the "exporter" part clear in the name, but then repo itself is not an actual exporter. Maybe prometheus/exporter-toolkit?

[Brian Brazil]

One thing to keep in mind is how this will look in terms of go imports, I'd presume we'd have everything in sub-packages off the repo.

Brian

 

Plus we are speaking about more than TLS but possible also nice landing
page, config pretty print, logging/CLI options etc.

>
> regards,
> >
> > --
> > Julien Pivotto
> > @roidelapluie
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to prometheus-devel...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-developers/20200602212055.GA1260858%40oxygen
> > .
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAChBsdAevqfKLJMZXPsb%3DkrO%2Be_uEeZ9vKhyR0Z8YVF_hHTLKA%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

--

Brian Brazil

www.robustperception.io

[Julien Pivotto]

https://github.com/prometheus/exporter-toolkit has been created. Let's
PR README's and so on in the coming days.


--
Julien Pivotto
@roidelapluie