Prometheus violating Go modules semantic versioning rules - what to do?

[Julius Volz]

Hi everyone,

Go modules require semantic import versioning on a Go API level (see https://github.com/golang/go/wiki/Modules#semantic-import-versioning and https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher), not just on the level of user-visible application features. The Prometheus server repo has been converted to a single Go module, but we have never followed the mandatory rule of bumping the major version of Prometheus for any breaking Go-level API change within the repo (and we probably don't want to). This means we are currently using Go modules in a broken way, which is bad for external consumers.

How do we want to deal with this situation? I see four options, none of which sounds great:

1) Keep the status quo of illegal Go modules usage. The pain of this would be carried by anyone who depends externally on packages from the main repo. E.g. the Go tooling would assume that a minor version is backward-compatible with a previous one, although it really isn't. Currently (probably for lack of a v2 subdir) it seems that the Prometheus module is being marked as "+incompatible" when used as a dependency, so its treated as if it hasn't opted into Go modules?

2) Move away from Go modules: probably not a real option, given that this is becoming a Go standard.

3) Bump the major version for each Go-level API change: Doesn't seem like a real option unless we want to switch to a completely different feel of user-visible versions. We would have Prometheus version 113.3.1 very soon then.

4) Introduce separate modules for the binary (cmd/prometheus, cmd/promtool) packages and the rest of the packages in the repo. This would be the "proper" way to handle this with Go modules, but it does mean that you now have to manage dependency versions between the binary module(s) and the rest. We always want to build the Prometheus server against the latest versions of those other packages, so for local development you either have to add local "replace" directives into go.mod to make sure the latest version from the FS is picked up, and during Prometheus releases you have to remove those directives and instead release proper versions of the non-main modules and then update to them in the main module. (The separated modules could still be part of the same repo (https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories), or they could be split into separate repos.)

I'm tending towards either 1) or 4), with 1) meaning accepting flagrantly violating laws of Go OSS citizenship, and 4) meaning a lot more work... or maybe a combination of 1) + 4) where we keep the broken state for some packages, but separate out important packages that are frequently used externally.

What do people think? Did I miss an option?

Cheers,

Julius

[Brian Brazil]

I think 1) is the way to go. I've seen 4) done in Java, where there's also tooling to support it, and it's still hard to get right.

I view it that the Prometheus repo is for building the Prometheus binary, if others wish to import our internals that's on their head.

--

Brian Brazil

www.robustperception.io

[Julius Volz]

That goes to the core of the question of how much we would want to encourage / discourage / be neutral about others using certain packages externally. So far we've been mostly in the neutral camp, breaking things as we wanted to, basically saying "you're on your own, but if it's simple enough we'll change stuff for you".

I believe we should be more encouraging of the external usage of certain packages, especially the ones that are central to OSS projects built to augment / replace parts of Prometheus, and that are hard to re-build from scratch for people. The most notable ones coming to mind here would be the TSDB (currently its own repo, but currently being moved into the main server repo) and the PromQL package. Take for example PromQL: Cortex, Thanos, Flux transpiler, certain auth gateways, etc., all use it externally, and IMO those projects provide great value to the community. So generally I'm all for being more encouraging of external usage of some of these packages, but whether that rises to the level of being worth the pain of option 4) I'm still on the fence about. Probably yes, for some packages (where to draw the line is another difficult question though).

[Tariq Ibrahim]

I agree with Brian in that the prometheus repo should be solely meant for building the prom binary and not used as an external library. However, with that being said, it would be worth finding out which sub-modules within prometheus are being used the most as an external dependency. One can look into the feasibility of extracting that code of out the main repo and maintaining it in a separate project that is compliant with go modules' SIV. For example, I know there was some discussion around migrating the prometheus' service discovery component. While I am not exactly advocating for this, it could definitely be considered if the frequency of usage justifies it.

 

[Tariq Ibrahim]

If I wasn't clear in my previous email, my vote is for approach (1).

[Brian Brazil]

On Mon, 29 Jul 2019 at 19:24, Tariq Ibrahim <tariq...@gmail.com> wrote:

I agree with Brian in that the prometheus repo should be solely meant for building the prom binary and not used as an external library. However, with that being said, it would be worth finding out which sub-modules within prometheus are being used the most as an external dependency. One can look into the feasibility of extracting that code of out the main repo and maintaining it in a separate project that is compliant with go modules' SIV. For example, I know there was some discussion around migrating the prometheus' service discovery component. While I am not exactly advocating for this, it could definitely be considered if the frequency of usage justifies it.

SD is one where the long term plan of record it to have it as a consumable library, so at some point it'll move out. That's not the case for any other parts of Prometheus. 

--

Brian Brazil

www.robustperception.io

[Sylvain Rabot]

Hi Julius,

Is there a discussion somewhere about the goal of merging the tsdb repository into the main repository ?

Shouldn't it be let as it since there is currently no major release that infrige the go modules versioning spec and move out promql & all the other popular packages used by external projects to be able to control their versioning independently ?

One goal of the go modules versioning spec is to be able to use two versions of the same lib in a single project. Wouldn't that be handy in the TSDB case for example if someone wanted to make tools to migrate tdsb instances between two incompatible versions ?

Regards.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6Yow0AYqty_xh0z%2BvfD--XcsCXi_%2BcxiSYhi%3D%3D%2BWD9KP11w%40mail.gmail.com.

--

Sylvain Rabot <syl...@abstraction.fr>

[Julius Volz]

On Mon, Jul 29, 2019 at 8:40 PM Sylvain Rabot <syl...@abstraction.fr> wrote:

Hi Julius,

Is there a discussion somewhere about the goal of merging the tsdb repository into the main repository ?

Shouldn't it be let as it since there is currently no major release that infrige the go modules versioning spec and move out promql & all the other popular packages used by external projects to be able to control their versioning independently ?

One goal of the go modules versioning spec is to be able to use two versions of the same lib in a single project. Wouldn't that be handy in the TSDB case for example if someone wanted to make tools to migrate tdsb instances between two incompatible versions ?

The move of the "tsdb" repo into the main server repo (pending in this PR: https://github.com/prometheus/prometheus/pull/5805) was discussed and decided at the Prometheus DevSummit after KubeCon Barcelona. The reasons were that managing components in separate repos with separate versioning etc., is a pain when the Prometheus server always just wants to use the very latest commit version of them all the time (currently you have to do releases of TSDB for each Prometheus release and remember to update the version in Prometheus's go.mod). But I think nobody at the time of the discussion really brought up the downsides mentioned in this thread about Go modules, so I wonder if that should inform a reconsideration of the decision.

Another downside of having all these externally-used packages in a single big module is that anyone depending on them will inherit / download a bunch of dependencies (like k8s) from our go.mod, even if the specific packages in our module that they are using don't use those dependencies at all. And then if we e.g. define a newer version of k8s than those external users do, they will have to use a "replace" directive in their go.mod to ensure that they use the version they intended to use (and yeah, k8s also currently is violating Go module semver).

[Julius Volz]

Maybe it *should* be the case for other parts more than SD even. PromQL would even seem like a more obvious candidate for an externally well-consumable library than SD, given that it's already being used externally by several well-known external projects. 

[Tariq Ibrahim]

> Maybe it *should* be the case for other parts more than SD even. PromQL would even seem like a more obvious candidate for an externally well-consumable library than SD, given that it's already being used externally by several well-known external projects. 

I definitely agree that PromQL is a very compelling candidate here, I only mentioned SD as an example. Extracting that out and maintaining it separately in the "go modules" way sounds like a great plan moving forward. Moreover, it makes more sense IMO to move promQL out of the repo as far as clear separation of concerns goes. Come to think of it, I wonder why it wasn't done earlier.

[Julius Volz]

On Mon, Jul 29, 2019 at 9:04 PM Tariq Ibrahim <tariq...@gmail.com> wrote:

> Maybe it *should* be the case for other parts more than SD even. PromQL would even seem like a more obvious candidate for an externally well-consumable library than SD, given that it's already being used externally by several well-known external projects. 

I definitely agree that PromQL is a very compelling candidate here, I only mentioned SD as an example. Extracting that out and maintaining it separately in the "go modules" way sounds like a great plan moving forward. Moreover, it makes more sense IMO to move promQL out of the repo as far as clear separation of concerns goes. Come to think of it, I wonder why it wasn't done earlier.

Well, the obvious downside is what I mentioned about the TSDB being separate: having to do multiple releases, updating dependency versions, etc., for each of these separated-out components. So there's definitely a cost associated with that as well that has to be worth it. 

[Sylvain Rabot]

On Mon, 29 Jul 2019 at 20:55, Julius Volz <juliu...@gmail.com> wrote:

On Mon, Jul 29, 2019 at 8:40 PM Sylvain Rabot <syl...@abstraction.fr> wrote:

Hi Julius,

Is there a discussion somewhere about the goal of merging the tsdb repository into the main repository ?

Shouldn't it be let as it since there is currently no major release that infrige the go modules versioning spec and move out promql & all the other popular packages used by external projects to be able to control their versioning independently ?

One goal of the go modules versioning spec is to be able to use two versions of the same lib in a single project. Wouldn't that be handy in the TSDB case for example if someone wanted to make tools to migrate tdsb instances between two incompatible versions ?

The move of the "tsdb" repo into the main server repo (pending in this PR: https://github.com/prometheus/prometheus/pull/5805) was discussed and decided at the Prometheus DevSummit after KubeCon Barcelona. The reasons were that managing components in separate repos with separate versioning etc., is a pain when the Prometheus server always just wants to use the very latest commit version of them all the time (currently you have to do releases of TSDB for each Prometheus release and remember to update the version in Prometheus's go.mod). But I think nobody at the time of the discussion really brought up the downsides mentioned in this thread about Go modules, so I wonder if that should inform a reconsideration of the decision.

That's nothing a bit of scripting launched by the ci could not solve. We could have scripts executed by Travis/Circle-ci when PRs are merged into master (or when tags are created) in the externalized packages that would create PRs to update go.mod and vendor/ in the main repo. That would take away the burden of keeping the main repo in sync with its external dependencies.

Another downside of having all these externally-used packages in a single big module is that anyone depending on them will inherit / download a bunch of dependencies (like k8s) from our go.mod, even if the specific packages in our module that they are using don't use those dependencies at all. And then if we e.g. define a newer version of k8s than those external users do, they will have to use a "replace" directive in their go.mod to ensure that they use the version they intended to use (and yeah, k8s also currently is violating Go module semver).

That's right.

If the main repo is only to build binaries as Brian suggested then I believe there are more benefits than downsides in keeping/moving out the main repository popular packages to ease the life of the community that rely on those packages.

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

[Sylvain Rabot]

On Mon, 29 Jul 2019 at 21:19, Sylvain Rabot <syl...@abstraction.fr> wrote:

On Mon, 29 Jul 2019 at 20:55, Julius Volz <juliu...@gmail.com> wrote:

On Mon, Jul 29, 2019 at 8:40 PM Sylvain Rabot <syl...@abstraction.fr> wrote:

Hi Julius,

Is there a discussion somewhere about the goal of merging the tsdb repository into the main repository ?

Shouldn't it be let as it since there is currently no major release that infrige the go modules versioning spec and move out promql & all the other popular packages used by external projects to be able to control their versioning independently ?

One goal of the go modules versioning spec is to be able to use two versions of the same lib in a single project. Wouldn't that be handy in the TSDB case for example if someone wanted to make tools to migrate tdsb instances between two incompatible versions ?

The move of the "tsdb" repo into the main server repo (pending in this PR: https://github.com/prometheus/prometheus/pull/5805) was discussed and decided at the Prometheus DevSummit after KubeCon Barcelona. The reasons were that managing components in separate repos with separate versioning etc., is a pain when the Prometheus server always just wants to use the very latest commit version of them all the time (currently you have to do releases of TSDB for each Prometheus release and remember to update the version in Prometheus's go.mod). But I think nobody at the time of the discussion really brought up the downsides mentioned in this thread about Go modules, so I wonder if that should inform a reconsideration of the decision.

That's nothing a bit of scripting launched by the ci could not solve. We could have scripts executed by Travis/Circle-ci when PRs are merged into master (or when tags are created) in the externalized packages that would create PRs to update go.mod and vendor/ in the main repo. That would take away the burden of keeping the main repo in sync with its external dependencies.

I suggested so time ago to use dependabot (now acquired by github) in that regard: https://github.com/prometheus/prometheus/issues/5265

It does not do vendoring at the time but I made a script for one of my side project that does.

Another downside of having all these externally-used packages in a single big module is that anyone depending on them will inherit / download a bunch of dependencies (like k8s) from our go.mod, even if the specific packages in our module that they are using don't use those dependencies at all. And then if we e.g. define a newer version of k8s than those external users do, they will have to use a "replace" directive in their go.mod to ensure that they use the version they intended to use (and yeah, k8s also currently is violating Go module semver).

That's right.

If the main repo is only to build binaries as Brian suggested then I believe there are more benefits than downsides in keeping/moving out the main repository popular packages to ease the life of the community that rely on those packages.

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

--

Sylvain Rabot <syl...@abstraction.fr>

[Calle Pettersson]

Could the idea be reversed, so that the things that could be modules are (automatically) mirrored out into separate repos for those who wish to consume them, but Prometheus itself uses them from its own repo? The goal would be to ensure the Prometheus dev experience is as easy as possible, while allowing somewhat clean modules for external users which do not get all the dependencies of Prometheus itself.

Some care would still be needed to ensure these modules are still standalone when extracted though, not sure if that is likely to bite us somewhere down the line...

Calle

[Julius Volz]

On Mon, Jul 29, 2019 at 9:20 PM Sylvain Rabot <syl...@abstraction.fr> wrote:

On Mon, 29 Jul 2019 at 20:55, Julius Volz <juliu...@gmail.com> wrote:

On Mon, Jul 29, 2019 at 8:40 PM Sylvain Rabot <syl...@abstraction.fr> wrote:

Hi Julius,

Is there a discussion somewhere about the goal of merging the tsdb repository into the main repository ?

Shouldn't it be let as it since there is currently no major release that infrige the go modules versioning spec and move out promql & all the other popular packages used by external projects to be able to control their versioning independently ?

One goal of the go modules versioning spec is to be able to use two versions of the same lib in a single project. Wouldn't that be handy in the TSDB case for example if someone wanted to make tools to migrate tdsb instances between two incompatible versions ?

The move of the "tsdb" repo into the main server repo (pending in this PR: https://github.com/prometheus/prometheus/pull/5805) was discussed and decided at the Prometheus DevSummit after KubeCon Barcelona. The reasons were that managing components in separate repos with separate versioning etc., is a pain when the Prometheus server always just wants to use the very latest commit version of them all the time (currently you have to do releases of TSDB for each Prometheus release and remember to update the version in Prometheus's go.mod). But I think nobody at the time of the discussion really brought up the downsides mentioned in this thread about Go modules, so I wonder if that should inform a reconsideration of the decision.

That's nothing a bit of scripting launched by the ci could not solve. We could have scripts executed by Travis/Circle-ci when PRs are merged into master (or when tags are created) in the externalized packages that would create PRs to update go.mod and vendor/ in the main repo. That would take away the burden of keeping the main repo in sync with its external dependencies.

If we decide to go this route, then certainly some automation is a good idea, but I wouldn't discount the remaining extra complexity and mental overhead in any case. It's one more thing to know and understand (even if all you have to know is that you don't need to worry about it).

[Julius Volz]

Interesting, maybe. One might also run into the fact that then it's not 100% the same types as in the main repo, which might cause type incompatibilities with other libs... but maybe not, if everyone external uses the split-out library versions. (One thought would also be to use type aliases to make the types identical, but then you'd have to depend back on the Prometheus repo...)

[Calle Pettersson]

Indeed. Perhaps migrate the in-Prometheus ones to internal/ to encourage moving over to the externalized repos (in case my idea is even practical, I haven't tested)? I think the end goal of this approach should be that noone imports prometheus/prometheus directly, and it probably shouldn't be possible.

Calle

[Matthias Rampke]

I am intrigued by the "replace" option for modules. My understanding is that the versioning for the binary (cmd/prometheus) doesn't really affect anyone outside of our own builds. Could we split the modules, but use replace directives within the repo to force cmd/prometheus to always pull in the latest of these libraries? That would mean different parts of the repo have different "versions" but it would not matter for building Prometheus itself, and by pulling one repository you'd still get the consistent state.

/MR

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/21a1a8fc-e34b-4ce9-a5bb-8f19f814e3a5%40www.fastmail.com.

[Julius Volz]

On Tue, Jul 30, 2019 at 10:04 AM Matthias Rampke <m...@soundcloud.com> wrote:

I am intrigued by the "replace" option for modules. My understanding is that the versioning for the binary (cmd/prometheus) doesn't really affect anyone outside of our own builds. Could we split the modules, but use replace directives within the repo to force cmd/prometheus to always pull in the latest of these libraries? That would mean different parts of the repo have different "versions" but it would not matter for building Prometheus itself, and by pulling one repository you'd still get the consistent state.

Hmm, you know what, that actually sounds like an interesting idea. I hadn't fully thought of this option because I had only used "replace" between separate repos, not with multiple modules in a single repo.

I was worried that this would break "go get" of the cmd packages, but in a demo setup (https://github.com/juliusv/moduletest, "replace" at https://github.com/juliusv/moduletest/blob/master/cmd/hello/go.mod#L7) it actually seems to work just fine:

    go get go get github.com/juliusv/moduletest/cmd/hello

    go run github.com/juliusv/moduletest/cmd/hello

If we did this, then we'd have to write "replace" directives once, but otherwise not care about updating versions for repo-local dependencies. But everyone else could consume the non-cmd packages in a better way.

I wonder if anyone sees problems with this that we're overlooking?

[Julius Volz]

On Tue, Jul 30, 2019 at 12:22 PM Julius Volz <juliu...@gmail.com> wrote:

On Tue, Jul 30, 2019 at 10:04 AM Matthias Rampke <m...@soundcloud.com> wrote:

I am intrigued by the "replace" option for modules. My understanding is that the versioning for the binary (cmd/prometheus) doesn't really affect anyone outside of our own builds. Could we split the modules, but use replace directives within the repo to force cmd/prometheus to always pull in the latest of these libraries? That would mean different parts of the repo have different "versions" but it would not matter for building Prometheus itself, and by pulling one repository you'd still get the consistent state.

Hmm, you know what, that actually sounds like an interesting idea. I hadn't fully thought of this option

*"hadn't" I meant :)

[Brian Brazil]

On Tue, 30 Jul 2019 at 11:22, Julius Volz <juliu...@gmail.com> wrote:

On Tue, Jul 30, 2019 at 10:04 AM Matthias Rampke <m...@soundcloud.com> wrote:

I am intrigued by the "replace" option for modules. My understanding is that the versioning for the binary (cmd/prometheus) doesn't really affect anyone outside of our own builds. Could we split the modules, but use replace directives within the repo to force cmd/prometheus to always pull in the latest of these libraries? That would mean different parts of the repo have different "versions" but it would not matter for building Prometheus itself, and by pulling one repository you'd still get the consistent state.

Hmm, you know what, that actually sounds like an interesting idea. I hadn't fully thought of this option because I had only used "replace" between separate repos, not with multiple modules in a single repo.

I was worried that this would break "go get" of the cmd packages, but in a demo setup (https://github.com/juliusv/moduletest, "replace" at https://github.com/juliusv/moduletest/blob/master/cmd/hello/go.mod#L7) it actually seems to work just fine:

    go get go get github.com/juliusv/moduletest/cmd/hello

    go run github.com/juliusv/moduletest/cmd/hello

If we did this, then we'd have to write "replace" directives once, but otherwise not care about updating versions for repo-local dependencies. But everyone else could consume the non-cmd packages in a better way.

How would that work when we change a dependency? Having to update 6-7 of them in the right order is likely to be quite error prone.

Brian

 

I wonder if anyone sees problems with this that we're overlooking?

/MR

On Mon, Jul 29, 2019 at 8:45 PM Calle Pettersson <ca...@cape.nu> wrote:

On Mon, 29 Jul 2019, at 22:39, Julius Volz wrote:

On Mon, Jul 29, 2019 at 9:45 PM Calle Pettersson <ca...@cape.nu> wrote:

On Mon, 29 Jul 2019, at 21:06, Julius Volz wrote:

On Mon, Jul 29, 2019 at 9:04 PM Tariq Ibrahim <tariq...@gmail.com> wrote:

> Maybe it *should* be the case for other parts more than SD even. PromQL would even seem like a more obvious candidate for an externally well-consumable library than SD, given that it's already being used externally by several well-known external projects. 

I definitely agree that PromQL is a very compelling candidate here, I only mentioned SD as an example. Extracting that out and maintaining it separately in the "go modules" way sounds like a great plan moving forward. Moreover, it makes more sense IMO to move promQL out of the repo as far as clear separation of concerns goes. Come to think of it, I wonder why it wasn't done earlier.

Well, the obvious downside is what I mentioned about the TSDB being separate: having to do multiple releases, updating dependency versions, etc., for each of these separated-out components. So there's definitely a cost associated with that as well that has to be worth it. 

Could the idea be reversed, so that the things that could be modules are (automatically) mirrored out into separate repos for those who wish to consume them, but Prometheus itself uses them from its own repo? The goal would be to ensure the Prometheus dev experience is as easy as possible, while allowing somewhat clean modules for external users which do not get all the dependencies of Prometheus itself.

Some care would still be needed to ensure these modules are still standalone when extracted though, not sure if that is likely to bite us somewhere down the line...

Interesting, maybe. One might also run into the fact that then it's not 100% the same types as in the main repo, which might cause type incompatibilities with other libs... but maybe not, if everyone external uses the split-out library versions. (One thought would also be to use type aliases to make the types identical, but then you'd have to depend back on the Prometheus repo...)

Indeed. Perhaps migrate the in-Prometheus ones to internal/ to encourage moving over to the externalized repos (in case my idea is even practical, I haven't tested)? I think the end goal of this approach should be that noone imports prometheus/prometheus directly, and it probably shouldn't be possible.

Calle

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/21a1a8fc-e34b-4ce9-a5bb-8f19f814e3a5%40www.fastmail.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6Yoyx%3Dd1tjdO%2Bawat%3DG44Dvp%3DV4BRMLrU2pvsvRAKYOngyQ%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Julius Volz]

On Tue, Jul 30, 2019 at 3:13 PM Brian Brazil <brian....@robustperception.io> wrote:

On Tue, 30 Jul 2019 at 11:22, Julius Volz <juliu...@gmail.com> wrote:

On Tue, Jul 30, 2019 at 10:04 AM Matthias Rampke <m...@soundcloud.com> wrote:

I am intrigued by the "replace" option for modules. My understanding is that the versioning for the binary (cmd/prometheus) doesn't really affect anyone outside of our own builds. Could we split the modules, but use replace directives within the repo to force cmd/prometheus to always pull in the latest of these libraries? That would mean different parts of the repo have different "versions" but it would not matter for building Prometheus itself, and by pulling one repository you'd still get the consistent state.

Hmm, you know what, that actually sounds like an interesting idea. I hadn't fully thought of this option because I had only used "replace" between separate repos, not with multiple modules in a single repo.

I was worried that this would break "go get" of the cmd packages, but in a demo setup (https://github.com/juliusv/moduletest, "replace" at https://github.com/juliusv/moduletest/blob/master/cmd/hello/go.mod#L7) it actually seems to work just fine:

    go get go get github.com/juliusv/moduletest/cmd/hello

    go run github.com/juliusv/moduletest/cmd/hello

If we did this, then we'd have to write "replace" directives once, but otherwise not care about updating versions for repo-local dependencies. But everyone else could consume the non-cmd packages in a better way.

How would that work when we change a dependency? Having to update 6-7 of them in the right order is likely to be quite error prone.

Do you mean for non-cmd modules in the repo that depend on each other?

I see two options:

- Split the repo into only 2 modules, cmd and non-cmd. Then the non-cmd module packages don't have to worry about updating versions among each other (and the "cmd" one doesn't either, due to "replace"). Solves the problem of being able to have Go-style semantic versioning in the non-cmd module, but doesn't solve the problem of external users depending on too large of a module (and then getting e.g. our k8s dep version prescribed even though they only include "promql").

- Split the repo into cmd and multiple non-cmd modules. Yeah, then we have more inter-module version updating pain, but it'd be nicer for external consumers.

[Matthias Rampke]

Does `replace` work on transitive dependencies?

[Julius Volz]

Yes, "replace" only acts on the current main module you're building and are ignored in dependencies. They hard-override anything that's specified in go.mod files of modules you depend on.

[Matthias Rampke]

So the recursive-update would only affect the library modules among each other? What would the modules be that we would want to be reusable, and do they depend on each other?

/MR

[Brian Brazil]

On Tue, 30 Jul 2019 at 15:37, Matthias Rampke <m...@soundcloud.com> wrote:

So the recursive-update would only affect the library modules among each other? What would the modules be that we would want to be reusable, and do they depend on each other?

There are external groups using basically everything, and they do depend on each other.

I would prefer solutions that have any burden be placed not on us.

Brian

--

Brian Brazil

www.robustperception.io

[Simon Pasquier]

On Tue, Jul 30, 2019 at 12:22 PM Julius Volz <juliu...@gmail.com> wrote:
>
> On Tue, Jul 30, 2019 at 10:04 AM Matthias Rampke <m...@soundcloud.com> wrote:
>>
>> I am intrigued by the "replace" option for modules. My understanding is that the versioning for the binary (cmd/prometheus) doesn't really affect anyone outside of our own builds. Could we split the modules, but use replace directives within the repo to force cmd/prometheus to always pull in the latest of these libraries? That would mean different parts of the repo have different "versions" but it would not matter for building Prometheus itself, and by pulling one repository you'd still get the consistent state.
>
>
> Hmm, you know what, that actually sounds like an interesting idea. I hadn't fully thought of this option because I had only used "replace" between separate repos, not with multiple modules in a single repo.
>
> I was worried that this would break "go get" of the cmd packages, but in a demo setup (https://github.com/juliusv/moduletest, "replace" at https://github.com/juliusv/moduletest/blob/master/cmd/hello/go.mod#L7) it actually seems to work just fine:
>
> go get go get github.com/juliusv/moduletest/cmd/hello
> go run github.com/juliusv/moduletest/cmd/hello
>
> If we did this, then we'd have to write "replace" directives once, but otherwise not care about updating versions for repo-local dependencies. But everyone else could consume the non-cmd packages in a better way.

IIUC this is similar to what github.com/hashicorp/consul is doing
(though not exactly the same case). github.com/hashicorp/consul and
github.com/hashicorp/consul/api each have a go.mod file. The former
references the latter [1] but it also has a replace directive to make
sure that it is always built with the local bits. API consumers have
only a reference to github.com/hashicorp/consul/api [2] which has a
versioning distinct from the main Consul module.

[1] https://github.com/hashicorp/consul/blob/a1725e6b5299c6ce12e8273205f90fba31403686/go.mod#L5
[2] https://github.com/prometheus/consul_exporter/blob/6a90bd50bd5233d81f355807fba87d9ef4edabf4/go.mod#L5

By the way, if we manage to split
github.com/prometheus/prometheus/discovery from the rest (which we
already agree would be good to externalize), we would get most of our
dependencies out of the main go.mod file. Even though not satisfying
(eg the module path wouldn't have the v2 suffix), it might be good
enough for people consuming anything other than service discovery.

>
> I wonder if anyone sees problems with this that we're overlooking?
>
>> /MR
>>
>> On Mon, Jul 29, 2019 at 8:45 PM Calle Pettersson <ca...@cape.nu> wrote:
>>>
>>>
>>> On Mon, 29 Jul 2019, at 22:39, Julius Volz wrote:
>>>
>>> On Mon, Jul 29, 2019 at 9:45 PM Calle Pettersson <ca...@cape.nu> wrote:
>>>
>>>
>>> On Mon, 29 Jul 2019, at 21:06, Julius Volz wrote:
>>>
>>> On Mon, Jul 29, 2019 at 9:04 PM Tariq Ibrahim <tariq...@gmail.com> wrote:
>>>
>>> > Maybe it *should* be the case for other parts more than SD even. PromQL would even seem like a more obvious candidate for an externally well-consumable library than SD, given that it's already being used externally by several well-known external projects.
>>>
>>> I definitely agree that PromQL is a very compelling candidate here, I only mentioned SD as an example. Extracting that out and maintaining it separately in the "go modules" way sounds like a great plan moving forward. Moreover, it makes more sense IMO to move promQL out of the repo as far as clear separation of concerns goes. Come to think of it, I wonder why it wasn't done earlier.
>>>
>>>
>>> Well, the obvious downside is what I mentioned about the TSDB being separate: having to do multiple releases, updating dependency versions, etc., for each of these separated-out components. So there's definitely a cost associated with that as well that has to be worth it.
>>>
>>>
>>> Could the idea be reversed, so that the things that could be modules are (automatically) mirrored out into separate repos for those who wish to consume them, but Prometheus itself uses them from its own repo? The goal would be to ensure the Prometheus dev experience is as easy as possible, while allowing somewhat clean modules for external users which do not get all the dependencies of Prometheus itself.
>>> Some care would still be needed to ensure these modules are still standalone when extracted though, not sure if that is likely to bite us somewhere down the line...
>>>
>>>
>>> Interesting, maybe. One might also run into the fact that then it's not 100% the same types as in the main repo, which might cause type incompatibilities with other libs... but maybe not, if everyone external uses the split-out library versions. (One thought would also be to use type aliases to make the types identical, but then you'd have to depend back on the Prometheus repo...)
>>>
>>> Indeed. Perhaps migrate the in-Prometheus ones to internal/ to encourage moving over to the externalized repos (in case my idea is even practical, I haven't tested)? I think the end goal of this approach should be that noone imports prometheus/prometheus directly, and it probably shouldn't be possible.
>>>
>>> Calle
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/21a1a8fc-e34b-4ce9-a5bb-8f19f814e3a5%40www.fastmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6Yoyx%3Dd1tjdO%2Bawat%3DG44Dvp%3DV4BRMLrU2pvsvRAKYOngyQ%40mail.gmail.com.

[Bjoern Rabenstein]

On 29.07.19 18:53, Julius Volz wrote:
>
> 3) Bump the major version for each Go-level API change: Doesn't seem like a
> real option unless we want to switch to a completely different feel of
> user-visible versions. We would have Prometheus version 113.3.1 very soon then.

BTW, nothing forces us to use the Go module version for the released
binary. We could use another tag convention for released binaries
(e.g. `release-p.q.r`) and leave `vx.y.z` for Go modules. I'm not
saying I'm for it (I find a lot of the suggestions so far more
intriguing), but we should keep in mind that coupling release versions
to Go module version is not inevitable.

In different news, I think another problem is to use one and the same
version tagging for a huge selection of packages, as we have it in the
prometheus/prometheus repo. It's called _modules_ for a reason. If you
have a monolithic codebase of many many packages, that's not really a
module. Having too many things in such a mega-module will lead to a
lot of major version bumps (each of which will seem irrelevant for
most of the users: one package has an incompatible change, everybody
sees a major version bump, not only those using that one package). And
it will create a lot of dependencies that everybody has to pull in.

As a result, I don't think it's just the versioning that's a
problem. Too fully embrace and leverage Go modules, you have to
actually modularize your code base. Which leads us to the fundamental
difference between a monolithic and a modularized code base. A
monolithic code base makes code-sharing easy (but also somewhat dirty)
and makes tight coupling between different parts of the code base
feasible (most prominently visible when you atomically change
interdependent parts of the code base). (Insert a lot of more good
things said about monorepos here.) The benefits have driven us towards
moving TSDB (back) into prometheus/prometheus (which I think is not
only motivated by relase chore work but also by the coupling between
Prometheus and TSDB that is tighter than we originally assumed or
aspired). They have also driven some of us towards disliking the
prometheus/common repo. The latter is a way to share code between
modularized code bases. Ironically, prometheus/common itself would not
work well as _one_ module but had to be many modules to make sense.

A modularized code base requires discipline of decoupling different
parts of the code base. I think that's the real reason why
prometheus/common and prometheus/tsdb create friction. You could see
it as a burden, or you can see it as a benefit. It will result in a
cleaner and better maintainable code base. However, I believe the most
relevant reason for using a modularized code base is that it enables
reusability in an open-source world that is not (and never will be) a
monorepo. Yes, doing so will put a burden on us that mostly external
users of the code will benefit from. But the other extreme, namely
putting all Go code that is not in prometheus/client_golang into one
Prometheus monorepo, would make it super hard for others to reuse our
code at all. We could claim, of course, that that's exactly what we
want. However, then we can hardly praise the whole wide and open
Prometheus ecosystem with all the integrations and collaborating
projects. And in particular we couldn't really blame projects like
Victoriametrics that they implement everything on their own.

Bottom line: I think we should try harder to make our code reusable
and sharable. Last time I really thought about it was a time when
there was no gigantic Prometheus ecosystem, so back then it seemed
fair to me to refuse any burden on us. We had to get our stuff
working, and making it easily accessible for others was not a
priority. I think that has changed now, with TSDB, PromQL, SD, even
Alertmanager (which gets embedded into other projects) being prime
examples for shared functionality. From that perspective, the problems
at hand with Go modules are more symptoms of our too-monolithic code
base than problems of the Go modules approach itself. This all makes
me think we should probably embrace Go modules rather than work around
it, for the good of the project and ecosystem. It doesn't matter so
much if it happens with more repositories or with multiple modules per
repository. Various approaches to make any approach feasible have
already been mentionted in the discussion so far.

In different news, I really like this discussion thread. A lot of good
ideas and well-reasoned opinions here. Just by reading it, I had many
eye-opening moments.

Cheers,
--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Julius Volz]

On Tue, Jul 30, 2019 at 5:28 PM Simon Pasquier <spas...@redhat.com> wrote:

On Tue, Jul 30, 2019 at 12:22 PM Julius Volz <juliu...@gmail.com> wrote:
>
> On Tue, Jul 30, 2019 at 10:04 AM Matthias Rampke <m...@soundcloud.com> wrote:
>>
>> I am intrigued by the "replace" option for modules. My understanding is that the versioning for the binary (cmd/prometheus) doesn't really affect anyone outside of our own builds. Could we split the modules, but use replace directives within the repo to force cmd/prometheus to always pull in the latest of these libraries? That would mean different parts of the repo have different "versions" but it would not matter for building Prometheus itself, and by pulling one repository you'd still get the consistent state.
>
>
> Hmm, you know what, that actually sounds like an interesting idea. I hadn't fully thought of this option because I had only used "replace" between separate repos, not with multiple modules in a single repo.
>
> I was worried that this would break "go get" of the cmd packages, but in a demo setup (https://github.com/juliusv/moduletest, "replace" at https://github.com/juliusv/moduletest/blob/master/cmd/hello/go.mod#L7) it actually seems to work just fine:
>
>     go get go get github.com/juliusv/moduletest/cmd/hello
>     go run github.com/juliusv/moduletest/cmd/hello
>
> If we did this, then we'd have to write "replace" directives once, but otherwise not care about updating versions for repo-local dependencies. But everyone else could consume the non-cmd packages in a better way.

IIUC this is similar to what github.com/hashicorp/consul is doing
(though not exactly the same case). github.com/hashicorp/consul and
github.com/hashicorp/consul/api each have a go.mod file. The former
references the latter [1] but it also has a replace directive to make
sure that it is always built with the local bits. API consumers have
only a reference to github.com/hashicorp/consul/api [2] which has a
versioning distinct from the main Consul module.

Nice, thanks for the real-life example of that! I like it :)

 

[1] https://github.com/hashicorp/consul/blob/a1725e6b5299c6ce12e8273205f90fba31403686/go.mod#L5
[2] https://github.com/prometheus/consul_exporter/blob/6a90bd50bd5233d81f355807fba87d9ef4edabf4/go.mod#L5

By the way, if we manage to split
github.com/prometheus/prometheus/discovery from the rest (which we
already agree would be good to externalize), we would get most of our
dependencies out of the main go.mod file. Even though not satisfying
(eg the module path wouldn't have the v2 suffix), it might be good
enough for people consuming anything other than service discovery.

That is a good point, maybe a good way to start. Regarding v2 suffix: You don't necessarily have to have the v2 suffix in the directory path, you can just have it in your go.mod instead (see https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher). And the versions for SD would be separate/new anyway.

[Julius Volz]

On Tue, Jul 30, 2019 at 8:02 PM Bjoern Rabenstein <bjo...@rabenste.in> wrote:

On 29.07.19 18:53, Julius Volz wrote:
>
> 3) Bump the major version for each Go-level API change: Doesn't seem like a
> real option unless we want to switch to a completely different feel of
> user-visible versions. We would have Prometheus version 113.3.1 very soon then.

BTW, nothing forces us to use the Go module version for the released
binary. We could use another tag convention for released binaries
(e.g. `release-p.q.r`) and leave `vx.y.z` for Go modules. I'm not
saying I'm for it (I find a lot of the suggestions so far more
intriguing), but we should keep in mind that coupling release versions
to Go module version is not inevitable.

Yep, we could do that, but IMO we should keep the current versioning scheme as it's currently being used. I bet a whole bunch of users depend on that in their scripts and all.

IIUC as long as any modules we want to split out are housed in subdirectories, their version tags will contain the subdirectory anyway (like "discovery/v1.2.3"), so they wouldn't clash with our current binary versioning (see https://github.com/golang/go/wiki/Modules#what-are-multi-module-repositories).

 

In different news, I think another problem is to use one and the same
version tagging for a huge selection of packages, as we have it in the
prometheus/prometheus repo. It's called _modules_ for a reason. If you
have a monolithic codebase of many many packages, that's not really a
module. Having too many things in such a mega-module will lead to a
lot of major version bumps (each of which will seem irrelevant for
most of the users: one package has an incompatible change, everybody
sees a major version bump, not only those using that one package). And
it will create a lot of dependencies that everybody has to pull in.

Agreed (though there's of course the stated reasons to bunch things together too - tradeoffs).

Good points! Yes, I think what's fundamentally changing these days is that most of the *new* value in the Prometheus ecosystem is being created by high-impact projects outside of the main Prometheus server (Thanos, Cortex, etc.). And a lot of those projects depend on our "internal" packages that are a pain to consume. So I think it makes sense for the project to reconsider what maintenance burden is reasonable for this. IMO it would be great value for the Prometheus ecosystem to make our packages better consumable like you mentioned.

 

In different news, I really like this discussion thread. A lot of good
ideas and well-reasoned opinions here. Just by reading it, I had many
eye-opening moments.

Same! 

[Julius Volz]

On Wed, Jul 31, 2019 at 12:36 PM Julius Volz <juliu...@gmail.com> wrote:

On Tue, Jul 30, 2019 at 5:28 PM Simon Pasquier <spas...@redhat.com> wrote:

By the way, if we manage to split
github.com/prometheus/prometheus/discovery from the rest (which we
already agree would be good to externalize), we would get most of our
dependencies out of the main go.mod file. Even though not satisfying
(eg the module path wouldn't have the v2 suffix), it might be good
enough for people consuming anything other than service discovery.

That is a good point, maybe a good way to start.

Actually the main module would still inherit all the dependencies of what it depends on, so this wouldn't help with the main module. But it could help if we also have separate modules for some of the other packages which can then really only depend on what they need (and those can be used externally without going through the main module). 

[Martín Ferrari]

On 29/07/2019 17:53, Julius Volz wrote:
> Go modules require semantic import versioning on a Go API level (see
> have never followed the mandatory rule of bumping the major version of
> Prometheus for any breaking Go-level API change within the repo (and we
> probably don't want to). This means we are currently using Go modules in
> a broken way, which is bad for external consumers.

> 3) Bump the major version for each Go-level API change: Doesn't seem like a real option unless we want to switch to a completely different feel of user-visible versions. We would have Prometheus version 113.3.1 very soon then.

I don't know much about Go modules, but this surprises me.. Are there
that many breaking API changes for this to be a problem? And if so,
wouldn't it be a good thing to try to have a more stable API?

--
Martín Ferrari (Tincho)

[Brian Brazil]

There's a fair few, and these are all APIs internal to Prometheus. New features often require new APIs.

--

Brian Brazil

www.robustperception.io

[Julius Volz]

One more observation:

I did a local experiment creating a separate module for the "promql" package and ran into one more maintenance burden: we "go mod vendor" everything into the "vendor" directory and then build using "go build -mod=vendor" (using only the local vendor copy for all dependency resolution), so before I was able to build Prometheus, I also had to run "go mod vendor" again to make sure that the "promql" package was vendored. I guess this would have to be done for every change to a local module.

[Matthias Rampke]

And there's no way to not vendor local modules?

[Julius Volz]

On Thu, Aug 1, 2019 at 2:49 PM Matthias Rampke <m...@soundcloud.com> wrote:

And there's no way to not vendor local modules?

It doesn't seem like it, both on the vendoring side and the build side, it looks like it's an all-or-nothing approach.

[Krasimir Georgiev]

Why not post an issue in the go repo? Usually they are quite helpful with suggestions.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxnaNCoeMAyQddzqWcigHDuSpf9_XKGhJx%2BbfWZi6BqWQ%40mail.gmail.com.

[Simon Pasquier]

On Thu, Aug 1, 2019 at 2:52 PM Julius Volz <juliu...@gmail.com> wrote:
>
> On Thu, Aug 1, 2019 at 2:49 PM Matthias Rampke <m...@soundcloud.com> wrote:
>>
>> And there's no way to not vendor local modules?
>
>
> It doesn't seem like it, both on the vendoring side and the build side, it looks like it's an all-or-nothing approach.

Looking again at the consul example, I noticed that they explicitly
remove the "sub-module" parts from the vendor/ directory:
https://github.com/hashicorp/consul/blob/a1725e6b5299c6ce12e8273205f90fba31403686/GNUmakefile#L192-L198

[Brian Brazil]

On Thu, 1 Aug 2019 at 15:02, Simon Pasquier <spas...@redhat.com> wrote:

On Thu, Aug 1, 2019 at 2:52 PM Julius Volz <juliu...@gmail.com> wrote:
>
> On Thu, Aug 1, 2019 at 2:49 PM Matthias Rampke <m...@soundcloud.com> wrote:
>>
>> And there's no way to not vendor local modules?
>
>
> It doesn't seem like it, both on the vendoring side and the build side, it looks like it's an all-or-nothing approach.

Looking again at the consul example, I noticed that they explicitly
remove the "sub-module" parts from the vendor/ directory:
https://github.com/hashicorp/consul/blob/a1725e6b5299c6ce12e8273205f90fba31403686/GNUmakefile#L192-L198

As this is a library, wouldn't we not have a vendor directory as older versions of Go will have issues with that?

Brian

 

>
>>
>> On Thu, Aug 1, 2019 at 12:47 PM Julius Volz <juliu...@gmail.com> wrote:
>>>
>>> One more observation:
>>>
>>> I did a local experiment creating a separate module for the "promql" package and ran into one more maintenance burden: we "go mod vendor" everything into the "vendor" directory and then build using "go build -mod=vendor" (using only the local vendor copy for all dependency resolution), so before I was able to build Prometheus, I also had to run "go mod vendor" again to make sure that the "promql" package was vendored. I guess this would have to be done for every change to a local module.
>>>
>>> On Wed, Jul 31, 2019 at 1:05 PM Julius Volz <juliu...@gmail.com> wrote:
>>>>
>>>> On Wed, Jul 31, 2019 at 12:36 PM Julius Volz <juliu...@gmail.com> wrote:
>>>>>
>>>>> On Tue, Jul 30, 2019 at 5:28 PM Simon Pasquier <spas...@redhat.com> wrote:
>>>>>>
>>>>>> By the way, if we manage to split
>>>>>> github.com/prometheus/prometheus/discovery from the rest (which we
>>>>>> already agree would be good to externalize), we would get most of our
>>>>>> dependencies out of the main go.mod file. Even though not satisfying
>>>>>> (eg the module path wouldn't have the v2 suffix), it might be good
>>>>>> enough for people consuming anything other than service discovery.
>>>>>
>>>>>
>>>>> That is a good point, maybe a good way to start.
>>>>
>>>>
>>>> Actually the main module would still inherit all the dependencies of what it depends on, so this wouldn't help with the main module. But it could help if we also have separate modules for some of the other packages which can then really only depend on what they need (and those can be used externally without going through the main module).

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAM6RFu6OsAkroXTfC%2BTTMkWR5Tm7397YmFa7k16MA6iNDG6qsA%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Julius Volz]

On Thu, Aug 1, 2019 at 4:02 PM Simon Pasquier <spas...@redhat.com> wrote:

On Thu, Aug 1, 2019 at 2:52 PM Julius Volz <juliu...@gmail.com> wrote:
>
> On Thu, Aug 1, 2019 at 2:49 PM Matthias Rampke <m...@soundcloud.com> wrote:
>>
>> And there's no way to not vendor local modules?
>
>
> It doesn't seem like it, both on the vendoring side and the build side, it looks like it's an all-or-nothing approach.

Looking again at the consul example, I noticed that they explicitly
remove the "sub-module" parts from the vendor/ directory:
https://github.com/hashicorp/consul/blob/a1725e6b5299c6ce12e8273205f90fba31403686/GNUmakefile#L192-L198

Haha, oh my.

I couldn't make that work locally yet. Deleted the vendored "promql" directory and entries in modules.txt and "make build" says:

$ make build
>> building binaries
GO111MODULE=on /home/julius/gosrc/bin/promu build --prefix /home/julius/gosrc/src/github.com/prometheus/prometheus
 >   prometheus
build github.com/prometheus/prometheus/cmd/prometheus: cannot load github.com/prometheus/prometheus/promql: open /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/prometheus/promql: no such file or directory

[...]

[Julius Volz]

On Thu, Aug 1, 2019 at 4:31 PM Brian Brazil <brian....@robustperception.io> wrote:

On Thu, 1 Aug 2019 at 15:02, Simon Pasquier <spas...@redhat.com> wrote:

On Thu, Aug 1, 2019 at 2:52 PM Julius Volz <juliu...@gmail.com> wrote:
>
> On Thu, Aug 1, 2019 at 2:49 PM Matthias Rampke <m...@soundcloud.com> wrote:
>>
>> And there's no way to not vendor local modules?
>
>
> It doesn't seem like it, both on the vendoring side and the build side, it looks like it's an all-or-nothing approach.

Looking again at the consul example, I noticed that they explicitly
remove the "sub-module" parts from the vendor/ directory:
https://github.com/hashicorp/consul/blob/a1725e6b5299c6ce12e8273205f90fba31403686/GNUmakefile#L192-L198

As this is a library, wouldn't we not have a vendor directory as older versions of Go will have issues with that?

The vendoring is in the main (server) module.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLrOFUnJPYmch5pki0YH0fzbqBcCQYtw8_pdXvCLpjGzfw%40mail.gmail.com.

[Julius Volz]

On Thu, Aug 1, 2019 at 4:32 PM Julius Volz <juliu...@gmail.com> wrote:

On Thu, Aug 1, 2019 at 4:02 PM Simon Pasquier <spas...@redhat.com> wrote:

On Thu, Aug 1, 2019 at 2:52 PM Julius Volz <juliu...@gmail.com> wrote:
>
> On Thu, Aug 1, 2019 at 2:49 PM Matthias Rampke <m...@soundcloud.com> wrote:
>>
>> And there's no way to not vendor local modules?
>
>
> It doesn't seem like it, both on the vendoring side and the build side, it looks like it's an all-or-nothing approach.

Looking again at the consul example, I noticed that they explicitly
remove the "sub-module" parts from the vendor/ directory:
https://github.com/hashicorp/consul/blob/a1725e6b5299c6ce12e8273205f90fba31403686/GNUmakefile#L192-L198

Haha, oh my.

I couldn't make that work locally yet. Deleted the vendored "promql" directory and entries in modules.txt and "make build" says:

$ make build
>> building binaries
GO111MODULE=on /home/julius/gosrc/bin/promu build --prefix /home/julius/gosrc/src/github.com/prometheus/prometheus
 >   prometheus
build github.com/prometheus/prometheus/cmd/prometheus: cannot load github.com/prometheus/prometheus/promql: open /home/julius/gosrc/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/prometheus/promql: no such file or directory

[...]

Basically "go build" works (but then ignores *all* vendoring), while "go -mod=vendor build" doesn't work when something is missing in vendor/.

[Bjoern Rabenstein]

Isn't the `vendor` directory some kind of legacy support anyway?

If you are in "full Go modules" mode, why use a `vendor` directory at
all?

[Julius Volz]

On Thu, Aug 1, 2019 at 7:43 PM Bjoern Rabenstein <bjo...@rabenste.in> wrote:

Isn't the `vendor` directory some kind of legacy support anyway?

Not really, it's still an option that won't go away: https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away

 

If you are in "full Go modules" mode, why use a `vendor` directory at
all?

 To protect against sources that may disappear over time or be unavailable.

[Matthias Rampke]

That sounds more like backups … do we need vendoring as part of the build then, especially if maintaining the vendoring is what causes the additional work?

/MR

[Julius Volz]

On Fri, Aug 2, 2019 at 9:51 AM Matthias Rampke <m...@soundcloud.com> wrote:

That sounds more like backups … do we need vendoring as part of the build then, especially if maintaining the vendoring is what causes the additional work?

Not necessarily, but if we don't (and can't) build from vendor, then people will forget to update the vendoring after go.{mod,sum} dependency changes, and nobody would notice that it's falling out of date. It would also be one more weirdness to learn as a beginner: why is there a "vendor" dir, but it's not used in builds?

Btw. another benefit of using "vendor" during builds is no dependence on the network.

Overall I don't feel too strongly about keeping vendor or not though, if it causes too many problems.

[Matthias Rampke]

I suppose with go modules you can still build locally if you've built locally before?

[Julius Volz]

On Thu, Aug 1, 2019 at 2:56 PM Krasimir Georgiev <kgeo...@redhat.com> wrote:

Why not post an issue in the go repo? Usually they are quite helpful with suggestions.

I wasn't sure that this should be an issue, as it's more of a question, so I started a thread on golang-nuts: https://groups.google.com/forum/#!forum/golang-nuts

[Julius Volz]

On Fri, Aug 2, 2019 at 11:59 AM Matthias Rampke <m...@soundcloud.com> wrote:

I suppose with go modules you can still build locally if you've built locally before?

Yeah, there's a module cache in $GOPATH/pkg/mod.

[Matthias Rampke]

Then I would consider ditching vendor/ altogether not a burden – except for "software has disappeared from the internet altogether" all its use cases seem more or less covered by modules. I already find it annoying to maintain even in a single-module (exporter) repo.

/MR

[Sylvain Rabot]

I wouldn't, it's not like it has never happened, and even if with modules it wouldn't impact people that already have it cached locally, all CI processes that build from scratch would fail if a module would disappear from github & co.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAFU3N5X3VjT%3D7jO6LHg%3DFuYw0KFxVy3o45N8F_wG4vghGY6EfQ%40mail.gmail.com.

--

Sylvain Rabot <syl...@abstraction.fr>

[Julius Volz]

On Fri, Aug 2, 2019 at 12:02 PM Julius Volz <juliu...@gmail.com> wrote:

On Thu, Aug 1, 2019 at 2:56 PM Krasimir Georgiev <kgeo...@redhat.com> wrote:

Why not post an issue in the go repo? Usually they are quite helpful with suggestions.

I wasn't sure that this should be an issue, as it's more of a question, so I started a thread on golang-nuts: https://groups.google.com/forum/#!forum/golang-nuts

Ah whoops, my thread isn't there yet, as "This group is moderated. If you are a first-time poster it may take 24 hours for your post to appear in the group."

[Julius Volz]

On Fri, Aug 2, 2019 at 1:40 PM Julius Volz <juliu...@gmail.com> wrote:

On Fri, Aug 2, 2019 at 12:02 PM Julius Volz <juliu...@gmail.com> wrote:

On Thu, Aug 1, 2019 at 2:56 PM Krasimir Georgiev <kgeo...@redhat.com> wrote:

Why not post an issue in the go repo? Usually they are quite helpful with suggestions.

I wasn't sure that this should be an issue, as it's more of a question, so I started a thread on golang-nuts: https://groups.google.com/forum/#!forum/golang-nuts

Ah whoops, my thread isn't there yet, as "This group is moderated. If you are a first-time poster it may take 24 hours for your post to appear in the group."

Here's the thread now: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/golang-nuts/SH0Wg_IhGeg/SG7vB623FQAJ

[Sylvain Rabot]

I stand corrected, with GOPROXY set we should be safe from dependencies disappearing.

See: https://arslan.io/2019/08/02/why-you-should-use-a-go-module-proxy/

--

Sylvain Rabot <syl...@abstraction.fr>

[Julius Volz]

Yeah, that sounds like a decent compromise. I guess in the end I'm for getting rid of "vendor/" and instead relying on a GOPROXY (most likely the Go team's for a start) if that's the only way we can introduce proper modules without having to internally re-vendor all the time.

[Brian Brazil]

On Fri, 9 Aug 2019 at 19:08, Julius Volz <juliu...@gmail.com> wrote:

Yeah, that sounds like a decent compromise. I guess in the end I'm for getting rid of "vendor/" and instead relying on a GOPROXY (most likely the Go team's for a start) if that's the only way we can introduce proper modules without having to internally re-vendor all the time.

We're still need to update all the go.mod files all the time, which isn't really better.

Brian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YowvMneZCjJm0AZPZF6YkdW9L5PT%2BjhNdnA7-wbHnEqfpQ%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Julius Volz]

On Fri, Aug 9, 2019 at 8:16 PM Brian Brazil <brian....@robustperception.io> wrote:

On Fri, 9 Aug 2019 at 19:08, Julius Volz <juliu...@gmail.com> wrote:

Yeah, that sounds like a decent compromise. I guess in the end I'm for getting rid of "vendor/" and instead relying on a GOPROXY (most likely the Go team's for a start) if that's the only way we can introduce proper modules without having to internally re-vendor all the time.

We're still need to update all the go.mod files all the time, which isn't really better.

No, that part is dealt with by using a "replace" directive for repo-local Go modules, so it always uses the one from the local FS.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLq1jQ4uUGiXACCc_NJvxAhi29hb7rEs%3D4zNGA3eoEjdiA%40mail.gmail.com.

[Duco van Amstel]

Hello everyone,

A belated follow-up here. To provide a little bit of context: I just joined the mailing list but my colleague Bartek Płotka (bwplotka) attracted my attention on this thread when it was originally raised. He knew of my interest and relative familiarity with Go modules.

Based on the pre-existing discussion plus some brainstorming we came up with a potential solution to help the Prometheus project adopt Go modules while correctly using semantic versions.

This potential solution has now matured enough in the form of a design document that we want to share it with the prometheus-dev community to receive the comments and constructive criticism from anyone here interested in the topic.

Last but not least I wanted to thank all people who participated in the thread up to now as all your perspectives have provided critical insights and ideas that lead to our proposal!

Without further waiting, you can access and comment on the document here: https://docs.google.com/document/d/1g7wIlxn9JBJkc-2lHAb2MfCYLltWZKyNlIO6SBW1x-8/edit?usp=sharing

Duco

On Monday, 29 July 2019 17:53:58 UTC+1, Julius Volz wrote:

Hi everyone,

Go modules require semantic import versioning on a Go API level (see https://github.com/golang/go/wiki/Modules#semantic-import-versioning and https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher), not just on the level of user-visible application features. The Prometheus server repo has been converted to a single Go module, but we have never followed the mandatory rule of bumping the major version of Prometheus for any breaking Go-level API change within the repo (and we probably don't want to). This means we are currently using Go modules in a broken way, which is bad for external consumers.

How do we want to deal with this situation? I see four options, none of which sounds great:

1) Keep the status quo of illegal Go modules usage. The pain of this would be carried by anyone who depends externally on packages from the main repo. E.g. the Go tooling would assume that a minor version is backward-compatible with a previous one, although it really isn't. Currently (probably for lack of a v2 subdir) it seems that the Prometheus module is being marked as "+incompatible" when used as a dependency, so its treated as if it hasn't opted into Go modules?

2) Move away from Go modules: probably not a real option, given that this is becoming a Go standard.

3) Bump the major version for each Go-level API change: Doesn't seem like a real option unless we want to switch to a completely different feel of user-visible versions. We would have Prometheus version 113.3.1 very soon then.

4) Introduce separate modules for the binary (cmd/prometheus, cmd/promtool) packages and the rest of the packages in the repo. This would be the "proper" way to handle this with Go modules, but it does mean that you now have to manage dependency versions between the binary module(s) and the rest. We always want to build the Prometheus server against the latest versions of those other packages, so for local development you either have to add local "replace" directives into go.mod to make sure the latest version from the FS is picked up, and during Prometheus releases you have to remove those directives and instead release proper versions of the non-main modules and then update to them in the main module. (The separated modules could still be part of the same repo (https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories), or they could be split into separate repos.)

I'm tending towards either 1) or 4), with 1) meaning accepting flagrantly violating laws of Go OSS citizenship, and 4) meaning a lot more work... or maybe a combination of 1) + 4) where we keep the broken state for some packages, but separate out important packages that are frequently used externally.

What do people think? Did I miss an option?

Cheers,

Julius

[Sylvain Rabot]

Hi,

Since I believe, not unlike Bjoern, that the discussion "Accepting features into TSDB required by other users" is closely related to this one I'll try to make my case.

As far as I'm concerned, as an occasional gopher and intensive prometheus ecosystem user and enthusiast, the only considerable choices are 3 and a 5th choice I'll share.

I'll start with this "We would have Prometheus version 113.3.1 very soon then.", yes, maybe, and so what ? How is that any worse that 2.113.x ? It's true that there are only a few projects that are keen to releases a lot of major versions (I can only name github.com/Azure/azure-sdk-for-go/ off the top of my head) but why would it be considered a bad thing ?

It might look like to some that you are releasing breaking changes one after another, but, hey, why should you care ? To the vast majority it will look like you are intensively working on the project and committed to make it better, even if it implies rewrite things in an incompatible way. And by sticking to the spec of SEMVER you'll make it considerably easier for third parties relying on prometheus internals to manage the dependency and this is a win for everyone.

I strongly believe that people developing the prometheus ecosystem are mechanically potential future prometheus contributors or even maintainer (living proof of that  being Bartek). I also believe that prometheus contributors/maintainers are also prone to develop the prometheus ecosystem outside of prometheus itself. Making it easier to manage the prometheus dependency is a conditional to encourage that dynamic and everyone would benefit from it.

This said, the benefit I see for diving entirely in the Go mod spec at the prometheus project level, is to be able to have access to all major versions of the API in the same project.

If I'm not mistaken the release of Prometheus v2 was because of a new TSDB format. If it were to happen again, I believe that having access to both v2 and vX TSDB API at the same time would make it considerably easier to write tools for migrating existing v2 TSDB instances. The same would apply for PromQL for example, even if it is not likely to happen, I'm pretty sure a breaking change in PromQL today would imply a new major Prometheus release and I can only assume that having access to both v2 and vX PromQL parsers and such at the same time would be a major convenience.

If you're still not convinced, the 5th choice I would like you to consider is to still make prometheus go mod compatible once for all, even if you won't abide by the SEMVER spec in terms of breaking changes versioning. Having to deal with "github.com/prometheus/prometheus v1.8.2-0.20200110114423-1e64d757f711" imports is really tedious for cortex/thanos people as well as for others currently working in the dark trying to enrich the prometheus ecosystem.

Making a "v3" now that is go mod compatible should not be that hard and come in handy the day that you consider a new "v4" major version is released.

Regards.

On Mon, 29 Jul 2019 at 18:53, Julius Volz <juliu...@gmail.com> wrote:

Hi everyone,

Go modules require semantic import versioning on a Go API level (see https://github.com/golang/go/wiki/Modules#semantic-import-versioning and https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher), not just on the level of user-visible application features. The Prometheus server repo has been converted to a single Go module, but we have never followed the mandatory rule of bumping the major version of Prometheus for any breaking Go-level API change within the repo (and we probably don't want to). This means we are currently using Go modules in a broken way, which is bad for external consumers.

How do we want to deal with this situation? I see four options, none of which sounds great:

1) Keep the status quo of illegal Go modules usage. The pain of this would be carried by anyone who depends externally on packages from the main repo. E.g. the Go tooling would assume that a minor version is backward-compatible with a previous one, although it really isn't. Currently (probably for lack of a v2 subdir) it seems that the Prometheus module is being marked as "+incompatible" when used as a dependency, so its treated as if it hasn't opted into Go modules?

2) Move away from Go modules: probably not a real option, given that this is becoming a Go standard.

3) Bump the major version for each Go-level API change: Doesn't seem like a real option unless we want to switch to a completely different feel of user-visible versions. We would have Prometheus version 113.3.1 very soon then.

4) Introduce separate modules for the binary (cmd/prometheus, cmd/promtool) packages and the rest of the packages in the repo. This would be the "proper" way to handle this with Go modules, but it does mean that you now have to manage dependency versions between the binary module(s) and the rest. We always want to build the Prometheus server against the latest versions of those other packages, so for local development you either have to add local "replace" directives into go.mod to make sure the latest version from the FS is picked up, and during Prometheus releases you have to remove those directives and instead release proper versions of the non-main modules and then update to them in the main module. (The separated modules could still be part of the same repo (https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories), or they could be split into separate repos.)

I'm tending towards either 1) or 4), with 1) meaning accepting flagrantly violating laws of Go OSS citizenship, and 4) meaning a lot more work... or maybe a combination of 1) + 4) where we keep the broken state for some packages, but separate out important packages that are frequently used externally.

What do people think? Did I miss an option?

Cheers,

Julius

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YowRKiMM-RVCoN3bmg%3Dpoq2mc4RBo3DRf1BmN%3DQk2ZJDvg%40mail.gmail.com.

--

Sylvain Rabot <syl...@abstraction.fr>

[Matthias Rampke]

Please do not tie Prometheus server versions to library interface changes.

As someone who uses Prometheus-the-server more than Prometheus-the-module, I would lose any benefit from semantic versioning. I would be unable to tell (without close study of the changelog) whether a given bump was because a performance improvement changed a library interface, or because I need to re-write all my rules.

Thank you,

MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CADjtP1E_YKjEUPF%2BHZbBs-8X7XXsyk1AFJN2fUN6zd%2BUwm6FKw%40mail.gmail.com.

[Sylvain Rabot]

I'm a "Prometheus-the-server" user myself. I don't understand your position, today the bump in the minor versions guarantees you nothing, you have to read the changelog to know what you are dealing with.

--

Sylvain Rabot <syl...@abstraction.fr>

[Julien Pivotto]

On 05 Feb 10:43, Sylvain Rabot wrote:
> I'm a "Prometheus-the-server" user myself. I don't understand your
> position, today the bump in the minor versions guarantees you nothing, you
> have to read the changelog to know what you are dealing with.

That is of course not true.

A running prometheus 2.0.0 can be updated seamlessly to 2.16.0-rc.0.
If that's not the case then you can fill in bug requests.

> >>> <https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YowRKiMM-RVCoN3bmg%3Dpoq2mc4RBo3DRf1BmN%3DQk2ZJDvg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>> .

> >>>
> >>
> >>
> >> --
> >> Sylvain Rabot <syl...@abstraction.fr>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Prometheus Developers" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to prometheus-devel...@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/prometheus-developers/CADjtP1E_YKjEUPF%2BHZbBs-8X7XXsyk1AFJN2fUN6zd%2BUwm6FKw%40mail.gmail.com

> >> <https://groups.google.com/d/msgid/prometheus-developers/CADjtP1E_YKjEUPF%2BHZbBs-8X7XXsyk1AFJN2fUN6zd%2BUwm6FKw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >
>
> --
> Sylvain Rabot <syl...@abstraction.fr>
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CADjtP1HueRmOj%2Byynza5xEXxkvwZqf55ECGdnKyvbLVgCOJDMg%40mail.gmail.com.

--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[Julien Pivotto]

On 05 Feb 10:47, Julien Pivotto wrote:
> On 05 Feb 10:43, Sylvain Rabot wrote:
> > I'm a "Prometheus-the-server" user myself. I don't understand your
> > position, today the bump in the minor versions guarantees you nothing, you
> > have to read the changelog to know what you are dealing with.
>
>
> That is of course not true.

Sorry for the bad wording.. Please read:

That should not be true.

[Sylvain Rabot]

I meant at the level Matthias talked about:

- performance improvement

- library interface

- needing to rewrite the rules

--

Sylvain Rabot <syl...@abstraction.fr>

[Matthias Rampke]

I have not had to rewrite any rules since 2.0?

[Julius Volz]

Doing Go-level semantic versioning of the entire Prometheu server is something that's very unlikely to happen.

Currently our semantic versioning is signalling towards the user of Prometheus, of whom there are vastly more than reusers of our Go packages, and who are the target audience of the versioning, first and foremost. While the Prometheus server major version does not guarantee stability in all areas of Prometheus, we do guarantee no breaking changes (except for accidental ones) in many defined areas: https://prometheus.io/docs/prometheus/latest/stability/

I too am really annoyed by the trouble that reusing Prometheus packages causes outside of Prometheus, but that pain doesn't warrant such a big (and completely unrelated to user-visible features) versioning change toward the user. There would still be the option of putting the server itself (just the main binary, none of its other packages) into its own Go module and version everything else in a way that would be strictly Go-modules-compatible. Duco had some ideas around creating automated tooling around that (https://docs.google.com/document/d/1g7wIlxn9JBJkc-2lHAb2MfCYLltWZKyNlIO6SBW1x-8/edit), but I admit that I haven't followed that thread of the effort yet.

[Duco van Amstel]

Duco had some ideas around creating automated tooling around that (https://docs.google.com/document/d/1g7wIlxn9JBJkc-2lHAb2MfCYLltWZKyNlIO6SBW1x-8/edit), but I admit that I haven't followed that thread of the effort yet.

Just too follow-up on that bit, something I should have done quite some time ago: I've been actively developing the modularise tool, although in a private repo for the moment. Following-up on my proposal in August / September I had to take 2 months of off developing it for personal reasons but I've been working on it nearly daily since mid-December in my free time. As I posted on this issue here (https://github.com/coreos/prometheus-operator/issues/2988) where Bartek mentioned the tool, I hope to have a functional end-to-end prototype in the next month or two which I will share in this thread as soon as it's ready.

You received this message because you are subscribed to a topic in the Google Groups "Prometheus Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-developers/F1Vp0rLk3TQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YowMuG__k4%2BLiJ9QEyLJ53EeSJ5GDFGk47Y7RK9LSyr2FQ%40mail.gmail.com.

[Sylvain Rabot]

I believe the last time you had to is when new rule format was added and it happened in 2.0 Alpha.3 (https://prometheus.io/blog/2017/06/21/prometheus-20-alpha3-new-rule-format/) so indeed it was part of a major version release.

But if you have not been reading the changelogs since 2.0.0 you couldn't have known if the releases contained performance improvements or internal library changes.

--

Sylvain Rabot <syl...@abstraction.fr>

[Brian Brazil]

On Wed, 5 Feb 2020 at 10:11, Duco van Amstel <duco.va...@gmail.com> wrote:

Duco had some ideas around creating automated tooling around that (https://docs.google.com/document/d/1g7wIlxn9JBJkc-2lHAb2MfCYLltWZKyNlIO6SBW1x-8/edit), but I admit that I haven't followed that thread of the effort yet.

Just too follow-up on that bit, something I should have done quite some time ago: I've been actively developing the modularise tool, although in a private repo for the moment. Following-up on my proposal in August / September I had to take 2 months of off developing it for personal reasons but I've been working on it nearly daily since mid-December in my free time. As I posted on this issue here (https://github.com/coreos/prometheus-operator/issues/2988) where Bartek mentioned the tool, I hope to have a functional end-to-end prototype in the next month or two which I will share in this thread as soon as it's ready.

Per the last developer summit, the current intention is that someone can run this tool and output it to a different github organisation - but we're not going to make changes to the Prometheus org to facilitate it at this time.

Brian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAP8_5EDAcFv4Tep%3DV%3DGV39zMKLz3LYWm2dhiRrbNtup8dpwtHQ%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Julius Volz]

Thanks, great to hear :)

[Sylvain Rabot]

On Wed, 5 Feb 2020 at 11:03, Julius Volz <juliu...@gmail.com> wrote:

Doing Go-level semantic versioning of the entire Prometheu server is something that's very unlikely to happen.

Currently our semantic versioning is signalling towards the user of Prometheus, of whom there are vastly more than reusers of our Go packages, and who are the target audience of the versioning, first and foremost. While the Prometheus server major version does not guarantee stability in all areas of Prometheus, we do guarantee no breaking changes (except for accidental ones) in many defined areas: https://prometheus.io/docs/prometheus/latest/stability/

All right, that is a very good point (the versioning done towards the user).

Could it still be considered to make prometheus/prometheus go mod compatible with a v3 just for the sake of cortex/thanos/...etc maintainers ?

I'm currently trying to give it a go (https://github.com/sylr/prometheus/commits/v3).

--

Sylvain Rabot <syl...@abstraction.fr>

[Julius Volz]

On Wed, Feb 5, 2020 at 11:23 AM Sylvain Rabot <syl...@abstraction.fr> wrote:

On Wed, 5 Feb 2020 at 11:03, Julius Volz <juliu...@gmail.com> wrote:

Doing Go-level semantic versioning of the entire Prometheu server is something that's very unlikely to happen.

Currently our semantic versioning is signalling towards the user of Prometheus, of whom there are vastly more than reusers of our Go packages, and who are the target audience of the versioning, first and foremost. While the Prometheus server major version does not guarantee stability in all areas of Prometheus, we do guarantee no breaking changes (except for accidental ones) in many defined areas: https://prometheus.io/docs/prometheus/latest/stability/

All right, that is a very good point (the versioning done towards the user).

Could it still be considered to make prometheus/prometheus go mod compatible with a v3 just for the sake of cortex/thanos/...etc maintainers ?

If I understand you correctly that would mean releasing a Prometheus 3.x with no user-visible breaking changes, but then to stay a strictly valid Go modules user, we'd still have to update our major version on any Go-level API changes? IMO either of those is unlikely to happen, but I generally think the approach of completely mirroring our packages to a new place, but with Go versioning rules being better met, is the most likely avenue at the moment. I'd also not be super opposed to having separate (and correctly Go-versioned) submodules within the same repo for the external code users, but so far it looks like people think that's imposing too much of a maintenance burden on people developing on the Prometheus codebase itself.

[Sylvain Rabot]

No, I meant making a v3 that is go module compatible but still continue to use the current way of releasing prometheus versions.

The only immediate benefit would be to ease the burden of third parties like cortex and thanos who could have clean "github.com/prometheus/prometheus v3.x.x" reference in their go.sum.

But since we are not tied to respect SEMVER for internal breaking changes we could maybe make v2 go mod compatible so no major release with no user visible breaking change has to be released.

--

Sylvain Rabot <syl...@abstraction.fr>

[Sylvain Rabot]

On Wed, 5 Feb 2020 at 11:44, Sylvain Rabot <syl...@abstraction.fr> wrote:

No, I meant making a v3 that is go module compatible but still continue to use the current way of releasing prometheus versions.

The only immediate benefit would be to ease the burden of third parties like cortex and thanos who could have clean "github.com/prometheus/prometheus v3.x.x" reference in their go.sum.

Sorry, I meant go.mod.

 

But since we are not tied to respect SEMVER for internal breaking changes we could maybe make v2 go mod compatible so no major release with no user visible breaking change has to be released.

On Wed, 5 Feb 2020 at 11:31, Julius Volz <juliu...@gmail.com> wrote:

On Wed, Feb 5, 2020 at 11:23 AM Sylvain Rabot <syl...@abstraction.fr> wrote:

On Wed, 5 Feb 2020 at 11:03, Julius Volz <juliu...@gmail.com> wrote:

Doing Go-level semantic versioning of the entire Prometheu server is something that's very unlikely to happen.

Currently our semantic versioning is signalling towards the user of Prometheus, of whom there are vastly more than reusers of our Go packages, and who are the target audience of the versioning, first and foremost. While the Prometheus server major version does not guarantee stability in all areas of Prometheus, we do guarantee no breaking changes (except for accidental ones) in many defined areas: https://prometheus.io/docs/prometheus/latest/stability/

All right, that is a very good point (the versioning done towards the user).

Could it still be considered to make prometheus/prometheus go mod compatible with a v3 just for the sake of cortex/thanos/...etc maintainers ?

If I understand you correctly that would mean releasing a Prometheus 3.x with no user-visible breaking changes, but then to stay a strictly valid Go modules user, we'd still have to update our major version on any Go-level API changes? IMO either of those is unlikely to happen, but I generally think the approach of completely mirroring our packages to a new place, but with Go versioning rules being better met, is the most likely avenue at the moment. I'd also not be super opposed to having separate (and correctly Go-versioned) submodules within the same repo for the external code users, but so far it looks like people think that's imposing too much of a maintenance burden on people developing on the Prometheus codebase itself.

--

Sylvain Rabot <syl...@abstraction.fr>

--

Sylvain Rabot <syl...@abstraction.fr>

[Duco van Amstel]

But since we are not tied to respect SEMVER for internal breaking changes we could maybe make v2 go mod compatible so no major release with no user visible breaking change has to be released.

If they are internal these API's & packages should be in an internal subtree. As far as Go + SemVer is concerned, any change of the exported symbols or the semantics of exported functions & methods in a non-internal package is a breaking change. Having a "stability" document is good and encouraged but it's merely a description of intent. If you don't follow the actual SemVer definition you will still be breaking users, whether that is your intent or not, sometimes in vary unexpected ways (example: via incompatible transitive dependencies).

You are also now introducing a unnecessary second step in someone's understanding of whether a Prometheus upgrade will break them or not:

1. Asses whether the upgrade changes the major of the project. If it is then the upgrade will likely require extra work to not break them.
2. If it's not a major change consult the "stability" document on whether the packages they import are included.

That second step should not be necessary: follow SemVer. Whether it is by increasing your major version or by moving the packages not guaranteed to be stable into an internal subtree (and potentially do a one-of major version bump to represent such a breaking API change inside your non-stable changes).

Note: This is speaking from a pure perspective of how Go Modules work and the associated best-practices, sharing my knowledge in this area. I am not personally a daily user of Prometheus and it's interfaces myself and hence can not meaningfully discuss the low-level semantics and details of the specific project.

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-developers/F1Vp0rLk3TQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CADjtP1GDtuZ%2BD%3DxEq4MmNmDMiU9GWRyVFARpWd%3Daiej3HQv6qQ%40mail.gmail.com.

[Julien Pivotto]

On 05 Feb 11:15, Sylvain Rabot wrote:
> I believe the last time you had to is when new rule format was added and it
> happened in 2.0 Alpha.3 (
> https://prometheus.io/blog/2017/06/21/prometheus-20-alpha3-new-rule-format/)
> so indeed it was part of a major version release.
>
> But if you have not been reading the changelogs since 2.0.0 you couldn't
> have known if the releases contained performance improvements or internal
> library changes.

You don't know about internal library changes in the changelog neither.

[Bjoern Rabenstein]

I think it became already obvious from the discussion so far: Semantic
versioning for the user of the Prometheus server is completely
different from semantic versioning for the user of Go packages
contained in the Prometheus code base.

Conflating both will be bad for both parties.

We could have a _separate_ versioning, using vx.y.z tags for the Go
module versioning and a new tag naming schema (e.g. release-x.y.z) for
user releases.

Besides the obvious concerns about this, I also think that semantic
versioning for a large number of Go packages together is problematic
in any case. Any breaking change in any one package will bump the
major version for _all_ packages, leading to new import paths in the
Go modules world, which can lead to interoperability problems for
indirect users of the packages. (They could find themselves in the
situation of assigning a v2/model.Label spit out by one of their
direct dependencies to a v3/model.Label required by another of their
direct dependencies.)

That's why I think Duco's Modularize project has some
merits. Alternatively (as I said before), I could see better tooling
for maintaining multiple Go modules within the same Git repo as a
viable (and less invasive) solution.

Therefore, I'd prefer to see how Modularize turns out in practice (or
if alternatively somebody comes up with tooling and/or easy-to-follow
best practices for multi-module repos) and then reconsider what we can
do in prometheus/prometheus.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Sylvain Rabot]

After giving it more search, it came to (my) light that go mod handles "special" git tags for go mod sub-versioning.

https://github.com/golang/go/wiki/Modules#what-are-multi-module-repositories

The tag for version 1.2.3 of module "my-repo/foo/rop" is "foo/rop/v1.2.3".

I.E, "go get github.com/prometheus/prometheus/tsdb/v...@v2.13.0" would resolve "github.com/prometheus/prometheus.git" at tag "tsdb/v2.13.0"

With that in mind, I believe having multiple independent go modules in the same git repo is OK.

With this we can leverage the initial reason of having those "core" modules inside prometheus/prometheus (no sync overhead) and a decent independent development oriented life cycle.

We currently have release shepherds, I think that if we were to implement this, we would need a shepherd per module responsible for releasing versions of "his" module(s).

I made a Draft PR to get a sense of it: https://github.com/prometheus/prometheus/pull/6888/commits/44149825edb0b5c92885f362b6f3ae1643e2f451

It's pretty light if we exclude removing vendoring altogether (because I don't believe vendoring makes sense anymore with the latest versions of go and because each module having its own go mod needs its own vendor/ dir).

The only problem I encountered came from promu. In the case we want to build a binary in a module that has its own go.mod file (like tsdb), the go build command needs to be run from within the dir of the module (because of the relative paths in the go.mod replace directive I assume). I already have a patch for this (https://github.com/sylr/promu/commit/c6e6a2bcdf5d50338b1ee70022c40152a8992c82).

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

[Sylvain Rabot]

Ok so I've been talking to Duco about this and he convinced me that using in repo replace directives with relative paths is not a good idea.

If he has the time he can explain it much better than I could.

--

Sylvain Rabot <syl...@abstraction.fr>

[Duco van Amstel]

For what it's worth I've just started a fresh topic with my first results using the modularise tool.