Moving prometheus/tsdb to prometheus-junkyard/tsdb

[Julien Pivotto]

Dear community,

In the dev summit 2019/1, it was decided to move tsdb to the junkyard:
https://docs.google.com/document/d/1NQIX78nwBhfLZD3pAb0PK-uBKYqnkzjjVhOQ-kIaEGU/edit

The https://github.com/prometheus/tsdb has been moved in Augustus 2019 inside
the prometheus repo itself: https://github.com/prometheus/prometheus/tsdb

I would like to trigger the move of the old repo to
https://github.com/prometheus-junkyard/tsdb at the end of this month (29
February). That is 200 days after the repo has been merged in the
prometheus server repo.

GitHub says that "All links to the previous repository location are
automatically redirected to the new location":
https://help.github.com/en/github/administering-a-repository/transferring-a-repository
which means that links to pull requests etc should still work after
the rename.

If someone has strong arguments against this, please reply to this
message.

--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[Julien Pivotto]

On 14 Feb 00:19, Julien Pivotto wrote:
> Dear community,
>
> In the dev summit 2019/1, it was decided to move tsdb to the junkyard:
> https://docs.google.com/document/d/1NQIX78nwBhfLZD3pAb0PK-uBKYqnkzjjVhOQ-kIaEGU/edit
>
> The https://github.com/prometheus/tsdb has been moved in Augustus 2019 inside
> the prometheus repo itself: https://github.com/prometheus/prometheus/tsdb
>
> I would like to trigger the move of the old repo to
> https://github.com/prometheus-junkyard/tsdb at the end of this month (29
> February). That is 200 days after the repo has been merged in the
> prometheus server repo.

At the same time (Feb 29), I will mark the existing
https://github.com/prometheus-junkyard as "archived" and therefore make
them read only.

[Bartłomiej Płotka]

LGTM, thanks for doing this. (:

Kind Regards,

Bartek

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200216095831.GA27933%40oxygen.

[Bartłomiej Płotka]

But also just enjoy the weekend =D

[Sylvain Rabot]

Hi,

Before doing so, could we make sure this action can be reverted ?

I'd hate that, if we realize TSDB needs its own lifecycle, we wouldn't be able to reopen github.com/prometheus/tsdb because of some github logic.

Regards.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200213231949.GA18483%40oxygen.

--

Sylvain Rabot <syl...@abstraction.fr>

[Bartłomiej Płotka]

Hm, why would it need to have its own lifecycle?

We waited for some time exactly to make sure that all is safe for the move.

Bartek

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CADjtP1GFSFug0WYQO0svnJv2%3D2Xgz-GSzfL1ThN_kHeoznOQsg%40mail.gmail.com.

[Sylvain Rabot]

I strongly believe that TSDB is a cornerstone of the prometheus ecosystem (and not prometheus/prometheus alone) and as such should have its own lifecycle.

I also believe the original reason for the move ("Keeping them in sync, versioning etc, is a pain") should have been solved by tooling.

I'm sure at one point people using TSDB outside of prometheus will complain about the TSDB versioning being tied to prometheus.

So I'd like to make sure we can go back because even if the move is considered safe now, I'm persuaded it only benefits internal prometheus developments at the expense of the whole ecosystem.

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

[Julien Pivotto]

On 16 Feb 12:52, Sylvain Rabot wrote:
> I strongly believe that TSDB is a cornerstone of the prometheus ecosystem
> (and not prometheus/prometheus alone) and as such should have its own
> lifecycle.
>
> I also believe the original reason for the move ("Keeping them in sync,
> versioning etc, is a pain") should have been solved by tooling.
>
> I'm sure at one point people using TSDB outside of prometheus will complain
> about the TSDB versioning being tied to prometheus.
>
> So I'd like to make sure we can go back because even if the move is
> considered safe now, I'm persuaded it only benefits internal prometheus
> developments at the expense of the whole ecosystem.
>
> Regards.

There are discussions in progress outside of this discussion.

I would like to add that golang versioning totally tolerate multiple
modules in one git repo with different versioning schemes.

https://github.com/hashicorp/consul/tree/master/api

is a go module on its own, module github.com/hashicorp/consul/api
inside the github.com/hashicorp/consul repo.

They have dedicated versions (see
https://github.com/hashicorp/consul/tree/api/v1.4.0): consul/api is v1.4.0,
while consul is v1.7.0.

So it looks like we could get the advantages of a single repo and a
dedicated module lifecycle if we need to.

> >> <https://groups.google.com/d/msgid/prometheus-developers/CADjtP1GFSFug0WYQO0svnJv2%3D2Xgz-GSzfL1ThN_kHeoznOQsg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >
>
> --
> Sylvain Rabot <syl...@abstraction.fr>

--

[Bartłomiej Płotka]

Thanks for your opinion Sylvian! I agree. I totally see the versioning cycle being a problem. In fact as Thanos maintainers, together with Cortex maintainers, we are probably the biggest users of TSDB alone, so we feel the pain (which is not really THAT big).

That's why we started a discussion around proper go module versioning to solve this: https://groups.google.com/forum/#!searchin/prometheus-developers/versioning%7Csort:date/prometheus-developers/F1Vp0rLk3TQ/XyXngVP8AAAJ

Hopefully once solved, we can keep mono-repo like structure but have separate versioning for TSDB. My personal opinion is that the work we do with Duco with help here: https://github.com/Helcaraxan/modularise-example-core

We will see it in the soon future. 

Kind Regards,

Bartek

[Bartłomiej Płotka]

s/Sylvian/Sylvain

Sorry (:

[Sylvain Rabot]

On Sun, 16 Feb 2020 at 13:00, Julien Pivotto <roidel...@inuits.eu> wrote:

On 16 Feb 12:52, Sylvain Rabot wrote:
> I strongly believe that TSDB is a cornerstone of the prometheus ecosystem
> (and not prometheus/prometheus alone) and as such should have its own
> lifecycle.
>
> I also believe the original reason for the move ("Keeping them in sync,
> versioning etc, is a pain") should have been solved by tooling.
>
> I'm sure at one point people using TSDB outside of prometheus will complain
> about the TSDB versioning being tied to prometheus.
>
> So I'd like to make sure we can go back because even if the move is
> considered safe now, I'm persuaded it only benefits internal prometheus
> developments at the expense of the whole ecosystem.
>
> Regards.

There are discussions in progress outside of this discussion.

Yes, I know, I still do not understand why TSDB has been moved into prometheus before those discussions have been sorted ...

 

I would like to add that golang versioning totally tolerate multiple
modules in one git repo with different versioning schemes.

https://github.com/hashicorp/consul/tree/master/api

is a go module on its own, module github.com/hashicorp/consul/api
inside the github.com/hashicorp/consul repo.

They have dedicated versions (see
https://github.com/hashicorp/consul/tree/api/v1.4.0): consul/api is v1.4.0,
while consul is v1.7.0.

So it looks like we could get the advantages of a single repo and a
dedicated module lifecycle if we need to.

I must admit I did not know that but after a first quick glance I'm sure it will become a mess. Having several go modules in the same repo, which ultimately, if their versions differ, will no longer reflect git tags ... I wouldn't do that, ever.

--

Sylvain Rabot <syl...@abstraction.fr>

[Sylvain Rabot]

No worries :)

--

Sylvain Rabot <syl...@abstraction.fr>

[Sylvain Rabot]

On Sun, 16 Feb 2020 at 13:01, Bartłomiej Płotka <bwpl...@gmail.com> wrote:

Thanks for your opinion Sylvian! I agree. I totally see the versioning cycle being a problem. In fact as Thanos maintainers, together with Cortex maintainers, we are probably the biggest users of TSDB alone, so we feel the pain (which is not really THAT big).

Yeah I knew that you, most of all people, would feel that pain. But I beg to differ on the "which is not really THAT big", I spent a couple of afternoons trying to keep prometheus up to date in thanos and I do not keep good memories of it.

--

Sylvain Rabot <syl...@abstraction.fr>

[Bartłomiej Płotka]

At least when it fails it fails immediately in build time :p

So not sure how separate versioning would help you there. It would be a new major version of tsdb module, sure. But what then? It's will require extra manual bump anyway then you will have same work to be done, am I right?

Kind Regards,

Bartek

[Sylvain Rabot]

It wouldn't have, separate versioning can't solve merging upstreams that have breaking changes.

It's not about that, it's about respect towards people relying on TSDB.

I'd like prometheus people to acknowledge that their project is growing outside their reach. This is a recognition that they did well, very well, and they should be really proud of that and embrace it. I believe that embracing it means starting to respect common development rules and it should start with go modules conventions and SEMVER.

Like I said, TSDB is a cornerstone of the prometheus ecosystem.

If we want to make the ecosystem thrive, I believe major components like TSDB should have their own development oriented lifecycle (prometheus/prometheus has an user oriented lifecycle) which respects both go module conventions and SEMVER git tags to make life easier for third parties that use them.

That does not prevent anyone to make breaking changes to them and that advertises those who rely on it what they can expect when managing their prometheus dependencies.

You as Thanos maintainer want to upgrade from TSDB v0.7.0 to v0.7.2, you should have almost nothing to do. But, if you want to upgrade from v0.7.2 to v2.0.3, you would know that should roll up your sleeves because you're in for some heavy lifting.

Today if you want to upgrade prometheus in Thanos from 2.15.0 to 2.16.0 you have absolutely no idea what you are stepping into. Well, you personally might as you are quite involved in both projects but that is not the case for everyone.

That's why I believe that the TSDB move was a bad thing and should never have happened. Now that is has, I just want to make sure it's not something we can't go back from. It might never happen, or it might, only time will tell.

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

[Bartłomiej Płotka]

I fully agree with your arguments here. Since both Cortex and Thanos are very both active in Prometheus Team, it's easier for us to understand that potential obstacles each version introduces. That's why I am looking forward for proper versioning in some way of another. 

If we did move prematurely... Maybe, but we acknowledged that only Prometheus, Thanos, Cortex, Loki and Conprof are using TSDB. Any other project, if using, did not give any objections on public discussions if to move TSDB or not. Since it's was very painful for Prometheus we went ahead. If you think we were wrong or we could improve something in the process, let us know. (:

In the same way decision to put TSDB now to junkyard was done in public and with several months ahead.

Anyway I think when moved we can get back TSDB alone anytime, so this should answer your concern. However, in fact you can fork it and bring back as your fork anytime as well: it's open source. But we, as a team, agreed to maintain it inside Prometheus repo and IMO we see many benefits of it: e.g reduction of complexity around tons of unnecessary interfaces, velocity of release process and tsdb bug fixes! I think it was a good decision. (:

Kind Regards,

Bartek

[Sylvain Rabot]

On Sun, 16 Feb 2020 at 14:34, Bartłomiej Płotka <bwpl...@gmail.com> wrote:

I fully agree with your arguments here. Since both Cortex and Thanos are very both active in Prometheus Team, it's easier for us to understand that potential obstacles each version introduces. That's why I am looking forward for proper versioning in some way of another. 

If we did move prematurely... Maybe, but we acknowledged that only Prometheus, Thanos, Cortex, Loki and Conprof are using TSDB. Any other project, if using, did not give any objections on public discussions if to move TSDB or not. Since it's was very painful for Prometheus we went ahead. If you think we were wrong or we could improve something in the process, let us know. (:

Björn comment about waiting for the outcome of the go module compliance has been ignored (https://github.com/prometheus/prometheus/pull/5805#issuecomment-516387796) although 3 people, me included, sided with him with a thumb up.

--

Sylvain Rabot <syl...@abstraction.fr>

[Sylvain Rabot]

On Sun, 16 Feb 2020 at 15:08, Sylvain Rabot <syl...@abstraction.fr> wrote:

On Sun, 16 Feb 2020 at 14:34, Bartłomiej Płotka <bwpl...@gmail.com> wrote:

I fully agree with your arguments here. Since both Cortex and Thanos are very both active in Prometheus Team, it's easier for us to understand that potential obstacles each version introduces. That's why I am looking forward for proper versioning in some way of another. 

If we did move prematurely... Maybe, but we acknowledged that only Prometheus, Thanos, Cortex, Loki and Conprof are using TSDB. Any other project, if using, did not give any objections on public discussions if to move TSDB or not. Since it's was very painful for Prometheus we went ahead. If you think we were wrong or we could improve something in the process, let us know. (:

Björn comment about waiting for the outcome of the go module compliance has been ignored (https://github.com/prometheus/prometheus/pull/5805#issuecomment-516387796) although 3 people, me included, sided with him with a thumb up.

To be fair he later green lighted it: https://github.com/prometheus/prometheus/pull/5805#issuecomment-520774612

--

Sylvain Rabot <syl...@abstraction.fr>

[Julien Pivotto]

All this seems at least unrelated to the move of an archived repo to the junkyard after 200 days. I will test moving a repo back and forth before moving tsdb, just to be 100÷ sure that we can do it, so we can close the discussion here.

>>> and *they should be really proud of that and embrace it*. I believe

>>> that embracing it means starting to respect common development rules and it
>>> should start with go modules conventions and SEMVER.
>>>
>>> Like I said, TSDB is a cornerstone of the prometheus ecosystem.
>>>

>>> If we want to make the ecosystem thrive, I believe *major components
>>> like TSDB* should have their own *development oriented lifecycle*

[Sylvain Rabot]

Thank you.

--

Sylvain Rabot <syl...@abstraction.fr>

[Julien Pivotto]

Hi Sylvain & list,

tsdb has been moved to prometheus-junkyard.

As requested by Sylvain I ran a test first: I moved test-infra-old back and forth and it worked.

----- Original Message -----
From: Sylvain Rabot <syl...@abstraction.fr>
To: Julien Pivotto <roidel...@inuits.eu>
Cc: prometheus-developers <prometheus...@googlegroups.com>
Sent: Sun, 16 Feb 2020 12:25:39 +0100 (CET)
Subject: Re: [prometheus-developers] Moving prometheus/tsdb to prometheus-junkyard/tsdb

[Bartłomiej Płotka]

Thanks!

And thanks Sylvain for raising a concern, now we know we can revert any time.

Bartek

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/310334397.2186.1582959709055.JavaMail.zimbra%40inuits.eu.