Changing Prometheus from lazy to rough consensus?

[Richard Hartmann]

Dear all,

as most of you know, the Prometheus governance[1] has three levels of
formalized decision making. In increasing order of strength and
requirements, they are:
* Consensus
* Majority vote
* Supermajority vote


The consensus definition is taken from CouchDB[2] and goes deeply into
its intention to be lightweight and easy to reverse. Yet, a careful
and literal reading of it enshrines what is factually a veto
mechanism. Vetos are easy to utter but hard to unblock short of voting
or an in-between secondary consensus mechanism we have come to call
"Call for Consensus" on the mailing lists.

In practice, it sometimes feels as if consensus is heavier a process
than outright voting, an order which goes against the initial
intention behind our governance.

Contrast this to the IETF definition[3] which is more
formal, but explicitly acknowledges that "Rough consensus is when all
issues are addressed, but not necessarily accommodated". IETF has
decades of experience with highly contentious topics which are,
sometimes, discussed by directly opposed parties. It's important to
note that the IETF rough consensus mechanism presumes the existence of
Chairs, but I think we would not need to rely on such heavyweight
process.

Personally, I am more used to the IETF definition and have not seen
the CouchDB version seen wide adoption, which is part of the reason
why I adopted rough IETF consensus for the Grafana governance[4]
recently released.

In order to make Prometheus move more quickly and more smoothly, I
would like to suggest adoption the following change:
https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491


Best,
Richard

[1] https://prometheus.io/governance/
[2] https://couchdb.apache.org/bylaws.html#lazy
[3] https://tools.ietf.org/html/rfc7282
[4] https://github.com/grafana/grafana/blob/master/GOVERNANCE.md

[Ben Kochie]

+1 for IETF Rough consensus

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgRvwnU%2BdFRMQ2n%2Bqf6D0oyej-btzsQYh8LjBV9FJ1a78w%40mail.gmail.com.

[Brian Brazil]

How does this work in practice, who decides if an issue is "addressed"? This sounds like it might boil down to the same thing in reality.

Brian

 

In order to make Prometheus move more quickly and more smoothly, I
would like to suggest adoption the following change:
https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491


Best,
Richard

[1] https://prometheus.io/governance/
[2] https://couchdb.apache.org/bylaws.html#lazy
[3] https://tools.ietf.org/html/rfc7282
[4] https://github.com/grafana/grafana/blob/master/GOVERNANCE.md

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgRvwnU%2BdFRMQ2n%2Bqf6D0oyej-btzsQYh8LjBV9FJ1a78w%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Richard Hartmann]

On Mon, May 25, 2020 at 2:30 PM Brian Brazil
<brian....@robustperception.io> wrote:

> How does this work in practice, who decides if an issue is "addressed"? This sounds like it might boil down to the same thing in reality.

In the IETF Working Groups the Chairs make this call.

For Prometheus, I think we should be able to do without Chairs as "has
this concern been addressed" should be somewhat self-evident even if
it has not necessarily been accommodated.

I did consider suggesting a Chair model and running for Chair, but I
wanted to start the discussion with the minimum amount of changes.


Best,
Richard

[Brian Brazil]

On Mon, 25 May 2020 at 14:00, Richard Hartmann <richih.ma...@gmail.com> wrote:

On Mon, May 25, 2020 at 2:30 PM Brian Brazil
<brian....@robustperception.io> wrote:

> How does this work in practice, who decides if an issue is "addressed"? This sounds like it might boil down to the same thing in reality.

In the IETF Working Groups the Chairs make this call.

For Prometheus, I think we should be able to do without Chairs as "has
this concern been addressed" should be somewhat self-evident even if
it has not necessarily been accommodated.

I'm not really convinced here, for example in the cases that come to mind the concern tends to be ignored rather than addressed. Also in the example wording "considered" is quite vague, what counts as considered enough and who does the considering?

Brian

 

I did consider suggesting a Chair model and running for Chair, but I
wanted to start the discussion with the minimum amount of changes.


Best,
Richard

--

Brian Brazil

www.robustperception.io

[Richard Hartmann]

On Mon, May 25, 2020 at 3:17 PM Brian Brazil
<brian....@robustperception.io> wrote:

> I'm not really convinced here, for example in the cases that come to mind the concern tends to be ignored rather than addressed. Also in the example wording "considered" is quite vague, what counts as considered enough

There's always some human element of interpretation and people with
differing positions will naturally evaluate arguments and assessments
differently; else, they would not disagree in the first place.


> who does the considering?

In IETF: Chairs

Are you suggesting we should have a Prometheus [Working Group] Chair
to reduce interpretative space?


Richard

[Brian Brazil]

My point is more that it sounds like a fundamental disagreement would change from a lack of lazy concensus to a lack of concensus on things being considered enough - which is no real change. After all if someone was already happy that their objection was sufficiently addressed, why would they block any further? I don't see how having an additional level of process on top of the existing maintainers by creating what could end up as a kingmaker would help, governance already has mechanisms to deal with disagreement where consensus isn't working out.

--

Brian Brazil

www.robustperception.io

[Richard Hartmann]

On Mon, May 25, 2020 at 3:57 PM Brian Brazil
<brian....@robustperception.io> wrote:

> governance already has mechanisms to deal with disagreement where consensus isn't working out.

As per my initial email, I think the balance is off between the
options and I would prefer to not default to votes.


Richard

[Julius Volz]

In general, +10 for a change in this direction, since currently the balance is too much in favor of inaction - it's really easy to veto something, but really hard to unblock it because team-wide votes are high-overhead.

I would also want to understand a bit more specifically how this would work in practice though. E.g. when there are three people arguing one way on an issue, and one person against, and all views have been considered and arguments are turning in circles, can the majority (with respect to that discussion) just go ahead and decide / merge things? I guess in the worst case, when someone feels unheard, they can still call for a team-wide vote, but the cost of calling for that vote would then be carried by the minority, not the majority? So things would be more biased towards action.

On Mon, May 25, 2020 at 2:04 PM Richard Hartmann <richih.ma...@gmail.com> wrote:

[Richard Hartmann]

On Mon, May 25, 2020 at 7:08 PM Julius Volz <juliu...@gmail.com> wrote:

> I would also want to understand a bit more specifically how this would work in practice though. E.g. when there are three people arguing one way on an issue, and one person against, and all views have been considered and arguments are turning in circles, can the majority (with respect to that discussion) just go ahead and decide / merge things? I guess in the worst case, when someone feels unheard, they can still call for a team-wide vote, but the cost of calling for that vote would then be carried by the minority, not the majority? So things would be more biased towards action.

That's one possible mode of operation, yes.

Hopefully, there would be fewer lockups by shifting from default-deny
to default-majority-ish.

If it does not work out to our shared satisfaction, we can always
refine the governance further, e.g. introduce a Chair system though,
again, I would like to avoid that.


Best,
Richard

[Julien Pivotto]

I would vote :+1: on changing to rough consensus.

--
Julien Pivotto
@roidelapluie

[Bartłomiej Płotka]

Hi

I support 100% of what Julius said: 

>  In general, +10 for a change in this direction, since currently, the balance is too much in favor of inaction - it's really easy to veto something, but really hard to unblock it because team-wide votes are high-overhead.

> I would also want to understand a bit more specifically how this would work in practice though. 

+1 on moving to a rough consensus as I think it's stepping into a good direction. 

Overall, I feel that our team is not a children's playground, we are team of professionals. This means that we trust each other, so no one in Prometheus will decide something totally against someone else's strong opposite vote. I would see it as rather a message to the community that we no longer accept blocking decisions for so long as we used to. If anyone has strong objections on a certain topic, they have to suggest alternatives or actionable items that have to be addressed first. This is what I would see as a practical form of such consensus which also hopefully avoids "concern tends to be ignored rather than addressed" problem that Brian's mentioned. 

On a separate note, while I believe in maintaining high project quality for everything we ship, I think we should be way more open into the experimenting bit more in Prometheus. Having some experimental features under a flag is something that gives quicker feedback if the API, feature, or given logic makes sense for a wider user base. For majority of cases we always reject those because "it's support burden", "this does not help much for Prometheus only Cortex/Thanos/XYZ", "this overlap with older, less efficient API", "we don't like those extra dependencies (e.g gRPC)", "people will not use it". This might be off-topic here, but I feel like this is another thing which could improve the velocity of Prometheus and usability of the project itself (: 

Kind Regards,

Bartek

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200525195320.GA983523%40oxygen.

[Julius Volz]

On Tue, May 26, 2020 at 10:24 PM Julien Pivotto <roidel...@prometheus.io> wrote:

On 26 May 21:16, Bartłomiej Płotka wrote:
> Hi

>
> On a separate note, while I believe in maintaining high project quality for
> everything we ship, I think we should be way more open into the
> experimenting bit more in Prometheus. Having some experimental features
> under a flag is something that gives quicker feedback if the API, feature,
> or given logic makes sense for a wider user base. For majority of cases we
> always reject those because "it's support burden", "this does not help much
> for Prometheus only Cortex/Thanos/XYZ", "this overlap with older, less
> efficient API", "we don't like those extra dependencies (e.g gRPC)",
> "people will not use it". This might be off-topic here, but I feel like
> this is another thing which could improve the velocity of Prometheus and
> usability of the project itself (:

I agree with that statement. I think that we could and should not have
taboos; we should be able to rethink some decisions and not be afraid to
accept more new features.

Maybe we should also be less concerned about pointing our users in the
"correct" direction and let them do more with their data.

Coming tomorrow to Prometheus: logs, tracing, deep analytics, and push ;)

In seriousness though, I agree. Maybe not to go that far, but to be open and evolve a bit with the times. There is certainly value to Prometheus having a core scope and not growing into a solution that tries to be everything for everyone, but I also think we can still do more and still be Prometheus.

 

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAMssQwZ3apTKpmUGuTJRs0ipDF7TQcypPOiXh_uX9%2B%3D2zVrFqg%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[Julius Volz]

On Tue, May 26, 2020 at 10:17 PM Bartłomiej Płotka <bwpl...@gmail.com> wrote:

Hi

I support 100% of what Julius said: 

>  In general, +10 for a change in this direction, since currently, the balance is too much in favor of inaction - it's really easy to veto something, but really hard to unblock it because team-wide votes are high-overhead.

> I would also want to understand a bit more specifically how this would work in practice though. 

+1 on moving to a rough consensus as I think it's stepping into a good direction. 

Overall, I feel that our team is not a children's playground, we are team of professionals. This means that we trust each other, so no one in Prometheus will decide something totally against someone else's strong opposite vote.

If we don't want to go totally against someone's strong opposing vote, then I think we could just leave the governance as it is. I thought the point was to make it possible to go against someone's strong opposing vote, just as long as it's considered. And if they are still against something, they can still call for a vote, but now that burden would be on the minority blocker party.

[Julius Volz]

On Tue, May 26, 2020 at 10:57 PM Julien Pivotto <roidel...@prometheus.io> wrote:

On 26 May 22:55, Julius Volz wrote:
> On Tue, May 26, 2020 at 10:17 PM Bartłomiej Płotka <bwpl...@gmail.com>
> wrote:
>
> > Hi
> >
> > I support 100% of what Julius said:
> >

> > *>  In general, +10 for a change in this direction, since currently, the

> > balance is too much in favor of inaction - it's really easy to veto
> > something, but really hard to unblock it because team-wide votes are

> > high-overhead.*
> > *> I would also want to understand a bit more specifically how this would
> > work in practice though. *

> >
> > +1 on moving to a rough consensus as I think it's stepping into a good
> > direction.
> >
> > Overall, I feel that our team is not a children's playground, we are team
> > of professionals. This means that we trust each other, so no one in
> > Prometheus will decide something totally against someone else's strong
> > opposite vote.
> >
>
> If we don't want to go totally against someone's strong opposing vote, then
> I think we could just leave the governance as it is. I thought the point
> was to make it possible to go against someone's strong opposing vote, just
> as long as it's considered. And if they are still against something, they
> can still call for a vote, but now that burden would be on the minority
> blocker party.

Yes that is what I think too. Note that rough consensus means that
sometimes the minority wins too.

Even if there's already an outspoken (relative) majority against that minority? How would that be rough consensus then?

 

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxDKMgf%2BEmyWG3%3D1KMW_Z99-Ky8kmFo12VgmejKRS%2Bk4w%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[Bartłomiej Płotka]

> 

Overall, I feel that our team is not a children's playground, we are team of professionals. This means that we trust each other, so no one in Prometheus will decide something totally against someone else's strong opposite vote.

> If we don't want to go totally against someone's strong opposing vote, then I think we could just leave the governance as it is. I thought the point was to make it possible to go against someone's strong opposing vote, just as long as it's considered. And if they are still against something, they can still call for a vote, but now that burden would be on the minority blocker party.

I think I was misunderstood. It's opposite: It is exactly to make it possible to go against someone's strong opposite vote. But it does not mean that if anyone disagrees we immediately ignore one person's opinion. It's more about stronger motivation to find consensus between everyone as there is an open, legal path for the project to move forward despite one person being strongly opposed. 

Let's think about this in the opposite way: If we trust each other, why having a rough consensus would be a problem? We will definitely try to address our concerns and items asked by the opposed party, right? It's the same with open source. Our license (Apache 2) allows almost anything. Free distribution, modification, forking etc. This ensures that there is always an open path if there will be a major blocker or conflict etc. It does not mean that forks are often.

Kind Regards,

Bartek

[Julius Volz]

On Tue, May 26, 2020 at 11:22 PM Bartłomiej Płotka <bwpl...@gmail.com> wrote:

> 

Overall, I feel that our team is not a children's playground, we are team of professionals. This means that we trust each other, so no one in Prometheus will decide something totally against someone else's strong opposite vote.

> If we don't want to go totally against someone's strong opposing vote, then I think we could just leave the governance as it is. I thought the point was to make it possible to go against someone's strong opposing vote, just as long as it's considered. And if they are still against something, they can still call for a vote, but now that burden would be on the minority blocker party.

I think I was misunderstood. It's opposite: It is exactly to make it possible to go against someone's strong opposite vote. But it does not mean that if anyone disagrees we immediately ignore one person's opinion. It's more about stronger motivation to find consensus between everyone as there is an open, legal path for the project to move forward despite one person being strongly opposed. 

Yep, agreed.

[Julien Pivotto]

Rough consensus is NOT about being PRO's or CON's. It is about raisin
issues that must be considered.

Ideally we reach an agreement of everyone. That should be the majority.
But if someone says no, we must take their arguments into consideration
and resolve the issue. When it is addressed/considered, either everyone
agrees or the chair can then call a 'rough' consensus.


7. Five people for and one hundred people against might still be rough
consensus
https://tools.ietf.org/html/rfc7282

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxEVQAerZx40j1vdz%2Bm5RO21qbxDKj7D3wvC38EDKYCdA%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[Brian Brazil]

The doc Richi linked covers it in some depth. Rough concensus can cover both a minority "winning" over a majority, and a majority being in favour not being sufficient for rough concensus. It's all about if objections are sufficiently considered.

The doc is more aspirational/philosophical rather than a formal policy, so I think it doesn't work as a governance in those terms as it's kinda vague.

Brian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxEVQAerZx40j1vdz%2Bm5RO21qbxDKj7D3wvC38EDKYCdA%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Julien Pivotto]

I have to say that it will be fun to address 'the PromQL evaluation
model doesn't allow looking into the future.' kind of arguments. Unless
it falls under the "unappealing" kind of concerns and can be dismissed
by a real "rough" consensus call:

Conversely, the working group might convince the objector that the
concerns can be addressed, or that the choice is simply unappealing
(i.e., something the objector can "live with") and not a show-stopper.

> > https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxEVQAerZx40j1vdz%2Bm5RO21qbxDKj7D3wvC38EDKYCdA%40mail.gmail.com
> > <https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxEVQAerZx40j1vdz%2Bm5RO21qbxDKj7D3wvC38EDKYCdA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> > .
> >
>
>
> --
> Brian Brazil
> www.robustperception.io

>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLqyupF1B_jd2_RrGv%3DDncWMPEMP3q3MmJJ%3D4cdHH-%3DxaQ%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[Julien Pivotto]

On 26 May 22:55, Julius Volz wrote:

> On Tue, May 26, 2020 at 10:17 PM Bartłomiej Płotka <bwpl...@gmail.com>
> wrote:
>
> > Hi
> >
> > I support 100% of what Julius said:
> >

> > *> In general, +10 for a change in this direction, since currently, the

> > balance is too much in favor of inaction - it's really easy to veto
> > something, but really hard to unblock it because team-wide votes are

> > high-overhead.*
> > *> I would also want to understand a bit more specifically how this would
> > work in practice though. *

> >
> > +1 on moving to a rough consensus as I think it's stepping into a good
> > direction.
> >
> > Overall, I feel that our team is not a children's playground, we are team
> > of professionals. This means that we trust each other, so no one in
> > Prometheus will decide something totally against someone else's strong
> > opposite vote.
> >
>
> If we don't want to go totally against someone's strong opposing vote, then
> I think we could just leave the governance as it is. I thought the point
> was to make it possible to go against someone's strong opposing vote, just
> as long as it's considered. And if they are still against something, they
> can still call for a vote, but now that burden would be on the minority
> blocker party.

Yes that is what I think too. Note that rough consensus means that
sometimes the minority wins too.

>
>

[Julien Pivotto]

On 26 May 21:16, Bartłomiej Płotka wrote:
> Hi
>

> On a separate note, while I believe in maintaining high project quality for
> everything we ship, I think we should be way more open into the
> experimenting bit more in Prometheus. Having some experimental features
> under a flag is something that gives quicker feedback if the API, feature,
> or given logic makes sense for a wider user base. For majority of cases we
> always reject those because "it's support burden", "this does not help much
> for Prometheus only Cortex/Thanos/XYZ", "this overlap with older, less
> efficient API", "we don't like those extra dependencies (e.g gRPC)",
> "people will not use it". This might be off-topic here, but I feel like
> this is another thing which could improve the velocity of Prometheus and
> usability of the project itself (:

I agree with that statement. I think that we could and should not have
taboos; we should be able to rethink some decisions and not be afraid to
accept more new features.

Maybe we should also be less concerned about pointing our users in the
"correct" direction and let them do more with their data.

>

[Bjoern Rabenstein]

tl;dr: Oh my...

I think we have an interesting discussion here, and a lot of valid and
good points were brought up.

However, I have a concern with Richi's suggestions on a much more
fundamental level.

I (re-)read https://tools.ietf.org/html/rfc7282 very carefully, and
it's indeed a very insightful document. (If you haven't read it
already, I strongly recommend to do so.)

I do not see any substantial contradiction between CouchDB-style lazy
consensus and RFC7282, though. Please note that lazy consensus
describes the "happy path", pretty much what RFC7282 sees as the
prefered variety of consensus. Rough consensus, however, kicks in if
no "regular" consensus can be reached. It is dealing with a case where
lazy consensus has already disappeared. In all its essence, it's a
_replacement_ for voting.

Thus, if we wanted to introduce rough consensus in our governance, we
would have to use it instead of the majority vote, not the lazy
consensus. Replacing the lazy consensus with rough consensus would
actually be against the spirit of RFC7282 because RFC7282 still
prefers a "smooth"/"good" consensus over a rough consensus, but
Richi's proposed change explicitly states that rough consensus is the
"default decision making mechanism".

To analyze the premise of RFC7282 a bit more:

Two relevant properties of the IETF are:

1. They do not vote. (They don't want to, but they also cannot.)
2. They have chairs.

Obviously, RFC7282 expresses a lot of reasons why votes are
problematic in general, but it also states that the IETF doesn't even
have the formal requirements for votes as their is no definition of
voting right.

For Prometheus, the situation is pretty much the oppsite:

1. We do vote. (We don't want to, but we can if need be.)
2. We have no chairs.

I want to emphasize that we do not _like_ votes. They are a last
resort. That's clearly stated in the governance. We probably all agree
with the problems of voting described in RFC7282, but as long as votes
don't happen often, those problems are not very relevant. As also
stated in RFC7282, the IETF has to revert to rough consensus quite
often. If we had to revert to votes very often, I'd agree that we have
a problem. The IETF has work groups with hundreds of people (and many
work groups), while the whole Prometheus team consists of 20
people. The much larger organization size makes it more likely that no
"smooth" consensus can be found. On the other hand, the larger size
goes along with some hierarchical structures, so there are work groups
with chairs, and the existence of chairs enables rough consensus. With
voting as the last resort, they would devolve into constant votes with
all its problems. So rough consensus is the better way out for them.

I have to disagree with Richi that rough consensus would work without
chairs. On the contrary, I believe that qualified chairs are the most
critical part of making rough consensus work. (As said by others in
the thread already: Who would otherwise make the binding call when the
rough consensus is reached.)

Or in other words: Rough consensus would solve a problem that we don't
have with means that we don't have, either.

Having said all that, the mail thread so far shows that there is a
fair amount of dissatisfaction with our decision-making performance so
far. In general, I believe we need to understand and discuss the the
underlying problems better to find a good solution for it. However,
here is my speculation why rough consensus appears appealing to some
of us: Precisely because we all know and feel that frequent voting
would be a problem, we avoid votes, and because we do that, we
compromise on the quality of our consensus, e.g. we shy away from
decisions where a consensus seems hard to find, or some of us
"capitulate" (as fittingly described in RFC7282), or the notion of
"veto power" spreads, and probably more... If that's true, the
problems of voting would indirectly have an effect even if actual
votes are rare. In that case, rough consensus might be a solution, but
note what I pointed out above: It would replace the voting, not the
lazy consensus part AKA we would still first try to find a "smooth"
consenus before we go "rough". And yes, it would require chairs. In
case of the small size of the Prometheus team, one chair would
probably be enough. Prometheus-team would then be the voting body to
elect the chair. But here is the point where I stop my speculation.

Summary of my points (all meant as IMHO, even if stated as facts here):

- The changes as suggested in
https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491
are _not_ following the spirit of RFC7282.

- To truly adopt rough consensus, we need to introduce a chair and get
rid of decision finding by voting (with change of governance,
admitting and removing team members, and electing the chair as
notable exceptions). But that would be a very invasive change of the
governance.

- Before we do that, I would prefer to not start a discussion with a
possible solution, but with the problem we are trying to solve. I
think we first have to understand the apparent or actual problems in
decision making much better. Then we are in much better shape to
find a solution.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Julien Pivotto]

On 27 May 00:54, Bjoern Rabenstein wrote:
> tl;dr: Oh my...
>
> I think we have an interesting discussion here, and a lot of valid and
> good points were brought up.
>
> However, I have a concern with Richi's suggestions on a much more
> fundamental level.
>
> I (re-)read https://tools.ietf.org/html/rfc7282 very carefully, and
> it's indeed a very insightful document. (If you haven't read it
> already, I strongly recommend to do so.)
>
> I do not see any substantial contradiction between CouchDB-style lazy
> consensus and RFC7282, though. Please note that lazy consensus
> describes the "happy path", pretty much what RFC7282 sees as the
> prefered variety of consensus. Rough consensus, however, kicks in if
> no "regular" consensus can be reached. It is dealing with a case where
> lazy consensus has already disappeared. In all its essence, it's a
> _replacement_ for voting.
>
> Thus, if we wanted to introduce rough consensus in our governance, we
> would have to use it instead of the majority vote, not the lazy
> consensus. Replacing the lazy consensus with rough consensus would
> actually be against the spirit of RFC7282 because RFC7282 still
> prefers a "smooth"/"good" consensus over a rough consensus, but
> Richi's proposed change explicitly states that rough consensus is the
> "default decision making mechanism".

I think that the proposal is more to default to "rfc7282" not "rough
consensus".

> To analyze the premise of RFC7282 a bit more:
>
> Two relevant properties of the IETF are:
>
> 1. They do not vote. (They don't want to, but they also cannot.)
> 2. They have chairs.
>
> Obviously, RFC7282 expresses a lot of reasons why votes are
> problematic in general, but it also states that the IETF doesn't even
> have the formal requirements for votes as their is no definition of
> voting right.
>
> For Prometheus, the situation is pretty much the oppsite:
>
> 1. We do vote. (We don't want to, but we can if need be.)
> 2. We have no chairs.

Quoting:

Although most of the examples in this document talk about working group
chairs, these principles apply to any person who is trying to lead a
group to rough consensus, whether a chair, a design team leader, a
document editor, an area director, or any community member who is
facilitating a discussion or trying to assess consensus.

I think that could then e.g. fallback to MAINTAINERS.md.

> I want to emphasize that we do not _like_ votes. They are a last
> resort. That's clearly stated in the governance. We probably all agree
> with the problems of voting described in RFC7282, but as long as votes
> don't happen often, those problems are not very relevant. As also
> stated in RFC7282, the IETF has to revert to rough consensus quite
> often. If we had to revert to votes very often, I'd agree that we have
> a problem. The IETF has work groups with hundreds of people (and many
> work groups), while the whole Prometheus team consists of 20
> people. The much larger organization size makes it more likely that no
> "smooth" consensus can be found. On the other hand, the larger size
> goes along with some hierarchical structures, so there are work groups
> with chairs, and the existence of chairs enables rough consensus. With
> voting as the last resort, they would devolve into constant votes with
> all its problems. So rough consensus is the better way out for them.

It is not easy to call a vote. When lazy consensus does not work and we
speak about niche feature or single line changes, it is embarrassing to
go in public and call a vote for "just" that.

> I have to disagree with Richi that rough consensus would work without
> chairs. On the contrary, I believe that qualified chairs are the most
> critical part of making rough consensus work. (As said by others in
> the thread already: Who would otherwise make the binding call when the
> rough consensus is reached.)

See my previous comment about MAINTAINERS.md.

> Or in other words: Rough consensus would solve a problem that we don't
> have with means that we don't have, either.
>
> Having said all that, the mail thread so far shows that there is a
> fair amount of dissatisfaction with our decision-making performance so
> far. In general, I believe we need to understand and discuss the the
> underlying problems better to find a good solution for it. However,
> here is my speculation why rough consensus appears appealing to some
> of us: Precisely because we all know and feel that frequent voting
> would be a problem, we avoid votes, and because we do that, we
> compromise on the quality of our consensus, e.g. we shy away from
> decisions where a consensus seems hard to find, or some of us
> "capitulate" (as fittingly described in RFC7282), or the notion of
> "veto power" spreads, and probably more... If that's true, the
> problems of voting would indirectly have an effect even if actual
> votes are rare. In that case, rough consensus might be a solution, but
> note what I pointed out above: It would replace the voting, not the
> lazy consensus part AKA we would still first try to find a "smooth"
> consenus before we go "rough". And yes, it would require chairs. In
> case of the small size of the Prometheus team, one chair would
> probably be enough. Prometheus-team would then be the voting body to
> elect the chair. But here is the point where I stop my speculation.

Yes that joins what I said just before. However it might be difficult
to give one person so much "power".

I think that we miss a way to achieve consensus _within the scope of a
PR/issue_. Voting on the developers mailing list to achieve consensus
would be strange if we would do it e.g. for each `Pmaybe` in the
Prometheus repository.

> Summary of my points (all meant as IMHO, even if stated as facts here):
>
> - The changes as suggested in
> https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491
> are _not_ following the spirit of RFC7282.
>
> - To truly adopt rough consensus, we need to introduce a chair and get
> rid of decision finding by voting (with change of governance,
> admitting and removing team members, and electing the chair as
> notable exceptions). But that would be a very invasive change of the
> governance.
>
> - Before we do that, I would prefer to not start a discussion with a
> possible solution, but with the problem we are trying to solve. I
> think we first have to understand the apparent or actual problems in
> decision making much better. Then we are in much better shape to
> find a solution.
>
> --
> Björn Rabenstein
> [PGP-ID] 0x851C3DA17D748D03
> [email] bjo...@rabenste.in
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200526225437.GR2326%40jahnn.

--
Julien Pivotto
@roidelapluie

[Bjoern Rabenstein]

On 27.05.20 01:10, Julien Pivotto wrote:
> On 27 May 00:54, Bjoern Rabenstein wrote:
> >
> > Thus, if we wanted to introduce rough consensus in our governance, we
> > would have to use it instead of the majority vote, not the lazy
> > consensus. Replacing the lazy consensus with rough consensus would
> > actually be against the spirit of RFC7282 because RFC7282 still
> > prefers a "smooth"/"good" consensus over a rough consensus, but
> > Richi's proposed change explicitly states that rough consensus is the
> > "default decision making mechanism".
>
> I think that the proposal is more to default to "rfc7282" not "rough
> consensus".

Yes, that would make sense, but it's different from what's written in
Richi's commit.

And it would still mean that votes about technical decisions wouldn't
happen anymore (so that that part of the governance needed to be
deleted, or in other words: still a way more invasive change than
suggested in the commit).

> Quoting:
>
> Although most of the examples in this document talk about working group
> chairs, these principles apply to any person who is trying to lead a
> group to rough consensus, whether a chair, a design team leader, a
> document editor, an area director, or any community member who is
> facilitating a discussion or trying to assess consensus.
>
> I think that could then e.g. fallback to MAINTAINERS.md.

One could indeed see Prometheus repos as the equivalent of work groups
and their maintainers as the chairs. However, several concerns:

- (minor) In case of multiple maintainers, we needed to pick the
"real" chair.

- (a bit more relevant) The whole prometheus-team is smaller than one
work group of the IETF. Do we really want to create the
fine-granular hierarchy of an organization with thousands of
participants for the 20 people in prometheus-team? (counterpoint:
discussions involve more than just prometheus-team, so we arguably
have hundreds of potential participants)

- (actually very relevant) Most of the controversial choking points in
PRs were happening because at least one side of the controversy
believed that the change has relevance for the whole project. In
which case we are back to square one: Who is actually the person
authorized to call the rough consensus? Similarly, we would still
not have a chair for all the decisions that are naturally not bound
to a particular repository.

> Yes that joins what I said just before. However it might be difficult
> to give one person so much "power".

At least the "parliament" (i.e. prometheus-team) could just elect
another chair if the current one goes wild.

> I think that we miss a way to achieve consensus _within the scope of a
> PR/issue_.

See the third of my concerns above. IIRC in the rare cases that the
decision of a maintainer of a repo was challenged, one of the
justifications was that the implications of the PR/issue in question
was believed to reach beyond just that repo.

[Stuart Clark]

On 27/05/2020 00:51, Bjoern Rabenstein wrote:
>
>> Yes that joins what I said just before. However it might be difficult
>> to give one person so much "power".
> At least the "parliament" (i.e. prometheus-team) could just elect
> another chair if the current one goes wild.

In a lot of organisations with chair based systems which work well the
opposite is often true.

The chair is there to facilitate the debate and therefore doesn't
express their own opinion. As a result, if the judgement call has to be
made (such as calling rough consensus) their is trust from all sides
that it is a reasonable decision not just following their personal belief.

(I've also seen and had similar advice for face-to-face meetings - it
works best if the chair/facilitator is not one of the active participants)

--
Stuart Clark

[Richard Hartmann]

On Tue, May 26, 2020 at 11:08 PM Julius Volz <juliu...@gmail.com> wrote:

> Even if there's already an outspoken (relative) majority against that minority? How would that be rough consensus then?

If 100 people wanted to support traces natively in Prometheus, it
would not need much of discussion to dismiss them.

It's relatively common on RIPE's address-WG that people are vocal
about IPv4/IPv6 and whatever they came up with after 20 whole minutes
of thought, and then they bring their friends.

Overall, it's a possible, but not very likely, that this would happen
within Prometheus, imo.

[Richard Hartmann]

On Wed, May 27, 2020 at 12:54 AM Bjoern Rabenstein <bjo...@rabenste.in> wrote:

> - The changes as suggested in
> https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491
> are _not_ following the spirit of RFC7282.

I disagree with the above, and with the longer-form text not quoted, as
* rough consensus obviously tries to go the happy path of full consensus first
* rough consensus as per the commit contains its mechanisms strictly
below the stronger mechanisms of majority and supermajority vote

> - To truly adopt rough consensus, we need to introduce a chair and get
> rid of decision finding by voting (with change of governance,
> admitting and removing team members, and electing the chair as
> notable exceptions). But that would be a very invasive change of the
> governance.

If "truly adopting rough consensus" was the proposal, I would agree.
That's _not_ the proposal, though.

The proposal is to improve the self-contained "consensus" bit while
keeping both voting mechanisms intact. As such, there's an easy way
out of any impassé.

The reason why this is the proposal is the imbalance we have between
the three decision making mechanisms:

* consensus:
* 100% agreement by team members who care to join the discussion
* not time-bound
* can contain an unlimited amount of explicit and implicit topics in
whatever freeform text and places
* no social pressure to voice an opinion

* majority vote:
* 50% agreement by team members who care to join the discussion
* time-bound
* can contain an explicit list of options on one single topic in a
specific place
* social pressure to voice an opinion

* supermajority vote:
* 66% agreement amongst all team members
* time-bound
* can contain an explicit list of options on one single topic in a
specific place
* I will track you down and make you vote (whichever way you prefer)
to make sure we reach a result one way or the other

Consensus should have the weakest, not the strongest, requirements for
passing within this system.

> - Before we do that, I would prefer to not start a discussion with a
> possible solution, but with the problem we are trying to solve. I
> think we first have to understand the apparent or actual problems in
> decision making much better. Then we are in much better shape to
> find a solution.

Personally, I feel as if I have a good grasp on the problem domain and
the thread seems to reflect that sentiment in others. This is not to
say that having a distinct wider discussion in parallel, or instead,
wouldn't be an option. Again, personally, I am content with the
proposal as is, but I am biased.


Richard

[Richard Hartmann]

On Wed, May 27, 2020 at 8:35 AM Stuart Clark <stuart...@jahingo.com> wrote:

> In a lot of organisations with chair based systems which work well the
> opposite is often true.
>
> The chair is there to facilitate the debate and therefore doesn't
> express their own opinion. As a result, if the judgement call has to be
> made (such as calling rough consensus) their is trust from all sides
> that it is a reasonable decision not just following their personal belief.
>
> (I've also seen and had similar advice for face-to-face meetings - it
> works best if the chair/facilitator is not one of the active participants)

This matches my experience 100%. From taking part in those groups,
from shadowing Chairs in the same office, to Chairing myself. It's
also how I hope people would agree the dev summits are structured.

In practice, our group is too small and too specialized to have much
in choice of external chairs so it means switching hats, but I believe
that there's trust in this group that this is happening properly. The
Call for Consensus during the dev summits is founded on all this.

Richard

[Bjoern Rabenstein]

On 27.05.20 16:13, Richard Hartmann wrote:
> On Wed, May 27, 2020 at 12:54 AM Bjoern Rabenstein <bjo...@rabenste.in> wrote:
>
> > - The changes as suggested in
> > https://github.com/prometheus/docs/commit/f7ee6ac5a9b841be6be25d92e2c403de15949491
> > are _not_ following the spirit of RFC7282.
>
> I disagree with the above, and with the longer-form text not quoted, as
> * rough consensus obviously tries to go the happy path of full consensus first

Weird. My reading is that "obviously" rough consensus is what you do
if you cannot reach "smooth" consensus but want to avoid voting.

> * rough consensus as per the commit contains its mechanisms strictly
> below the stronger mechanisms of majority and supermajority vote

So you are saying that, if you say "rough consensus", you mean a wider
concept around the idea of rough consensus, as described in RFC7282,
and not just rough consensus itself as a step during decision making.

> > - To truly adopt rough consensus, we need to introduce a chair and get
> > rid of decision finding by voting (with change of governance,
> > admitting and removing team members, and electing the chair as
> > notable exceptions). But that would be a very invasive change of the
> > governance.
>
> If "truly adopting rough consensus" was the proposal, I would agree.
> That's _not_ the proposal, though.

OK, but then the proposal is really difficult to understand. You don't
want to adopt RFC7282 completely, but you also say that "rough
consensus" is a whole method or procedure and more than just one step
during decision making.

I'd say we need a proper definition then and not just a link to
RFC7282, of which some parts are relevant and some not, and the reader
has to guess which are which.

> The proposal is to improve the self-contained "consensus" bit while
> keeping both voting mechanisms intact. As such, there's an easy way
> out of any impassé.

If you want to give a clearer definition of "consensus", fine. So far,
things have just become less clear to me.

> * consensus:
> * 100% agreement by team members who care to join the discussion

That's not how I understand consensus. As I have learned much earlier,
consensus may include many flavors of disagreement. "Disagree but
commit" is perhaps the strongest (and arguably a borderline case). The
excellent RFC7282 describes many other "good" varieties of consensus
(but that's not the first time I read about them).
Example quotes:

"Consensus doesn't require that everyone is happy and agrees that the
chosen solution is the best one. Consensus is when everyone is
sufficiently satisfied with the chosen solution, such that they no
longer have specific objections to it."

"Coming to consensus is when everyone (including the person making the
objection) comes to the conclusion that either the objections are
valid, and therefore make a change to address the objection, or that
the objection was not really a matter of importance, but merely a
matter of taste."

It also describes varieties of "not really consensus": "People might
very well agree that a solution is sufficient and have no objection to
it, but if they also don't actively think it's a good and correct
outcome, it's absurd to declare that the group has consensus." Or the
"capitulation" antipattern, when one party just has "simply given up
by trying to appease the others". or the "horse-trading" variety.

Note that none of the consensus examples above, neither the proper
ones nor the false ones, are called "rough consensus" in the RFC: "Of
course, coming to full consensus like that does not always happen.
That's why in the IETF, we talk about 'rough consensus'."

(Strictly speaking, you could argue that "rough consensus" is the
superset of the happy case and the rough case, but it's still very
confusing if the governance says "our default way of deciding is rough
consensus". It should be "our default way of deciding is by consensus,
if need be by a rough one".)

Circling back: If all you want is to define "consensus" better, then
let's just do it. I can see how the "as long as nobody objects" in the
governance can be read as including some of the "not really consensus"
cases from RFC7282, but as excluding some of the "proper consensus"
cases.

Here is a rough suggestion how a minimal governance change could look
like:
https://github.com/prometheus/docs/commit/66dcf05d9d8fa246ef7c557b4a1978c718455fd2

_However,_ I have no high hopes that such a governance change will on
its own change anything in our decision making. For one, I'm confident
that most of us were alredy aware that consensus does not equal "100%
agreement by team members". Furthermore, as has been said multiple
times by now, most of the difficult decision making had its root cause
in precisely the problem that not all participants saw their
objections addressed. If we cannot agree that all objections have been
addressed, and we have no chair that has the power to make that
assessment, there will still be no decision. The only situation that
would change with adopting "chair-free rough consensus" is the one
where everyone involved agrees that all their objections have been
addressed, but they still want them to also be accommodated, and
that's why they still continue to block the consensus. I would be
surprised if we ever had that situation or ever will have.

I wouldn't oppose a governance change as the one I have sketched out
above, but I don't think it would help beyond emphasizing a spirit
that we were already following anyway.

[Vishwajeet Singh]

  Hi Team,

We are running two prometheus servers with parallel insertions to achieve HA. I want to backfill the prometheus data in case my one node goes down. Can u pls help me how to achieve this? Please let me know with any custom golang code or any other way.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgRvwnU%2BdFRMQ2n%2Bqf6D0oyej-btzsQYh8LjBV9FJ1a78w%40mail.gmail.com.

[Richard Hartmann]

The current situation is what I want to avoid with my suggestion.

Short-circuiting discussions directly into votes, potentially after mere hours, is not healthy for the project long term and likely reflects frustrations built up over time.

Voting results are also harder to adapt and evolve as they will need new votes, not just consensus, to change. This might not be a problem in the two current well-scoped votes at the moment, but it will become one more and more over time.

While I can understand the underlying motivations for quick voting on ALL the things I would much prefer to fix the default mechanism, consensus, over moving to a different one, voting ,as the new default.

[Stuart Clark]

On 2020-05-28 23:36, Richard Hartmann wrote:
> The current situation is what I want to avoid with my suggestion.
>
> Short-circuiting discussions directly into votes, potentially after
> mere hours, is not healthy for the project long term and likely
> reflects frustrations built up over time.
>
> Voting results are also harder to adapt and evolve as they will need
> new votes, not just consensus, to change. This might not be a problem
> in the two current well-scoped votes at the moment, but it will become
> one more and more over time.
>
> While I can understand the underlying motivations for quick voting on
> ALL the things I would much prefer to fix the default mechanism,
> consensus, over moving to a different one, voting ,as the new default.
>

That sounds like a different issue to changing the definition of
consensus.

Maybe the voting process should have a third option which isn't a yes or
a no, but a "go back to discussion"?

Or you have some criteria (time, etc) which is needed to call a vote
(although that can also bring difficulties).

--
Stuart Clark

[Richard Hartmann]

On Fri, May 29, 2020 at 10:28 AM Stuart Clark <stuart...@jahingo.com> wrote:

> That sounds like a different issue to changing the definition of
> consensus.

How so?

> Maybe the voting process should have a third option which isn't a yes or
> a no, but a "go back to discussion"?

Debian's FD default option ("Further Discussion") in Condorcet votes
is a "no" and a "if people feel strongly, they will try again" at the
same time. I can see the difference in framing, but see little to no
practical difference.

> Or you have some criteria (time, etc) which is needed to call a vote
> (although that can also bring difficulties).

That's too close to a technical solution to a social problem for me;
and why I decided to not wait a day or so voting in Beorn's thread.


Richard

[Richard Hartmann]

https://grafana.com/blog/2020/04/16/community-series-on-asking-good-questions/