Issue triage again - labels

[Tim Hockin]

I wrote a little about this a few weeks ago and got some good pointers
to how people are operating today. Thanks.

The problem I am facing in sig-net is a little different from what
most people described. Specifically I have 250+ issues, some years
old and some days old, which need to be looked at by a human. I want
to scrub EVERY SINGLE ONE.

This raises some tactical issues. I raised it on slack, but this is a
better forum.

Preface: This is not about assigning sig/* labels. This is about
following up on the issues that are assigned sig/network. Making sure
they are real, have enough information, and are actionable.

As we march through these issues, we'll get some looked at every week.
It will take months. During this time, new issues will arrive. Some
issues will be easy to handle - obviously a bug, obviously a feature
request, obviously support (close it). Some will need more
information. Some will need to be reproduced to confirm.

How do I, iteratively, week-on-week, keep track of which issues are
acknowledged and which issues are in progress and which issues are yet
to be done? Assignee is not right - I am not asking people to FIX the
issues, just to investigate. And many people are not org members
anyway, so can not be assigned.

I am proposing that we do something along the lines of adding a new
label, e.g. "needs-sig-ack", which is automatically applied to ALL new
issues.

Now, I can search for sig/network && needs-sig-ack, and chip away at
that list. I can get volunteers to followup on issues and either
close them, ask for more info, or ACK them (removing that label). We
can also use that moment to look at kind/* and area/* labels.

We have a handful of triage/* labels, but they mostly don't make sense
to me, at least for non-closed issues. As historical justification
for closing they make sense, but seem like the wrong mechanism (they
are retroactively mutable). Not sure what to do with those. The
meaning of "triage" there doesn't match what I think of as bug
triage.

Eventually we will have scrubbed them all and the needs-sig-ack label
will represent new issues only.

Aaron correctly said "OMG another label, ugh". That's why I am here
to discuss. Some issues are assigned multiple SIGs and some are
legitimately cross-SIG. I think that's fine - if one SIG acknowledges
an issue, that should be good enough.

Discuss...

[Tim Hockin]

I got bounced from contribex. Resending.

[Tim St. Clair]

inline 

We treat un-labeled issues as needing triage.  Given that SCL repos are federated it's much easier then k/k.  But we could defaultly apply a need-triage(or needs-sig-ack) label to indicate that no one from the sig has triaged the issue. 

 

Now, I can search for sig/network && needs-sig-ack, and chip away at
that list.  I can get volunteers to followup on issues and either
close them, ask for more info, or ACK them (removing that label).  We
can also use that moment to look at kind/* and area/* labels.

We have a handful of triage/* labels, but they mostly don't make sense
to me, at least for non-closed issues.  As historical justification
for closing they make sense, but seem like the wrong mechanism (they
are retroactively mutable).  Not sure what to do with those.  The
meaning of "triage" there doesn't match what I think of as bug
triage.

Eventually we will have scrubbed them all and the needs-sig-ack label
will represent new issues only.

 

Aaron correctly said "OMG another label, ugh".  That's why I am here
to discuss.  Some issues are assigned multiple SIGs and some are
legitimately cross-SIG.  I think that's fine - if one SIG acknowledges
an issue, that should be good enough.

I think creating a simple flowchart on your process model and nuke un-used labels makes a ton of sense IMO. 

 

Discuss...

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-leads" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-l...@googlegroups.com.
To post to this group, send email to kubernetes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-leads/CAO_RewZnWAftdKGz99UjQfTYvv7GMgzc8EZFx-pQPMdyMe68eg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,
Timothy St. Clair

“Do all the good you can. By all the means you can. In all the ways you can. In all the places you can. At all the times you can. To all the people you can. As long as ever you can.” 

[Tim Pepper]

I’m not sure one sig ack’ing an issue is enough.  As I think about the state diagram it feels like doing this well is going to mean removing a triaged (past-tense) label upon removal of a sig label and upon addition of a sig label, or…it drives at per-sig triaged labels.

 

Often a first sig label is applied (eg: sig cluster lifecycle…”upgrade failed”), initial triage is done, and it’s on to next sig(s) and back in need of their triage.

 

-- 

Tim Pepper

Orchestration & Containers Lead

VMware Open Source Technology Center

[Tim Hockin]

Aside how fun to have a thread between Tim, Tim, and Tim. :)

On Fri, Mar 8, 2019 at 10:37 AM Tim Pepper <tpe...@vmware.com> wrote:
>
> I’m not sure one sig ack’ing an issue is enough. As I think about the state diagram it feels like doing this well is going to mean removing a triaged (past-tense) label upon removal of a sig label and upon addition of a sig label, or…it drives at per-sig triaged labels.

I disagree. An ACK means a human has looked at it and confirmed it's
a real thing. If I have an issue that crosses SIGs, I will consult
with those SIGs before ACKing. But even if I unilaterally ACk that
should be enough -- I am asserting with confidence that it is real.
It may need more input from another SIG, but it's a real thing.

That said, I don't care if we had per-sig ACKs, but that seems like
process for its own sake. I don't think it's needed.

> Often a first sig label is applied (eg: sig cluster lifecycle…”upgrade failed”), initial triage is done, and it’s on to next sig(s) and back in need of their triage.

Yeah, I would not ACK it if it is not legitimately my SIG's issue.

Tim

[Tim Allclair]

I just wanted to weigh in because ... we've recently been discussing how to better handle issue triage in SIG-Auth. We're not looking at anything too advanced right now, but plan on adding a triage section to our bi-weekly meetings, and ensure that all issues created since the previous meeting have been triaged.

-- Tim

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-leads" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-l...@googlegroups.com.
To post to this group, send email to kubernetes...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-leads/CAO_RewadYGxK_Y1czp06NjNz7%2B_V0fjQsiwWt68gTM6g_703gA%40mail.gmail.com.

[Tim Hockin]

Four of a kind! We win the game!

[Sahdev Zala]

LOL, very cool. Tim is all over in the news too :-) 

I like the idea as mentioned on the slack but trying to understand more:

So currently we have a `needs-sig` label that applies automatically to new issues. Once a `sig/ *` label is applied, the `needs-sig` goes away. Can we use the same label as such or by renaming it with `needs-sig-ack`, and, instead of removing it automatically, keep it until at least one sig remove it manually?  (so that we don’t create a new label, and keep it to one step instead of two steps). 

Secondly, once the proposed `needs-sig-ack` removed manually by a SIG, an issue will look like what it looks like today with one or more sig labels right? Shouldn’t the removal of `needs-sig-ack` label reflect the acknowledged ownership of a particular SIG in the issue? So that a SIG can easily search those issues later on to work on their true backlog? (I assume ack doesn’t mean the SIG will work on the issue right away). 

Thanks! 

Regards,

Sahdev Zala

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-leads/CAO_Rewb4Ue%2Bp5CHMv3cPNZMucmJuLbVZz1JFfO%2BFVPjVi6YyjQ%40mail.gmail.com.

[Tim Hockin]

On Fri, Mar 8, 2019 at 11:57 AM Sahdev Zala <sahdev...@gmail.com> wrote:
>
> LOL, very cool. Tim is all over in the news too :-)
>
>
> I like the idea as mentioned on the slack but trying to understand more:
>
>
> So currently we have a `needs-sig` label that applies automatically to new issues. Once a `sig/ *` label is applied, the `needs-sig` goes away. Can we use the same label as such or by renaming it with `needs-sig-ack`, and, instead of removing it automatically, keep it until at least one sig remove it manually? (so that we don’t create a new label, and keep it to one step instead of two steps).

In theory, yes. I think it's a little less clear, but I would roll
with it if that is what contribex thought best. Personally I like the
needs-* concept as single-purpose instructions. Needs a kind, needs a
sig, needs an ACK...

> Secondly, once the proposed `needs-sig-ack` removed manually by a SIG, an issue will look like what it looks like today with one or more sig labels right? Shouldn’t the removal of `needs-sig-ack` label reflect the acknowledged ownership of a particular SIG in the issue? So that a SIG can easily search those issues later on to work on their true backlog? (I assume ack doesn’t mean the SIG will work on the issue right away).

In my mind ACK means "we agree this is a valid bug report or RFE or
whatever". That includes "we did a repro and confirmed the bug" and
"this idea is not fundamentally impossible". It does not imply any
priority or timeline -- we have other labels for those things.

[Stephen Augustus]

I'm trying to look at it from two lenses: the SIG and the submitter/a passerby.

There's a difference between "Has someone even seen this?" and "Is this a valid issue/RFE?".

From the perspective of the user, I care about if something is even getting attention.

A bot could drop by and say, "Hey. Thanks for submitting. Your foo is pending triage/attention from a Kubernetes SIG member" or some such --> apply the label

Again from the perspective of the user, I'd want some sort of acknowledgement that something is happening behind the scenes.

"Hey! We got your foo and we're going to take a look" --> soft prio the issue, close if it's clear it's support, ensure SIG, milestone, priority labels are attached, assign if it's clear who it belongs to, else add help-wanted, and optionally good-first-issue, and remove the needs-{triage,attention,ack} label

(This could be made more accessible if there was a triage dashboard a la gubernator.)

Now the user knows there's some sort of movement.

From there, now that the issue/RFE is labeled appropriately, the SIG can jump in and give a deeper inspection, reprio as necessary, move to a KEP, close outright apply some process like the SIG Cluster Lifecycle grooming guidance: https://github.com/kubernetes/community/blob/master/sig-cluster-lifecycle/grooming.md

That first soft prio could potentially be done by any contributor with just enough context to validate and route to the appropriate SIG/people, which alleviate the burden on some of the top-level reviewer/approvers.

As far as grooming goes, I'm planning to make that the first part of the SIG Azure, Release, and PM meetings moving forward.

-- Stephen

You received this message because you are subscribed to the Google Groups "kubernetes-sig-contribex" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-con...@googlegroups.com.
To post to this group, send email to kubernetes-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-contribex/CAO_RewafjQ4JoNUT1aazhExr2VbEcf_NaU_uarRhSXhmURVrbg%40mail.gmail.com.

[Sahdev Zala]

On Fri, Mar 8, 2019 at 3:04 PM 'Tim Hockin' via kubernetes-sig-contribex <kubernetes-s...@googlegroups.com> wrote:

On Fri, Mar 8, 2019 at 11:57 AM Sahdev Zala <sahdev...@gmail.com> wrote:
>
> LOL, very cool. Tim is all over in the news too :-)
>
>
> I like the idea as mentioned on the slack but trying to understand more:
>
>
> So currently we have a `needs-sig` label that applies automatically to new issues. Once a `sig/ *` label is applied, the `needs-sig` goes away. Can we use the same label as such or by renaming it with `needs-sig-ack`, and, instead of removing it automatically, keep it until at least one sig remove it manually?  (so that we don’t create a new label, and keep it to one step instead of two steps).

In theory, yes.  I think it's a little less clear, but I would roll
with it if that is what contribex thought best.  Personally I like the
needs-* concept as single-purpose instructions.  Needs a kind, needs a
sig, needs an ACK...

Yes, I did think that it is less clear but thought it’s worth to think around it and see if we can avoid creating a new label.  I do not have a strong opinion and I am good with what we end up deciding.  

> Secondly, once the proposed `needs-sig-ack` removed manually by a SIG, an issue will look like what it looks like today with one or more sig labels right? Shouldn’t the removal of `needs-sig-ack` label reflect the acknowledged ownership of a particular SIG in the issue? So that a SIG can easily search those issues later on to work on their true backlog? (I assume ack doesn’t mean the SIG will work on the issue right away).

In my mind ACK means "we agree this is a valid bug report or RFE or
whatever".  That includes "we did a repro and confirmed the bug" and
"this idea is not fundamentally impossible".  It does not imply any
priority or timeline -- we have other labels for those things.

 OK, as long as ACK clearly gives ownership of an issue to a particular SIG which is easily searchable, that's good. 

 

You received this message because you are subscribed to the Google Groups "kubernetes-sig-contribex" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-con...@googlegroups.com.
To post to this group, send email to kubernetes-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-contribex/CAO_RewafjQ4JoNUT1aazhExr2VbEcf_NaU_uarRhSXhmURVrbg%40mail.gmail.com.

[Tim Hockin]

On Fri, Mar 8, 2019 at 12:34 PM Stephen Augustus <Ste...@agst.us> wrote:
>
> I'm trying to look at it from two lenses: the SIG and the submitter/a passerby.
> There's a difference between "Has someone even seen this?" and "Is this a valid issue/RFE?".
>
> From the perspective of the user, I care about if something is even getting attention.
> A bot could drop by and say, "Hey. Thanks for submitting. Your foo is pending triage/attention from a Kubernetes SIG member" or some such --> apply the label
>
> Again from the perspective of the user, I'd want some sort of acknowledgement that something is happening behind the scenes.
> "Hey! We got your foo and we're going to take a look" --> soft prio the issue, close if it's clear it's support, ensure SIG, milestone, priority labels are attached, assign if it's clear who it belongs to, else add help-wanted, and optionally good-first-issue, and remove the needs-{triage,attention,ack} label
> (This could be made more accessible if there was a triage dashboard a la gubernator.)

I don't think someone outside the sig has the ability to assess prio
properly in many cases, nor the good-first-issue-ness. This is not
"triage" this is routing. I very much value the work of routing and
getting issues to (possibly) relevant SIGs, but issue filers should
not interpret that as anything other than "it's in the hopper".

Users can kind of see that today with the change from needs-sig to
sig/foo right?

> Now the user knows there's some sort of movement.
>
> From there, now that the issue/RFE is labeled appropriately, the SIG can jump in and give a deeper inspection, reprio as necessary, move to a KEP, close outright apply some process like the SIG Cluster Lifecycle grooming guidance: https://github.com/kubernetes/community/blob/master/sig-cluster-lifecycle/grooming.md
>
> That first soft prio could potentially be done by any contributor with just enough context to validate and route to the appropriate SIG/people, which alleviate the burden on some of the top-level reviewer/approvers.

Yep - I just need some way to track whether that has or has not been
done yet without reading every PR. Github labels are approximately
the only mechanism we have, right?

You're fleshing out my hypothesis for me. A state machine for this
EXISTS. We even follow it most of the time. We just have never
actually written it down. We should actually draw a state diagram for
it and indicate at which states various categories of labels are
applied and what the meaning is.

[Tim Hockin]

On Fri, Mar 8, 2019 at 12:34 PM Sahdev Zala <sahdev...@gmail.com> wrote:
>
> On Fri, Mar 8, 2019 at 3:04 PM 'Tim Hockin' via kubernetes-sig-contribex <kubernetes-s...@googlegroups.com> wrote:
>>
>> On Fri, Mar 8, 2019 at 11:57 AM Sahdev Zala <sahdev...@gmail.com> wrote:
>> >
>> > LOL, very cool. Tim is all over in the news too :-)
>> >
>> >
>> > I like the idea as mentioned on the slack but trying to understand more:
>> >
>> >
>> > So currently we have a `needs-sig` label that applies automatically to new issues. Once a `sig/ *` label is applied, the `needs-sig` goes away. Can we use the same label as such or by renaming it with `needs-sig-ack`, and, instead of removing it automatically, keep it until at least one sig remove it manually? (so that we don’t create a new label, and keep it to one step instead of two steps).
>>
>> In theory, yes. I think it's a little less clear, but I would roll
>> with it if that is what contribex thought best. Personally I like the
>> needs-* concept as single-purpose instructions. Needs a kind, needs a
>> sig, needs an ACK...
>
>
> Yes, I did think that it is less clear but thought it’s worth to think around it and see if we can avoid creating a new label. I do not have a strong opinion and I am good with what we end up deciding.
>>
>>
>> > Secondly, once the proposed `needs-sig-ack` removed manually by a SIG, an issue will look like what it looks like today with one or more sig labels right? Shouldn’t the removal of `needs-sig-ack` label reflect the acknowledged ownership of a particular SIG in the issue? So that a SIG can easily search those issues later on to work on their true backlog? (I assume ack doesn’t mean the SIG will work on the issue right away).
>>
>> In my mind ACK means "we agree this is a valid bug report or RFE or
>> whatever". That includes "we did a repro and confirmed the bug" and
>> "this idea is not fundamentally impossible". It does not imply any
>> priority or timeline -- we have other labels for those things.
>
> OK, as long as ACK clearly gives ownership of an issue to a particular SIG which is easily searchable, that's good.

I think the cases where 2 SIGs are tagged and one ACKs and the other
disagrees are going to be vanishingly few. So I am not super worried
about it. I predict 99.9% of cases will be one SIG label and no
problems :)

[Tim Hockin]

Ping. A week later - any objections?

Anyone who knows theexisting system have time to collab on a more
thorough design?

[Tim St. Clair]

No objections, but maybe a requirement.  

If we add some new labels, can we get rid of some old ones? 

Cheers,

Tim 

[Daniel Smith]

I haven't completely read this thread, but I'd love a standardized triage system, one that works for PRs as well as issues.

We triage new issues and PRs by looking at things tagged SIG API Machinery and are greater than the last number we stopped at. This is better than nothing but we miss PRs/issues where the sig label is added after we have already passed that number.

If we had a normalized process, we could actually e.g. have the SIG allocate a triage role or roles and do a much better job here.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-contribex/CAO_Rewb6D%2Be7fF754A6CS8sRp99oM25k2Z1zV6T3dJXfRdyiAw%40mail.gmail.com.