RFC: Add a "needinfo" state to triaging
Daniel Moisset
while working on the sprint today doing triaging we noticed that a
lot of tickets were in the "Unreviewed" state because actually there's
not enough information to move it to any other state (they can not be
neither accepted/DDNd nor closed). In most cases we sent a reply back
to the submitter asking for more details about their problem, but the
ticket remains in the "Unreviewed" state, still taking the time of
other triagers looking for tickets to review.
Many projects have a "need information" state to report this
situation. I think adding this to the ticket state diagram [1] would
be helpful for triagers, and to developers reviewing the ticket list
and wanting to know how many tickets actually need to be looked at
before a release.
What are the thoughts of the core team on this?
Best regards,
D.
[1] http://docs.djangoproject.com/en/dev/internals/contributing/#ticket-triage
SmileyChris
> In most cases we sent a reply back
> to the submitter asking for more details about their problem, but the
> ticket remains in the "Unreviewed" state, still taking the time of
> other triagers looking for tickets to review.
>
> Many projects have a "need information" state to report this
> situation. I think adding this to the ticket state diagram [1] would
> be helpful for triagers, and to developers reviewing the ticket list
> and wanting to know how many tickets actually need to be looked at
> before a release.
>
> What are the thoughts of the core team on this?
triaging:
Have a "needsmoreinfo" resolution. Tickets can be closed with that
with a note to reopen it when more info is provided.
Daniel Moisset
>> What are the thoughts of the core team on this?
>
> Here's another more forward solution, requiring less follow-up
> triaging:
> Have a "needsmoreinfo" resolution. Tickets can be closed with that
> with a note to reopen it when more info is provided.
>
Sounds even better
D.
Iván Raskovsky
> triaging:
> Have a "needsmoreinfo" resolution. Tickets can be closed with that
> with a note to reopen it when more info is provided.
+1
That would be great! It would be much easier to go through the unreviewed list.
Iván
Gabriel Hurley
I look at the Django timeline regularly, but I have ticket detail
changes filtered out. Therefore I *don't* see changes like
"unreviewed" -> "accepted", etc. I do, however, see ticket status
changes (closing, reopening, etc.). Thereby these kinds of changes
will be more visible, especially when a ticket is re-opened,
presumably with more information.
I do have one concern, though... I worry that it might be off-putting
to new contributors who may not provide enough information if their
ticket is *closed* with needsmoreinfo rather than commented on with
"needs more info".
All in all, I like it.
- Gabriel
Andrew Godwin
> Hi,
> while working on the sprint today doing triaging we noticed that a
> lot of tickets were in the "Unreviewed" state because actually there's
> not enough information to move it to any other state (they can not be
> neither accepted/DDNd nor closed). In most cases we sent a reply back
> to the submitter asking for more details about their problem, but the
> ticket remains in the "Unreviewed" state, still taking the time of
> other triagers looking for tickets to review.
>
> Many projects have a "need information" state to report this
> situation. I think adding this to the ticket state diagram [1] would
> be helpful for triagers, and to developers reviewing the ticket list
> and wanting to know how many tickets actually need to be looked at
> before a release.
>
> What are the thoughts of the core team on this?
We have this on the South trac, and it helps me greatly in filtering
views to just tickets that need my attention/responses instead of just
getting confronted with "all open tickets". You can also set it up so
the default action once you're in infoneeded is to move out of it, so it
shouldn't even cause more work for responders as they have to change the
state back - it just does.
I'm +1 on adding this, but that's definitely biased by the fact that it
makes the two Tracs I use on a regular basis more similar.
Andrew
Russell Keith-Magee
To be clear, Andrew -- are you +1 to a "Needs Info" closing condition,
or a "Needs info" ticket state?
For me, it all hinges on whether a you consider that a reported issue
should be open or closed by default. I tend towards "closed" by
default, simply because our Trac instance is already a graveyard of
old tickets. The history of Django's Trac instance doesn't suggest
that an incomplete bug report will be updated after initial reporting;
unless the reporter is a known member of the community, if a ticket
isn't complete at time of reporting, it isn't going to get any better.
To that end, I'm not sure what a Needs Info ticket state gains us. If
someone opens a ticket that doesn't contain enough useful debugging
information, we should be closing the ticket, not keeping it open for
eternity in the hope that they'll come back and provide the detail
they should have provided in the first place.
If the reporter *is* inclined to provide more information, a "needs
info" close state gives them an indication that the ball is in their
court, differentiating between closed "Invalid" (i.e., You're wrong)
and closed "WorksForMe" (i.e., can't reproduce), while keeping the
ticket out of our working list until such time as they provide the
missing details.
So; I'm +0 to a close state, -0 to a ticket state. I'm +0 because this
is already possible right now -- close "invalid" with a comment that
indicates the ticket should be reopened if the reporter can provide
more details. However, I can see that there might be a slight
advantage to communicating that with something other than a comment.
Yours,
Russ Magee %-)
Tai Lee
to the reporter that they need to take the next step to re-open the
ticket with more info. I don't think that closed with "invalid" and a
comment makes this as clear.
However, I think there's another problem area where we need to be able
to make it clear that someone needs to take the next step, and that is
when a ticket has been "accepted", feedback has been given to the
reporter, and the reporter has actioned that feedback (e.g. by
uploading a patch with tests and docs as per the feedback). In this
case the ticket will often languish in "accepted" for months (or
years). Since it is frowned upon for a reporter to mark their own
ticket as RFC, there's no way for the reporter to flag the ticket as
feedback actioned and put it back in the court of the original triager
or core developer who accepted it and gave their feedback, who can
then either push it up to RFC or commit it themselves.
Cheers.
Tai.
On Nov 15, 12:35 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:
Russell Keith-Magee
> I like the idea of needmoreinfo as a resolution, which makes it clear
> to the reporter that they need to take the next step to re-open the
> ticket with more info. I don't think that closed with "invalid" and a
> comment makes this as clear.
>
> However, I think there's another problem area where we need to be able
> to make it clear that someone needs to take the next step, and that is
> when a ticket has been "accepted", feedback has been given to the
> reporter, and the reporter has actioned that feedback (e.g. by
> uploading a patch with tests and docs as per the feedback). In this
> case the ticket will often languish in "accepted" for months (or
> years). Since it is frowned upon for a reporter to mark their own
> ticket as RFC, there's no way for the reporter to flag the ticket as
> feedback actioned and put it back in the court of the original triager
> or core developer who accepted it and gave their feedback, who can
> then either push it up to RFC or commit it themselves.
Incorrect. There *is* a way for a reporter to flag that they have
followed through on feedback -- they update the 'need docs', 'needs
tests' and 'needs improvement' flags.
Example workflow:
* Alice creates a ticket, with an incomplete patch (no tests,
incorrect implementation)
* Bob reviews the patch, marks it "Accepted, needs tests, patch needs
improvement"
* Alice updates the patch, adding tests (but not changing the
implemenation). She removes the two flags.
* Charlie reviews the patch, resets the 'patch needs improvement flag'
* Alice updates the patch, fixing the implementation. She removes the
needs improvement flag.
* Daisy reviews the patch, marks it RFC.
At any point in this process, a search for tickets "Accepted & has
patch & !needs improvement & !needs docs & !needs tests" will reveal
tickets that need review of some kind. These tickets either need to be
moved to RFC, or need to have their flags set to indicate the
deficiency in the patch.
A search for tickets with "Accepted & has patch & (needs improvement |
needs tests | needs docs)" will reveal tickets that require some
action on the part of the reporter (or some other interested party).
This doesn't address the issue of people not keeping their ticket
metadata accurate. Badly tagged tickets will always fall through the
cracks. However, as I've said before, adding *more* flags doesn't fix
this problem.
The only solution I can think of for this is to provide better canned
"work that needs doing" searches -- i.e., a clearly referenced
"patches that need review" list, and a "patches that need updates"
list. This way, a reporter can check for themselves whether their
ticket will appear on a todo list.
Yours,
Russ Magee %-)
Andrew Godwin
It was ticket state, not closing state (sorry about the reply lag!). I
guess it's more that I don't like closing bugs purely because they've
not got enough information - it gives the wrong impression in my mind (I
only close once I have positive affirmation that it is invalid,
worksforme, etc.)
This is, I suspect, more a relic of the way I use Trac. I'm fine with a
closing state as well, I just feel like closing tickets before we even
know what they are sends the wrong impression.
On the other hand, we want to keep the trac as cruft-free as possible,
so I suppose closing tickets is fine in that regard. The ticket state
proposal has merit in that it can be crafted so that someone replying to
a ticket in needsinfo state moves it back into a "normal" state, thus
needing the minimum level of Trac competency from the reporter, and not
presenting a rather imposing "ticket closed!!!" interface - getting to a
project's ticket and finding it closed deters me from replying a lot
more than finding it in a "needsinfo" state.
Andrew
Daniel Moisset
OK... after seeing a generally positive response (even if there's some
bikeshedding about the actual implementation which I don't care much
about, any of the proposals solve *my* problem ;-) ), how can this be
moved forward? I know the process that django changes follow, but I
don't know what's the process to change process :) (or who takes the
decision)
Should I open a ticket about this proposal?
Regards,
D.
Luke Plant
> OK... after seeing a generally positive response (even if there's some
> bikeshedding about the actual implementation which I don't care much
> about, any of the proposals solve *my* problem ;-) ), how can this be
> moved forward? I know the process that django changes follow, but I
> don't know what's the process to change process :) (or who takes the
> decision)
>
> Should I open a ticket about this proposal?
Do open a ticket, because we need documentation patches for this, in the
'contributing' page. The current page says this:
“worksforme”
Used when the ticket doesn’t contain enough detail to replicate the
original bug.
That happens to be very similar to what we are proposing for
'needsinfo', so we at least need to think about how 'worksforme' is
different.
This will also require a change to Trac, which I for one don't know how
to do (I don't see the configuration pages I would need in the Trac
admin interface).
My 2 cents on state vs resolution:
If we go for a 'needsinfo' triage state, then at some point we will need
to go and close ancient tickets which have 'needsinfo' and the original
reporter has not provided it - presumably closing as INVALID. This will
require some work to do (manual or scripted), and it may well be
incorrect - we don't *know* that the report is invalid (according to our
current definition), we just don't know enough to say it *is* valid.
That pushes me in the direction of a 'needsinfo' resolution. We just
have to remember to say 'Please re-open the ticket when the information
has been provided' - and the 're-open' action is very obvious in Trac -
more obvious than the 'Triage stage' box.
Regards,
Luke
--
"I spilled spot remover on my dog. Now he's gone." (Steven Wright)
Luke Plant || http://lukeplant.me.uk/
Daniel Moisset
>
> Do open a ticket, because we need documentation patches for this
Done, http://code.djangoproject.com/ticket/14702
Thanks for the feedback.
Daniel
George Sakkis
wrote:
marking my own tickets RFC as a last resort to move them forward.
Unlike your example workflow, my experience is often like this:
* I create a ticket and submit a patch plus passing tests (no need for
docs if it's a bug or a promise to add them once it is reviewed and
accepted, i.e. it passes the DDN stage).
* The ticket stays unreviewed for days, weeks or months or it is
marked as accepted at best, without actual feedback how to proceed,
one way or another.
* At some point I mark it RFC since as far as I am concerned it *is*
ready for checkin.
* More time passes and still nobody bothers.
If I'm doing it wrong, please educate me.
George
Gabriel Hurley
tell you that making the actual change will require some hacking in
the trac.ini file. There's no way to do it through the admin.
All the best,
- Gabriel
Daniel Moisset
> to do (I don't see the configuration pages I would need in the Trac
> admin interface).
>
If you have the admin module ui enabled and TRAC_ADMIN permissions,
you should see an admin link to the right displaying a dashboard with
a "resolutions" link at the left where you can add ticket resolutions.
If you don't... you should, it helps a lot :)
Otherwise you'll need to add a row to a table in the trac db.
Regards,
D.
Tai Lee
you have it or if you disagree" because it clearly puts the onus back
on the reporter to re-open the ticket after addressing the feedback in
order to see any progress.
Leaving tickets "open" with potentially yet another status or flag
indicating that it needs some type of action will just contribute to
the problem we already have with many tickets stagnating in some
combination of "open" for literally years without any action.
If a ticket is marked as "needs more info" while remaining open, it
may be that only the original reporter (or somebody who is seriously
motivated to dig) will be able to provide that more info, and if it's
off their radar for whatever reason the ticket will likely remain in
that state forevermore.
Cheers.
Tai.
Tai Lee
project and that we're all volunteers, but I have found it it
extremely disheartening and a discouragement to further contribution
when I have invested the time to do the right thing and submit a
detailed report, with patch and docs and tests, and the tickets remain
untouched or merely accepted without feedback (or without further
feedback once initial feedback is addressed) for a prolonged period of
time.
Yes, it's possible to find tickets with many combinations of flags
that need certain types of attention, but the big problem is that
these tickets aren't being found. All the processes in the world won't
help if at the end of the day we just say that sorry, nobody was
interested in reviewing that particular combination, or a reviewer
wasn't notified or failed to follow up on a ticket where they provided
feedback that has been addressed and other reviewers simply left the
ticket alone because they assumed that it would be reviewed again by
the person who initially provided feedback.
I think that while core committers are only volunteers as well, they
should lead the community by example and follow up tickets that they
have accepted, or provided feedback on, or simply accepted without
feedback. Maybe trac can be improved in this respect by notifying
reviewers when tickets that they have closed, or accepted, or provided
feedback on, are updated.
Cheers.
Tai.
Luke Plant
> On Tue, Nov 16, 2010 at 12:02 PM, Luke Plant <L.Pla...@cantab.net> wrote:
> >
> > This will also require a change to Trac, which I for one don't know how
> > to do (I don't see the configuration pages I would need in the Trac
> > admin interface).
> >
>
> If you have the admin module ui enabled and TRAC_ADMIN permissions,
> you should see an admin link to the right displaying a dashboard with
> a "resolutions" link at the left where you can add ticket resolutions.
> If you don't... you should, it helps a lot :)
The admin UI is enabled, but there doesn't seem to be that 'resolutions'
list :-(
Gabriel Hurley
> reviewers when tickets that they have closed, or accepted, or provided
> feedback on, are updated.
(by anyone), the reporter, the owner, anyone on the cc list, and
anyone who has changed the ticket (by adding a comment, triaging,
etc.) gets an email. I get quite a few notifications every day from
Django's Trac install. I can only imagine what it's like for the most
active devs.
The issue which has already been alluded to is that these are
unfiltered streams of information. Getting 30 emails that say someone
added themselves as a CC on this ticket or that ticket makes it less
likely that you'll see the one email where someone reviewed a patch
and marked it as RFC. This is where good Trac queries come in.
As always, the rest comes down to the real hours we're all willing and
able to put in :-)
Gabriel Hurley
I don't have access to that server to see if the TracAdmin module is
current enough to support the "resolution add <value>" command. If so,
that'd be the easy way. If not, like Daniel said, it's straight to the
DB!
- Gabriel
On Nov 16, 5:37 pm, Daniel Moisset <dmois...@machinalis.com> wrote:
Russell Keith-Magee
> Bear in mind that this is a *very* old Trac installation... ;-)
Hopefully not for long.
Jacob is in the process of bringing a new server online to host
djangoproject.com, and part of that upgrade will be an updated Trac
install, running Trac 0.12 rather than 0.10. Once that upgrade is in
place, it will (hopefully) be a lot easier to implement this and other
Trac changes.
Yours,
Russ Magee %-)
Tai Lee
update is made, not anyone who added a comment. Unless a reviewer adds
themselves to the CCs when providing feedback or assigns the ticket to
themselves (which would be unlikely, when leaving feedback that the
reporter needs to take further action), the reviewer won't be
notified. But other people who stumble across the ticket might think
that since feedback was been provided, and possibly even actioned
already by the reporter, that the ticket will be monitored by the
reviewer and therefore leave the ticket alone themselves.
The problem is, as you've mentioned, that is is super easy for busy
people to skim through a large volume of trac email notifications when
the majority are just people adding a comment or adding themselves to
the CCs. There's nothing that makes it stand out to indicate who is
responsible for the next step, whether the next step is pushing it
back to the reporter, closing it, finding someone else to review it,
marking it RFC or actually committing it.
I think there needs to be more ownership of tickets and a more direct
way to make it clear to reporter, reviewer, and any other interested
parties, in whose court the ball lies for the next steps to be taken.
Similar to the way major features need to be championed by at least
one core committer, I'd like to see more of a mentorship type scenario
between reporter and reviewer, sort of like remote pair programming,
to avoid stagnation.
Explicitly, this would involve a commitment from the reporter to
action feedback given to them and a commitment from the reviewer to re-
visit the ticket once that feedback has been actioned and update its
status accordingly. At the moment all these tickets just sit in a pool
waiting for the lucky day when some random person will stumble across
it and hope that they don't just assume someone else is taking care of
it and leave it as is.
This doesn't have to be done with new flags on trac, but maybe a
change to the contributing guide that tells people that when they
review a ticket and provide feedback on it, they can expect the
reporter to assign the ticket back to them when the feedback has been
actioned making it clear that they should revisit it? I'm not sure if
this would trigger an email notification to the new "owner", if they
didn't take ownership of the ticket themselves?
Cheers.
Tai.
Russell Keith-Magee
To date, this hasn't been formally documented. This is something that
#14401 will hopefully address.
> Unlike your example workflow, my experience is often like this:
>
> * I create a ticket and submit a patch plus passing tests (no need for
> docs if it's a bug or a promise to add them once it is reviewed and
> accepted, i.e. it passes the DDN stage).
> * The ticket stays unreviewed for days, weeks or months or it is
> marked as accepted at best, without actual feedback how to proceed,
> one way or another.
> * At some point I mark it RFC since as far as I am concerned it *is*
> ready for checkin.
> * More time passes and still nobody bothers.
>
> If I'm doing it wrong, please educate me.
You aren't doing anything wrong (that I can see), but you perhaps have
some unrealistic expectations.
You need to remember that this is a volunteer project. Nobody is paid
to work on Django. Uploading a patch doesn't automatically give you
special priority. It puts your patch on a very long list of things
that need to be done. Given infinite time, this list of things will be
done -- but we don't have infinite time, so we need to prioritize.
Yes, you have 2 tickets currently in RFC state. Assuming your RFC
self-triage isn't premature, these should get committed to trunk
before 1.3 ships.
However, stand back and take an objective look at these tickets --
neither of them are especially critical. Yes, they're bugs, and yes,
it would be nice to resolve them. But they don't cause catastrophic
data loss. They've existed as misfeatures of the ORM since before 1.0.
And we haven't been overrun with people complaining about these
issues. I don't see a great deal of reason to prioritize these two
tickets over any of other issues vying for my time.
And hence, I haven't. Instead, I've concentrated on things that needed
to be done for the alpha or beta release (new features, doctest
ports), or other more immediate Django-related activities (mailing
lists, IRC). The other core developers have made similar value
judgements, and as a result your ticket hasn't reached the top of
anyones todo list.
Once the beta is out of the way and we have a feature freeze in place,
pure bugs like the tickets you describe will have a much better chance
of being committed.
As for other tickets languishing in accepted state -- they need to be
reviewed. We need people to volunteer to do that reviewing. By a
conservative query [1], there are there are currently 352 tickets in
that state. There are also 173 completely unreviewed tickets. Without
wanting to seem completely mercenary -- why are your tickets more
deserving than the 500 others that need review attention?
We've had a number of discussions recently about ways to improve Trac
to give better visibility to "work that needs doing", and better ways
to provide feedback on what the community as a whole considers
important. However, this won't increase the amount of time that is
volunteered; the best we can hope for is a better utilization of the
limited volunteer resources at our disposal.
If you have any ideas on how to increase the amount of volunteer time
that is available to us, I'm all ears. I'm also open to any
suggestions on how to better engage the community to do the work that
needs doing. But please be realistic -- this is a volunteer project,
we can't force anyone to do anything, and the core team is already
extremely busy.
Yours,
Russ Magee %-)
Gabriel Hurley
>
> >> * Alice creates a ticket, with an incomplete patch (no tests,
> >> incorrect implementation)
> >> * Bob reviews the patch, marks it "Accepted, needs tests, patch needs
> >> improvement"
> >> * Alice updates the patch, adding tests (but not changing the
> >> implemenation). She removes the two flags.
> >> * Charlie reviews the patch, resets the 'patch needs improvement flag'
> >> * Alice updates the patch, fixing the implementation. She removes the
> >> needs improvement flag.
> >> * Daisy reviews the patch, marks it RFC.
>
> >> At any point in this process, a search for tickets "Accepted & has
> >> patch & !needs improvement & !needs docs & !needs tests" will reveal
> >> tickets that need review of some kind. These tickets either need to be
> >> moved to RFC, or need to have their flags set to indicate the
> >> deficiency in the patch.
>
> > I admit I am guilty of breaking the (unknown to me) rule/etiquette of
> > marking my own tickets RFC as a last resort to move them forward.
>
> To date, this hasn't been formally documented. This is something that
> #14401 will hopefully address.
#14401. I'm gonna give that ticket one last call for "words of wisdom"
from anyone who'd like to contribute to the wiki, then write it up as
a patch.
Carl Meyer
> I believe that only the reporter, owner, and CCs are notified when an
> update is made, not anyone who added a comment.
notifications from Trac about changes to tickets that I have commented
on, but not added myself to CC on.
> I think there needs to be more ownership of tickets and a more direct
> way to make it clear to reporter, reviewer, and any other interested
> parties, in whose court the ball lies for the next steps to be taken.
> Similar to the way major features need to be championed by at least
> one core committer, I'd like to see more of a mentorship type scenario
> between reporter and reviewer, sort of like remote pair programming,
> to avoid stagnation.
to be made to encourage it; it's a matter of communication. I'm pretty
much always in the IRC channel and I read the mailing list; if there's
a ticket I've offered feedback on, and action has been taken on the
feedback, and I'm not getting around to the next steps in a timely
way, I'm quite receptive to being pinged about it here or in IRC.
Can't promise that'll mean I look at it right now, but at least I can
communicate my plans more clearly.
I don't think there's any way round the fact that with so many issues
to address, and limited time to address them, it requires some
initiative and persistence from the ticket reporter/owner to keep
moving a specific ticket forward; sometimes that will involve actively
reminding a member of the core team that the issue is in our court.
There's also the release cycle issue Russ mentioned. While I don't
think the release cycle timeframe should ever be used to discourage
someone from bringing something up for discussion, or working on it,
it still is going to have an impact on core team priorities. With an
alpha or beta release looming, I'm going to focus on new features that
need to go in before that deadline; I know there'll be time after the
beta to commit bugfixes. I'm sure I could do a better job of
communicating that to ticket owners/reporters in specific cases.
Carl
Daniel Moisset
Hi. This message intends to be a small nudge to reactivate
http://code.djangoproject.com/ticket/14702 which was waiting on the
trac migration.
I know this probably must be handled by a trac admin, but if I can
help in some way, I have enough experience setting up trac to help
here. Just let me know. I also can help updating
http://code.djangoproject.com/wiki/ContributingHowTo (which will be
needed for the #14401 doc update) once this change is done
Regards,
D.