Proposed Hudson/Jenkins Bug Triage Procedures
mlb5000
Disclaimer: I'm only just joining this community so I absolutely want
those of you with more experience developing/fixing bugs in Hudson and
elsewhere to chime in on this conversation. What I want to do is try
to come to some consensus on how Hudson bugs are to be identified,
disambiguated, prioritized, and ultimately, scheduled (more or less)
for a release. This is part of what I hope will be a larger overall
effort to bring the Technical Debt I've observed since diving into the
source code into check.
As of right now there are 1853 open bugs in Hudson, of which the
average age is just under a year at 359 days. Some users, myself
included, have been getting discouraged with cycle time on bugs,
particularly when the Continuous Integration server is supposed to be
the one thing you can most rely on. When teams make their CI system
the heart and soul of their day-to-day development, testing,
configuration management, and release processes, they need to able to
rely on it. If any tool can be categorized as mission-critical, it's
the CI system.
Here is what I am proposing:
1. Users submit bug reports at issues.hudson-ci.org like they always
have.
2. At some frequency (daily?, weekly?, as they pop up?), developers go
out and perform the following procedure:
a. Pick the oldest non-triaged bug.
b. Check whether or not the bug is a duplicate with another bug.
This will likely require aid from the more seasoned Hudson
contributors.
i. If a duplicate exists but it has been resolved, I would
recommend not reopening it, but just linking the two issues together
so the person who goes to fix the current incantation of the bug can
use it as a resource if necessary. Continue to the next step.
ii. If a duplicate exists, but is still open, link the two issues
for future reference, Close the new one as Duplicate, and redirect all
users to the older one. Go back to 2.a.
iii. If no duplicate exists, continue.
c. Is the bug reproducible in the most recent version of Hudson? I
don't think it's worth verifying the bug in their reported version
since they'll ultimately have to upgrade to get the bug fix anyway, so
we might as well work towards the shortest path.
i. If the bug is not reproducible in the latest version, close as
Cannot Reproduce and inform the user(s) that it appears to have been
fixed in version X, and go back to 2.a.
ii. If the bug appears to still exist in the current version, note
which version it was verified in and continue.
d. Gather as much information as possible from users. What version
of the system is this bug occurring in? What environment are they
using? Did this error start occurring after an upgrade? What version
did they upgrade to/from? Are they able to work around the issue or
has it stopped their usage of Hudson in its tracks? Ask any questions
that might help isolate the commit that introduced the bug, and help
other contributors make the best possible prioritization decisions.
e. Set the priority of the issue based on some criteria. I need
some suggestions from the community about what would be the best
criteria. I would actually suggest trying to get an open source
license of GreenHopper from Atlassian. Their ranking functionality
within GreenHopper works very well for this purpose since you rank
items relative to each other, not just with priorities.
f. Mark the bug as triaged and if there are still non-triaged issues
go back to 2.a.
3. Contributors grab the highest (relative) priority bug to fix and
get to it. Votes should also contribute to deciding which bug to fix
since that information is real-time and earlier triage information can
become more and more out of date.
I'm extremely interested in getting some feedback about these
procedures. I'm sure I missed some opportunity in this process or
overlooked something, so please let me know what you think. I hope
that by unifying these procedures across the community we can work
towards reducing that "average age" stat to below a month or two. I'm
not saying it will be easy, but I'm certain this community can do it.
Thanks,
Matt
Christoph Kutzinski
Some comments:
In my experience most of the very old bugs fall into one of these scenarios:
- happen only under very rare circumstances. Developer cannot reproduce
it easily
- are not reproducible because reporter didn't provide enough details
(and often doesn't respond to questions about details)
- fix would break some other functionality and/or would mean serious
re-engineering of some parts of Hudson
- are related to a plugin which is not (really) maintained any more
And then there are of course the 'normal' bugs, but I think they are the
minority.
This is just my very subjective feeling why there are so many old bugs.
What we can do easily IMHO:
- search actively for duplicates and close them
- ask reporters about more details
- if bug is old and reporter doesn't respond in a timely manner, then
close it as 'not reproducible'
- And what I really like to see, is a prominent message for reporters of
new bugs to provide as much details as possible ALREADY WHEN they enter
the bug initially.
So much time is wasted by asking for details and waiting for the answer.
Your list in 2d is very good. I'd like to add console output and logfile
to it.
vtintillier
Arnaud Héritier
David Karlsen
Some killer bugs like http://issues.hudson-ci.org/browse/HUDSON-7921 http://issues.hudson-ci.org/browse/HUDSON-7813 have been hanging around without any response for a long time - and are can be reproduced.
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen
mlb5000
own "Project" in the Hudson Jira instance. It's weird that they're
split into components, but maybe there's a reason for it?
On Jan 30, 3:49 pm, Arnaud Héritier <arnaud.herit...@exoplatform.com>
wrote:
Arnaud Héritier
mlb5000
be handled in a more aggressive fashion. Maybe we could pick a cutoff
date and then write a comment on any of the issues that were created
before then. In it we could notify the interested users that if they
do not respond and says that the bug still affects them (and that
they're running a reasonable version of Hudson), the issue will be
closed. We wouldn't be starting from scratch (I think that would be
ill-advised), but at least we could weed out bugs affecting
uninterested users, or bugs that the reporters no longer feel the
effects of.
Thoughts?
mlb5000
environment myself, I don't think it's that bad :). Either way, if
they went this direction each plugin would need a named maintainer to
be the "Project Lead" on that project.
On Jan 30, 8:36 pm, Arnaud Héritier <arnaud.herit...@exoplatform.com>
wrote:
Arnaud Héritier
mlb5000
like to get started closing out some of these old issues if at all
possible. I noticed there was a "!CLOSED && STALE" query already on
issues.jenkins-ci.org and I was wondering how we want to handle Bug
reports that fall into this category. An issue is considered STALE if
it hasn't been updated within the last year. I'd say that's a pretty
good criteria for just calling it and closing the issue without
discretion, but I don't personally have any insight into what some of
those older bug reports might be, and the last thing I want is to
upset the users.
At any rate, does anyone have any (further) suggestions for how to
handle these older bugs reports without having to go through them one
by one? Just to provide some perspective, 494 (http://issues.jenkins-
ci.org/secure/IssueNavigator.jspa?mode=hide&requestId=10243) of the
1862 currently unresolved bug reports (http://issues.jenkins-ci.org/
secure/IssueNavigator.jspa?mode=hide&requestId=10244) are considered
stale.
Also, would someone who has administrative privileges in the Jenkins
Jira instance be able to add a Boolean field to the Bug issue type for
marking whether or not a bug report has been Triaged, or does anyone
have another way to keep track of whether a bug report has been
vetted?
evernat
I fully agree with you that obsolete bugs should be closed, in order
for most users to know what are really the current issues and in order
for most developers to know what has to be fixed.
But I think that the 494 stale bugs can't be closed like that.
For example, a large number of issues were written by Kohsuke for
things to check or for things to fix (for example issue 5036).
A bug having a low priority for one year probably doesn't mean that it
doesn't need to be fixed.
I currently suggest that some, if not all, developers read each one of
the stale bugs in
http://issues.jenkins-ci.org/secure/IssueNavigator.jspa?mode=hide&requestId=10243
and that we close them when necessary.
I will try to do that myself (instead of sleeping) and as Christoph
said:
- search actively for duplicates and close them [we may also read the
changelogs of releases]
- ask reporters about more details [if needed]
then close it as 'not reproducible'
tested and write your test results in the issues : it will save most
of the time needed by the developers to know if the stale issues can
be closed ! (and it may help you later to submit patchs)
bye,
Emeric
Arnaud Héritier
Hi Matt,
I fully agree with you that obsolete bugs should be closed, in order
for most users to know what are really the current issues and in order
for most developers to know what has to be fixed.
But I think that the 494 stale bugs can't be closed like that.
For example, a large number of issues were written by Kohsuke for
things to check or for things to fix (for example issue 5036).
A bug having a low priority for one year probably doesn't mean that it
doesn't need to be fixed.
I currently suggest that some, if not all, developers read each one of
the stale bugs in
and that we close them when necessary.
I will try to do that myself (instead of sleeping) and as Christoph
said:
- search actively for duplicates and close them [we may also read the
changelogs of releases]
Richard Bywater
Arnaud Héritier
Ullrich Hafner
On 02/07/2011 12:29 AM, mlb5000 wrote:
> Things haven't completely calmed down since the rename yet, but I'd
> like to get started closing out some of these old issues if at all
> issues.jenkins-ci.org and I was wondering how we want to handle Bug
> reports that fall into this category. An issue is considered STALE if
> it hasn't been updated within the last year. I'd say that's a pretty
> good criteria for just calling it and closing the issue without
> discretion, but I don't personally have any insight into what some of
> those older bug reports might be, and the last thing I want is to
> upset the users.
>
I think in a first step we should categorize issues before we are
updating or closing them.
It makes sense to handle issues from core (and core components)
differently then for plug-ins. E.g., I have several issues in my
plug-ins that are quite old, but I did not yet find the time to fix
them. I would like to have these issues still open, since these issues
reflect real bugs (or feature requests).
We also should handle issues differently that have no assignee yet.
There are 1958 unresolved issues without assignee (from 3251). I think
for issues with assignee the assignee should look at his issues in order
to suggest if the issue could be closed or not.
> At any rate, does anyone have any (further) suggestions for how to
> handle these older bugs reports without having to go through them one
> by one? Just to provide some perspective, 494 (http://issues.jenkins-
> ci.org/secure/IssueNavigator.jspa?mode=hide&requestId=10243) of the
> 1862 currently unresolved bug reports (http://issues.jenkins-ci.org/
> secure/IssueNavigator.jspa?mode=hide&requestId=10244) are considered
> stale.
>
I think we need to look at an issue before we can close it...
> Also, would someone who has administrative privileges in the Jenkins
> Jira instance be able to add a Boolean field to the Bug issue type for
> marking whether or not a bug report has been Triaged, or does anyone
> have another way to keep track of whether a bug report has been
> vetted?
>
What do you mean with issues that are triaged? Accepted as a real bug or
feature? Or already planned for a release?
Ulli
Alan Harder
On 02/06/11 17:27, evernat wrote:
> - ask reporters about more details [if needed]
> - if bug is old and reporter doesn't respond in a timely manner,
> then close it as 'not reproducible'
I have done a fair amount of this over the past couple years (not
recently though).. asking if issues still exist on recent versions, and
closing if no response in a few weeks.
- Alan
mlb5000
me about the state of the Jenkins bug backlog :). I never suggested
you should lose sleep over it though, lol. Were you actually able to
make any progress? I can definitely carve out some time to verify
whether or not stale bugs still exist.
@arnaud I've never used that tool, but if you think it might help make
a dent in discovering duplicate bug reports it can't hurt, especially
if it doesn't cost anything. It might be difficult to decide who the
Open Source license gets assigned to until the Jenkins Governance
group gets the project under a legal umbrella.
QUESTION: Does anyone know who the custodian on Open Source software
licenses for Jenkins is?
@ulli Just to be clear, I'm only focusing on bug reports right now,
not the other issue types like Feature or Improvement. By your same
query there are 709 unresolved bug reports that have no one assigned
to them. It seems like this list would be a good place to start. As
far as the Jira request, what I'm looking for is a way that multiple
people can tackle this problem at once without stepping on each
other's toes and duplicating work. Perhaps simply posting a comment
saying "Trying to reproduce" while you work on it, and "Bug Reproduced
in version X" or "Bug not reproducible in version X" will suffice?
That should at least be enough to keep people from trying to do the
same thing, right? Making sure a bug is assigned to the right
"component" is also a good idea. That way we the developers can
prioritize the work that needs to be done, particularly if a bug is
associated with a plugin like you describe.
QUESTION: Who should have the authority to mark a bug as Resolved and
then Closed in Jira? I don't want to feel like I'm encroaching by
going in there and changing the state of a bunch of open issues
without getting some direction first. I personally believe that bugs
marked as Resolved should have a second step where someone else
verifies that a bug no longer exists, but how should we handle it if a
bug is not reproducible at all in the current version? Should we
Close as "Cannot Reproduce" right away, or make a note that it is not
reproducible on some version and then have someone else do the same
check just to be safe? I'm sure it will also have a lot to do with
who reported that it was not reproducible. I'm sure the community
would be much less likely to trust me than Kohsuke.
@Alan, would you suggest that we take some of the more stale issues
and post a comment asking the reporter(s) if they're still having the
problem in the latest version? It seems like the general idea I'm
getting from everyone is that we should make an effort to confirm a
bug's existence in these cases as well.
On Feb 6, 8:27 pm, evernat <ever...@free.fr> wrote:
> Hi Matt,
>
> I fully agree with you that obsolete bugs should be closed, in order
> for most users to know what are really the current issues and in order
> for most developers to know what has to be fixed.
> But I think that the 494 stale bugs can't be closed like that.
> For example, a large number of issues were written by Kohsuke for
> things to check or for things to fix (for example issue 5036).
> A bug having a low priority for one year probably doesn't mean that it
> doesn't need to be fixed.
>
> I currently suggest that some, if not all, developers read each one of
Alan Harder
> then Closed in Jira? I don't want to feel like I'm encroaching by
> going in there and changing the state of a bunch of open issues
> without getting some direction first. I personally believe that bugs
> marked as Resolved should have a second step where someone else
> verifies that a bug no longer exists,
I think to date we've interpreted "Resolved" as closed.. a few people
come back and mark things verified, but not many. I never query
Resolved issues and wonder if they need to be verified anyway..
> but how should we handle it if a
> bug is not reproducible at all in the current version? Should we
> Close as "Cannot Reproduce" right away, or make a note that it is not
> reproducible on some version and then have someone else do the same
> check just to be safe? I'm sure it will also have a lot to do with
> who reported that it was not reproducible. I'm sure the community
> would be much less likely to trust me than Kohsuke.
As with stale bugs I would add a comment here and give the reporter a
chance to give more detail.. there are a lot of environment specific
issues out there.
> @Alan, would you suggest that we take some of the more stale issues
> and post a comment asking the reporter(s) if they're still having the
> problem in the latest version? It seems like the general idea I'm
> getting from everyone is that we should make an effort to confirm a
> bug's existence in these cases as well.
It's the method I've used in the past, to avoid complaints as you said..
but it does require looking at all the issues, no large scale pruning..
- Alan
Jesse Farinacci
On Mon, Feb 7, 2011 at 7:57 PM, Alan Harder <mind...@dev.java.net> wrote:
>
> As with stale bugs I would add a comment here and give the reporter a chance
> to give more detail.. there are a lot of environment specific issues out
> there.
>
I'm the one that put that !CLOSED && STALE query out there, so I am
personally interested in how this unfolds.. Anyhow, it would be very
nice if we could somehow tag at least the ones which are in Resolved
as 'last-chance' or something similar. If that were possible, I am
sure we can bulk update add a comment and tag to the effect that the
issue is a cold case and to update the record with a fresh
confirmation of the problem. If no update is received in 5 weeks, we
then bulk close any !CLOSED && hasTag(last-chance) or some such.
There are so many issues like this that it doesn't make sense to do it
manually. There are something like 4300 issues in Resolved state,
there are 4100 !CLOSED && STALE, of which 1050 are !CLOSED &&
!RESOLVED && STALE. For this final batch, if they are in plugins then
we should probably just leave them; if they are in core, then they
should also receive last-chance warnings on the presumption that there
have been at least 52 releases since the bug could conceivably have
been open and we want a fresh confirmation of the problem to proceed.
-Jesse
--
There are 10 types of people in this world, those
that can read binary and those that can not.
Ullrich Hafner
> @ulli Just to be clear, I'm only focusing on bug reports right now,
> not the other issue types like Feature or Improvement. By your same
> query there are 709 unresolved bug reports that have no one assigned
> to them. It seems like this list would be a good place to start. As
> far as the Jira request, what I'm looking for is a way that multiple
> people can tackle this problem at once without stepping on each
> other's toes and duplicating work. Perhaps simply posting a comment
> saying "Trying to reproduce" while you work on it, and "Bug Reproduced
> in version X" or "Bug not reproducible in version X" will suffice?
> That should at least be enough to keep people from trying to do the
> same thing, right? Making sure a bug is assigned to the right
> "component" is also a good idea. That way we the developers can
> prioritize the work that needs to be done, particularly if a bug is
> associated with a plugin like you describe.
Yes, that would be an option.
What we can also do is a more formal issue submission and QA process
(that I used successfully in several projects): use an additional status
(e.g., Submitted) that will be assigned to all created issues. All those
submitted issues need to be verified by the assignee and could be then
moved to the state Open. And for those issues that don't get an
assignee: here we need some volunteers who are evaluating these issues
e.g. on a daily or weekly basis (trying to reproduce or write a comment
to request additional information from the reporter). All of our old
open and unassigned issues could be moved back to the Submitted state.
Ulli
Arnaud Héritier
> @evernat - See, I knew there were people here who knew a lot more than
> me about the state of the Jenkins bug backlog :). I never suggested
> you should lose sleep over it though, lol. Were you actually able to
> make any progress? I can definitely carve out some time to verify
> whether or not stale bugs still exist.
>
> @arnaud I've never used that tool, but if you think it might help make
> a dent in discovering duplicate bug reports it can't hurt, especially
> if it doesn't cost anything. It might be difficult to decide who the
> Open Source license gets assigned to until the Jenkins Governance
> group gets the project under a legal umbrella.
They weren't difficult to give mine when I asked it.
I don't think it will be a problem to not have a real foundation to receive it.
Francois Cottet
I really like the idea of the comment "Reproduced in version X" or
"unable to reproduce on Y platform" because it gives much more details
than just the version tag or the issue status.
Is there any way for a user to quickly describe his environment?
I was thinking of a simple link "get installation details" reporting
info like:
- OS version for the master and each node
- JDK used
- Version of Jenkins and all plugins
- job type (and eventually plugins used)
This could to reproduce the issues related to plugins or specific
platform.
Cheers,
Francois
Le 8 févr. 2011 à 17:45, Ullrich Hafner <ullrich...@gmail.com> a
écrit :
Ullrich Hafner
> Hi all,
>
> I really like the idea of the comment "Reproduced in version X" or
> "unable to reproduce on Y platform" because it gives much more details
> than just the version tag or the issue status.
>
> Is there any way for a user to quickly describe his environment?
> I was thinking of a simple link "get installation details" reporting
> info like:
> - OS version for the master and each node
> - JDK used
> - Version of Jenkins and all plugins
> - job type (and eventually plugins used)
>
> This could to reproduce the issues related to plugins or specific
> platform.
>
> Cheers,
> Francois
>
I think this is not possible out of the box.
However, one can write a simple plug-in (or core feature) that does add
a create issue link to a Jenkins installation, that will open a
pre-filled Jira create issue screen when clicked. Maybe we could use
such a feature to replace the current log message 'Please report this
bug at the mailing list...'.
Ulli
Erik Ramfelt
I would love to see something like that. Unfortunately Jira doesnt
seem to have any such feature in the webui today, but there are
several ways to do it in code, [1], [2] and of course there is a CLI
that is written in java that can create issues [3].
Such a plugin should
* Require that the user to login, so they can later be notified with
changes and perhaps close it when the issue is fixed.
* The user should be able to verify that it doesnt send any
confidential data to the jira.
* Maybe a hash of an exception could be added to the created issues,
so user wont create issue for the same error. Instead they can be
allowed to vote and/or watch an issue.
* Perhaps it could even be possible to group an exception on a
component/plugin dependent on the classes involved. But that may be
stretching it :)
Is there any such extension today that will allow any plugin to listen
on exception crashes or other problems?
[1] - Jira SOAP library -
https://studio.plugins.atlassian.com/wiki/display/JSOAPLIB/Jira+SOAP+library+(maven2)
[2] - Jira java API (early version) -
https://studio.atlassian.com/wiki/display/JRJC/Home
[3] - https://studio.plugins.atlassian.com/wiki/display/JCLI/JIRA+Command+Line+Interface
//Erik
> Ulli
>
Houghton, Sean
JIRA docs: http://confluence.atlassian.com/display/JIRA/Creating+Issues+via+direct+HTML+links
--
Sean Houghton
Kohsuke Kawaguchi
Please make sure to file an RFE out of this!
--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
mlb5000
engineering the solution, but by making it convenient for their users
to gather information relevant to their runtime environment in the
Administration section. That said, I believe Jenkins already gives
that information in "Manage Jenkins"->System Information. Maybe
someone should add a note to the "Environment" field in Jira asking
the user to provide the information from that page? While I'm not
opposed to a more robust solution, I think trying to detect bugs
system-wide and automatically sending them upstream to Jira is a bit
ambitious, but Rome wasn't built in a day! I think the least we can
do is gather all pertinent information in one place for users to get
at it when we ask them for it.
@All: I think this has been a productive conversation but since we're
starting to circle around the same few issues without much empirical
evidence nudging us in one direction or another, I think it's best to
just get started. We can learn more as we go. See you all in Jira!
On Feb 8, 8:06 am, Francois Cottet <fct.j...@gmail.com> wrote:
> Hi all,
>
> I really like the idea of the comment "Reproduced in version X" or
> "unable to reproduce on Y platform" because it gives much more details
> than just the version tag or the issue status.
>
> Is there any way for a user to quickly describe his environment?
> I was thinking of a simple link "get installation details" reporting
> info like:
> - OS version for the master and each node
> - JDK used
> - Version of Jenkins and all plugins
> - job type (and eventually plugins used)
>
> This could to reproduce the issues related to plugins or specific
> platform.
>
> Cheers,
> Francois
>
Erik Ramfelt
to the System configuration page; "Help make Jenkins better by sending
anonymous usage statistics and crash reports to the Jenkins project"
What happens to those crash reports?
//Erik
mlb5000
used.
On Feb 9, 4:28 am, Erik Ramfelt <eramf...@gmail.com> wrote:
> I just noticed that some crash data is supposed to be sent according
> to the System configuration page; "Help make Jenkins better by sending
> anonymous usage statistics and crash reports to the Jenkins project"
>
> What happens to those crash reports?
>
> //Erik
>
> On Tue, Feb 8, 2011 at 19:47, Kohsuke Kawaguchi
>
>
>
>
>
>
>
> <kkawagu...@cloudbees.com> wrote:
>
> > Please make sure to file an RFE out of this!
>
> > On 02/08/2011 05:14 AM, Erik Ramfelt wrote:
>
> >>https://studio.atlassian.com/wiki/display/JRJC/Home
> >> [3] -
Francois Cottet
Clearly the http://server//systemInfo is exactly what I was looking for.
We could require the page to be saved and attached to bug report.
The Issue Tracking page [1] is a must read before any issue report. Kudos Mirko.
Can we link it on the Jenkins homepage? Maybe replacing link of Bugtracker on left menu from direct issues.junking.org to that page which will then have the link to JIRA. Anyway it won't change anything for "hard core" developer, who all knows the JIRA url, but it will be useful for new comers.
My 2 cents.
[1] http://wiki.jenkins-ci.org/display/JENKINS/Issue+Tracking
Best regards,
Francois
PS: I noticed that the plugins section is quite unreadable, since the column 'name' is very very large (seen on Mac Safari)
Mirko Friedenhagen
another time including a hint to search for existing issues before
reporting a new one :-).
Regards
Mirko
--
http://illegalstateexception.blogspot.com/
https://github.com/mfriedenhagen/
https://bitbucket.org/mfriedenhagen/