Issue tracker and AOSP contributions

[Jean-Baptiste Queru]

Dear AOSP contributors,

Now that we have devices for which 100% of the proprietary device-specific binaries are available for AOSP (Nexus 7 and Nexus 4), it should be possible to contribute to all areas of the AOSP code base without being blocked the lack of some of those binaries.

In this state, having the AOSP issue tracker in a decent shape becomes relevant. There are some obviously desirable properties for such an issue tracker:

-it needs to be relevant: there's no point having issues reported in the AOSP issue tracker that can't be fixed in the AOSP source code.

-it needs to be accurate: it shouldn't contain reports for issues that have been resolved or that aren't relevant any more.

-it needs to be up to date: there should be no doubt when an issue is being worked on, even before there's any code visible.

At the same time, some aspects aren't quite so obvious:

-how do we keep the AOSP issue tracker public while still being a relevant technical resource that can directly result in code getting written and contributed?

-how far does it make sense to track hardware-specific issues? In fully-supported devices like Nexus 7 and Nexus 4? In partially supported devices like Galaxy Nexus and Nexus 10? In devices that aren't supported in the master branch like Nexus S and Xoom?

-how do we deal with issues that require changes both in source code that lives in AOSP and in proprietary device-specific binaries?

-how far does it make sense to track issues in all the other devices for which we don't have source code or support?

I'd like to hear your opinions about those points, and about which major points I haven't been considering.

Thanks,

JBQ

--
Jean-Baptiste M. "JBQ" Queru
Technical Lead, Android Open Source Project, Google.

Questions sent directly to me that have no reason for being private will likely get ignored or forwarded to a public forum with no further warning.

[Al Sutton]

Sounds like you're getting to the fun stuff :).

One of the things which I can see helping get this right is getting the bug lifecycle right. I'd suggest having the following states;

1) Reported

2) Verified / Closed (Can not reproduce) / Closed (Duplicate) / Closed (Issue resolved in later release) / Closed (Issue made irrelevant by later release)

3) In progress / Blocked (Resoluion dependant on other issue) / Blocked (Resolution dependant on non-AOSP component)

4) Resolution available

5) Fix Verified

6) Closed (Fix merged)

The typical flow would be;

- User reports issue; Issue is marked as Reported (Stage 1)

- Issue is triaged to determine which of the Stage 2 states it should go into.

- An appropriate developer makes the choice to move an issue from stage 2 to 3 and assign it an appropriate Stage 3 state. 

- The issue then either goes to stage 4 when the developer uploads a patch to Gerrit, or, if the issue has been marked as "In Progress" for over a week and no patch is uploaded the issue drops back to its' previous stage 2 state. Issues can stay in the Blocked state indefinately.

- Once the patch has been approved in Gerrit the issue goes to stage 5.

- Once the patch has been merged the issue goes to stage 6.

This would address the issues you've mentioned in the following way;

- Non relevant issues would go into a blocked state until the fault was fixed in the 3rd party component (or remain blocked if the 3rd party component doesn't get fixed).

- Issues which have already been fixed or made irrelevant by changes in a later release get caught at stage 2

- Code would only become visible after something has gone to stage 3, and would be merged into the codebase between stages 5 & 6.

- Public issue trackers which maintain relevance need moderators. The default view should be bugs which are stage 2 and later, moderators would be required to move bugs from stage 1 to stage 2.

- Bugs with AOSP and 3rd party change dependancies could go into stage 3, if the AOSP component is finished before the non-AOSP one it stays at Stage 3 moving from "In Progress" to "Blocked (Resolution dependant on non-AOSP component)", if the non-AOSP part is fixed first the bug moves from stage 3 to 4 with a note about the dependancy.

- People need to be encouraged to report issues to OEMs for components they can work to ensure a that OEMs are aware of the issue. Non-AOSP resolvable issues can be moved to Stage 3 - Blocked (Fault identified within non-AOSP component) which should be enough to signify that the issue is known, but a fix won't come solely from the AOSP.

Does this sound reasonable?

Al.

-- 

Al Sutton

Funky Android Ltd.
www.funkyandroid.com

The views expressed in this email are those of the author and not necessarily those of Funky Android Limited, it's associates, or it's subsidiaries. 

--
 
---
You received this message because you are subscribed to the Google Groups "Android Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-contr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jean-Baptiste Queru]

Al, I knew I could count on you for an opinion ;-) Indeed, I expect that AOSP will be more fun now, at least for me.

TL;DR: Good stuff. Gotta make it clearer to people reporting issues that the AOSP tracker has a narrow focus. There's clearly a grey area with issues that fall right at the edge of AOSP.

Your idea sounds reasonable. It has many similarities with the process we already have today, and a few differences.

The similarities:

-State 1 is "New" today.

-State 2+ (i.e. the incoming issue is acknowledged) is "Unassigned" today.

-We've got quite a few variants for state 2- (i.e. the incoming issue is rejected): Spam / Question / UserError / WrongForum / Duplicate / Unreproducible / WorkingAsIntended / Declined / Obsolete.

-There's also a loop around state 1: "NeedsInfo" which then turns back into any of the states above or "NotEnoughInformation"

-In state 3, we have "Assigned" and "Reviewed" (where "Reviewed" means "done in private by Google"). That second option is necessary to reflect the reality of Android.

-Skipping ahead to state 6 though paths where code gets written and approved, we have "FutureRelease" (in master), "Released" (in a tagged version).

-Another potential way out of state 3 is straight to "Obsolete".

Here are the differences I see:

-With the current process, the transitions out of state 3 all the way into state 6 don't formally exist in the status list in the issue tracker, they live in Gerrit. Note that for practical purposes state 5 rarely exists in Gerrit: in most cases, once a change is approved, it gets merged quickly.

-There's no notion of "Blocked" today.

I won't go through your analysis point-by-point, since it's excellent as it is. A couple of minor thoughts on that subject:

-If 1 is hidden from view by default (because of moderation), there's an even higher risk that people will file duplicates if for any reason moderation falls a bit behind (I take time off, sometimes). In fact that's quite an issue today with the AOSP issue tracker: the default search only shows open issues, and it's quite likely that people file duplicates of issues that have already been closed.

-I definitely agree that we need to find a better way to direct people to the right resources. First, AOSP is fundamentally a technical project, and we're not equipped to handle user support. Even if in some instances we might know the answer, bypassing the proper support structures prevents support people from doing their job properly and escalating the right problems. Also, even as a technical project, there isn't much we can do about issues caused by code changes in downstream forks, even if we receive all the information that someone working on that fork would need to diagnose the issue.

-I've got doubts about "Blocked", that I can't quite pinpoint. I can't imagine many scenarios where an issue would truly belong in AOSP (i.e. would be eventually resolved by a code change in AOSP) and yet be blocked on an external dependency. My gut feeling is that most issues that we could mark as "Blocked" would not really be AOSP issues and should be reported somewhere else.

Some additional thoughts:

-I'd rather handle states 4 and 5 with some automated linkage between the issue tracker and Gerrit, such that the presence of related changes in Gerrit gets reflected in the issue tracker.

-There's a massive grey area at the edge of AOSP, which might be the one you tried to capture as "Blocked". E.g. how do we deal with an issue observed on a 3rd-party device that can be tracked to AOSP code even though it's not reproducible on any AOSP-supported device?

Action items on my plate:

-A huge lot of cleanup. In order to be useful, the issue tracker needs to accurately represent the list of issues that can truly be handled in AOSP, and whether they're being worked on.

-A lot of clarification that the AOSP issue tracker is not the place to get user support.

JBQ

[Al Sutton]

Good to hear I wasn't totally off-base with my post :)

- Although the default view could be stage 2+ the default search could cover all stages. My main issue with showing stage 1 bugs on the default view is spam (I'm sure none of us want to see a "Cheap Home Loans Now" bug sit on the list until a moderator can get to it), so anyone looking for a specific issue would see un-triaged issues, but someone just browsing the site would only see 

confirmed issues.

- Maybe we should require a device field with a bug report and only allow AOSP supported devices. That way it should discourage/block bug reports for devices which aren't supported and push users towards the device manufacturer.

- The Blocked issue can be used where an issue lies in the BSP for a specific device but has user visible aspects (e.g. WiFi connectivity issues due to a problem with a WiFi driver in a platforms BSP). There should be very few (if any) instances of a Blocked issue which affects all devices, which may be what's making you a bit uncomfortable with it.

- Linking Gerrit and the bug tracker sounds like a great way to solve the issue of keeping the bug tracker up to date with the path from fix submission to deployment.

- Showing a bug doesn't exist on an AOSP supported device should be enough to mark it as either "Closed (Can not reproduce)", or "Blocked (Resolution dependant on non-AOSP component)". Being strict about doing that should discourage people from reporting issues which relate only to non-supported devices.

[Jean-Baptiste Queru]

-Oh, I really like how you would want to separate the default view and the default search. It's subtle, smart, and probably very effective. If spam becomes an issue again (it was once, during one short period), I'd rather solve it directly with the code.google.com team rather than having to adapt our processes around it.

-The first reason why I feel uncomfortable about "Blocked" is that such issues could stay open forever. The second reason is that those issues can't possibly be handled by the community. I'd like the AOSP issue tracker to be a resource for the Open Source community to be able to figure out what they can contribute to, and if we leave issues open forever that nobody can do anything about we'll be "polluting" the database with those dead issues.

-Let's be very precise about AOSP-supported devices: the fact that a piece of hardware is a usable target for AOSP work doesn't mean that we're the support forum for consumer issues with that hardware (or with the retail software that runs on it). Just like we (technical folks) don't like having to use multiple issue trackers at the same time, support folks don't like having to look for user reports in multiple forums at the same time. The people running the Nexus support channels have explicitly asked me to not talk support questions in AOSP and to redirect those questions to the appropriate channels, so that they don't stay under their radar. That doesn't mean that we can't track device-specific issues in AOSP that could be fixed with the source code that's in AOSP, but we've got to make the distinction quite clear.

JBQ

[Al Sutton]

- The problem I can see with not having a "Blocked" state is that closing/removing the issues could end up with duplicate reports along the lines of "You've closed it but it's not fixed" or "I can't see this bug so I'm going to report it". I'm not a fan of issues that linger, but I can see a situation where a dependancy exists which may never get resolved.

- Maybe we need a "Closed (Not resolvable in AOSP, Please contact device provider)" state.

[Jean-Baptiste Queru]

Those are definitely possible scenarios. In fact they're probable.

"Blocked" isn't the only state for which someone will say "You've closed it but it's not fixed". We'll fix something in the AOSP master, and it won't immediately be on their device. The change might not even make it to the next tagged release. And once it makes it to a tagged release that could be many months later, there's no way to know when or even if it'll get deployed to a given device.

Same for duplicates: I don't think that we should keep fixed issues open just because there are some devices out there where the issue hasn't been resolved.

I don't think that there's anything wrong with your proposals in absolute terms. However, I am trying to focus a lot more on making the issue tracker a valuable resource for the technical contributor community, and the more we have open issues that aren't fixable with community contributions the less valuable the database gets. I'm assuming here that anyone who can read the machine setup instructions won't have any difficulty searching the database for all items instead of just the open ones.

JBQ

[Jean-Baptiste Queru]

Al, you're right, there's an issue in my reasoning. If we push the database in a direction where it's harder for submitters to find potential duplicates before filing (because that's really what we're talking about here), we'll need more moderators to hunt those duplicates.

History shows that we haven't been moderating much over the last few years (how's that for an understatement?)

Can we break that deadlock with some strongly worded documentation? Expecting that people will read the documentation sounds horribly optimistic, I know, I should know better.

JBQ

[Al Sutton]

Relying on developers to ready documentation… Hmmmm… How can I put this…. ;)

One interesting approach I've seen is to introduce a multi-step issue submission process. Step 1 allows the entry of a one line title, step 2 shows a list of potential matches for a search on the keywords in that title and allows the reporter to +1/star/follow one of the search results or continue on to filling in a new bug report. That way it forces reporters through a search rather than letting them put in a new bug without checking for a duplicate.

To me blocked isn't the same as closed. Blocked means "We know it may still be an issue but the fix is beyond our control", whereas Closed means "There is no further work to be done", so it signifies to reports the difference between a known issue (which should be re-reported), and an issue which is believed resolved (and so should be re-reported/re-opened if the resolution doesn't work). 

I can understand that having lingering Blocked bugs is less than ideal, but I can see value in differentiating between the two states.

[Jean-Baptiste Queru]

I need to look into what the Chromium team did to implement a wizard in front of the bug submission form. I think that wizard might have required some black magic.

You're starting to convince me about Blocked issues, at least for a certain category: the theoretically fixable issues that exist when running the AOSP master on supported hardware (but can't be fixed via AOSP submissions because the fix is in proprietary files). That way, we don't drown under bugs related to unsupported 3rd-party hardware, and such issues eventually time out when the matching hardware falls out of AOSP support. Implicitly, I think that those issues need to be filed in a hardware-specific way.

I'd go with that.

JBQ

[Al Sutton]

Sounds good to me.