organising github-issues

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

The list of issue on https://github.com/QubesOS/qubes-issues is growing,
and it's getting harder and harder to maintain. Let me first build some
context and explain the situation. Specific questions are at the end of
the email.

Majority of tickets falls into one of two categories:

1. bugs (something is supposed to work, but it doesn't) - label "T: bug"
(660 issues)

2. features (something isn't supposed to work, but it would be useful to
have it working) - label "T: enhancement" (587 issues)

Lets go deeper:

1. Bugs - I can name a few sub-categories:

1.1. Purely about qubes software packages. This obviously belongs to
github-issues.

1.2. Incompatibilities with some software within a VM. This in many
cases indeed is about some missing feature an application expects. A
recent example[1] is Zoom that tries to show a big transparent window.
But since Qubes (intentionally) does not support VM-controlled window
transparency, the result is quite bad (a window covering the whole
screen).

1.3. Incompatibilities with some hardware.

1.4. Bugs in 3rd-party software we happen to ship with Qubes (as part of
distributions we use).

1.5. Bugs not diagnosed enough to assign to any category. This includes
also bugs that are hard to reproduce (happens rarely, for very few
users).


2. Features - here I don't have such elaborate categories, but still can
define some, based on their state nor really their nature:

2.1. Feature requests that makes sense (improve usability or security or
both), but are unlikely to be implemented by the core developers team in
a foreseeable future.

2.2. Features that are in progress and/or are planned for a specific
release. This includes both those developed by the core team, or
contributed by community member (or both).

## How to organize them ##

The main reason for this email is category 2.1. The great majority of
those feature requests won't be implemented ever. But also most are
quite good material for community contribution and we do have many
excellent examples of such, like:
- btrfs/reflink storage pool driver
- Archlinux template
- generic audio support for HVM domains including Windows (already
implemented but not released yet)

Generally we have those marked with a milestone "TBD" (previously "Far
in the future"). But I wonder if we can do better. The current approach
means the list of issues will grow indefinitely by design. This could
work fine _if we'd have a method to manage them_ - things like
predefined filters, more specific issues metadata to filter on. This BTW
may mean github is too limited for this use case.
But if we won't have easy and convenient way to manage this issues
category, I think they shouldn't be keep open in qubes-issues. I mean,
either close (those extremely unlikely to be implemented) or move to
another tracker (which may be another github repo, or something else).

To give you an idea what I'm talking about, lets take a look at the very
recent feature request "Add VeraCrypt and alternative encryption
support"[2]. This, as even the author notices, would be useful to
extremely small part of qubes users (which already is very small subset
of population). But also, for a determined contributor, shouldn't also
be very hard to implement. Maybe this can be even reduced to a
documented manual steps how to configure such thing.

Another criteria could be where such feature could live. Some things
require changes in core qubes components, but some can be easily
implemented as external packages. And since not long ago, we (finally)
have a separate packages repository for such contributions. Features
(and perhaps also bugs in them?) targeting qubesos-contrib repository
could be tracked in a separate place maybe?

Separating things into different places have also downsides: it greatly
reduce discoverability. This may result in even higher entry barrier for
contributors, but also in more duplicates. This is actually the main
reason why we have a common qubes-issues repository, instead of tracking
everything in appropriate code repository (leaving alone the fact that
some issues are about multiple repositories at once).

Now back to bugs categories.

Hardware support related issues are tricky to track - some are really
about generic Linux support (which then falls into category 1.4), but
some are triggered by a specific configuration we use in Qubes OS (which
also includes Xen). In some cases, it is hard to tell, especially as
reproducing an issue requires a specific hardware. Because of this,
generally we do not track problems with very specific hardware in
qubes-issues, unless affect significant user base (popular hardware,
affects a whole family of devices etc). Those more localized issues we
prefer to discuss just on qubes-users ML.
My main issue with this approach is lack of convenient method to get
steps required to get Qubes working on a specific machine. Hardware
Compatibility List serves as a place to get broad support status
(works/doesn't work/something in between), but if any extra steps are
needed, one needs to dig through (potentially lengthy) thread on a
mailing list. Or multiple of them. It would be useful to have a
convenient place for a summary of such thread(s), that can be easily
updated. A wiki page perhaps (either in github or somewhere else)?
This is IMO very similar to community documentation, so maybe some
relation to https://github.com/Qubes-Community/ could work?

Bugs in 3rd-party software generally we forward to upstream bug tracker
and close in qubes-issues. But it is still helpful to track that
upstream issue, especially when it affects some important function in
Qubes (like recently salt issues in Fedora 32, affecting template
updates[3]). Some issue trackers (bugzilla for example) have a feature of
tracking external bug references, but not github.
Note this is just about issue tracking, not necessary providing actual
fixes - in many cases we do help upstream projects fixing issues that
affect Qubes.

Bugs that are not diagnosed enough to be actionable also noticeable part
of github-issues. This makes it harder to find issues that can be worked
on.

## Specific questions/ideas ##

I think some of the above problems can be solved with combination of
more labels ("more-info-needed"? "diagnosed"? using more "help wanted"?)
and some predefined filters. Filters like "show bugs affecting current
stable release that have all the info required to fix it", or "show
features planned for the next release".

While preparing such filters isn't really hard, I have no idea how to
make them discoverable and easy to use. Any ideas?

Also, maybe someone have some ideas for better labels that would allow
easier filtering? For example I find priorities label hard to use,
because it doesn't allow listing for example "issues with priority major
or higher".

There is also "projects" feature on github. Some of us use it for medium
size task groups (too big for a single issue, but still small enough to
not result in unusable long list). But I'm not sure if it helps with
anything I mentioned in this email.


[1] https://github.com/QubesOS/qubes-issues/issues/5863
[2] https://github.com/QubesOS/qubes-issues/issues/5875
[3] https://github.com/QubesOS/qubes-issues/issues/5791

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl7YbdYACgkQ24/THMrX
1ywccQf+Oy7RLrabqM1yFenv/BMaJL/vplmsx3NkRXmhLkoij0Z1tapKz7si2QDS
FcCNs9eM7JViOrFO9JvOXxlAYreor05vTHWrIiWHTymMx78ryZ/AXbhA2ZThdL5g
ebmI5aqpc6t5Ras0lzweZNc60Cc20vJjiEFNq+jBYpoZnwAo85fTcC2nXTVQePQH
pb2gVp+c9zZqkacFWQv8/9INQ0/uMEk6Ig4mBBhhtqvD9q1Ong9x6HWToD2QtrLM
ftN8i4EWkQ7fVRw1RE0DBm5SrussWHI43sFbKIuf5dUiH3FPwRLYZpiZZ8wCz1oP
gJoj4TCyye/FM4A1bewUgCdWVGMDXQ==
=+ogv
-----END PGP SIGNATURE-----

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 2020-06-03 10:43 PM, Marek Marczykowski-Górecki wrote:
> Hi all,
>
> The list of issue on https://github.com/QubesOS/qubes-issues is
> growing, and it's getting harder and harder to maintain. Let me
> first build some context and explain the situation. Specific
> questions are at the end of the email.
>

Hi Marek,

Thanks for raising this topic!

TL;DR: I think we can handle this with more labels and links to
pre-defined filters, but I'm open to other methods.

> Majority of tickets falls into one of two categories:
>
> 1. bugs (something is supposed to work, but it doesn't) - label "T:
> bug" (660 issues)
>
> 2. features (something isn't supposed to work, but it would be
> useful to have it working) - label "T: enhancement" (587 issues)
>

Yes. I also count a third type: "T: task" which is for an action item
that is neither a bug nor an enhancement.

> Lets go deeper:
>
> 1. Bugs - I can name a few sub-categories:
>
> 1.1. Purely about qubes software packages. This obviously belongs
> to github-issues.
>
> 1.2. Incompatibilities with some software within a VM. This in
> many cases indeed is about some missing feature an application
> expects. A recent example[1] is Zoom that tries to show a big
> transparent window. But since Qubes (intentionally) does not
> support VM-controlled window transparency, the result is quite bad
> (a window covering the whole screen).
>
> 1.3. Incompatibilities with some hardware.
>

(We use the "hardware support" label for these.)

> 1.4. Bugs in 3rd-party software we happen to ship with Qubes (as
> part of distributions we use).
>
> 1.5. Bugs not diagnosed enough to assign to any category. This
> includes also bugs that are hard to reproduce (happens rarely, for
> very few users).
>
>
> 2. Features - here I don't have such elaborate categories, but
> still can define some, based on their state nor really their
> nature:
>
> 2.1. Feature requests that makes sense (improve usability or
> security or both), but are unlikely to be implemented by the core
> developers team in a foreseeable future.
>

(We have specific topic labels, like "security" and "UX." We also have
"help wanted" for things the core devs don't plan to tackle.)

> 2.2. Features that are in progress and/or are planned for a
> specific release. This includes both those developed by the core
> team, or contributed by community member (or both).
>
> ## How to organize them ##
>
> The main reason for this email is category 2.1. The great majority
> of those feature requests won't be implemented ever. But also most
> are quite good material for community contribution and we do have
> many excellent examples of such, like: - btrfs/reflink storage pool
> driver - Archlinux template - generic audio support for HVM domains
> including Windows (already implemented but not released yet)
>
> Generally we have those marked with a milestone "TBD" (previously
> "Far in the future"). But I wonder if we can do better.

Actually, I see the "help wanted" label and the "Far in the
future"/"TBD" milestone as quite distinct:

- One is a label; the other is a milestone
- "Help wanted" is about *who* will do it (core vs. community)
- "Far in the future"/"TBD" is about *when* it will be done

For example, we can have something that the core devs will do, but no
idea when they'll have time to do it (doesn't have "help wanted" but
is on the "TBD" milestone).

Alternatively, we might know it's something that the core devs will
never do, but a community contributor is already working on it for the
next release. In this case, "help wanted" would be replaced with
"community-dev" + a status ("S: [...]") label, and it would be on the
milestone for the target release.

> The current approach means the list of issues will grow
> indefinitely by design. This could work fine _if we'd have a method
> to manage them_ - things like predefined filters, more specific
> issues metadata to filter on. This BTW may mean github is too
> limited for this use case.

Generally speaking, my philosophy is that it's up to us to create an
organizational system out of generic building blocks (like labels)
rather than relying on the issue tracker itself to build in a
particular system of organization.

> But if we won't have easy and convenient way to manage this issues
> category, I think they shouldn't be keep open in qubes-issues. I
> mean, either close (those extremely unlikely to be implemented) or
> move to another tracker (which may be another github repo, or
> something else).
>

I think it would be reasonable to adopt a more aggressive policy about
closing issues. After all, closing an issue is not permanent. It's
easy to reopen an issue with a single click. Moreover, closed issues
are just as searchable as open issues. They also retain their labels
and can still have labels added to and removed from them.

> To give you an idea what I'm talking about, lets take a look at the
> very recent feature request "Add VeraCrypt and alternative
> encryption support"[2]. This, as even the author notices, would be
> useful to extremely small part of qubes users (which already is
> very small subset of population). But also, for a determined
> contributor, shouldn't also be very hard to implement. Maybe this
> can be even reduced to a documented manual steps how to configure
> such thing.
>

In that case, I would want to classify it as a documentation issue and
keep it open (somewhere). Otherwise, it would never get done. (Of
course, it may never get done even if we keep it open, but still.)

> Another criteria could be where such feature could live. Some
> things require changes in core qubes components, but some can be
> easily implemented as external packages. And since not long ago, we
> (finally) have a separate packages repository for such
> contributions. Features (and perhaps also bugs in them?) targeting
> qubesos-contrib repository could be tracked in a separate place
> maybe?
>
> Separating things into different places have also downsides: it
> greatly reduce discoverability. This may result in even higher
> entry barrier for contributors, but also in more duplicates. This
> is actually the main reason why we have a common qubes-issues
> repository, instead of tracking everything in appropriate code
> repository (leaving alone the fact that some issues are about
> multiple repositories at once).
>

I agree. I think the benefits of having all issues in one place
outweigh the costs, but I can see the arguments on both sides.

In general, I suppose it depends on how likely people are to go to the
right place when creating and searching for that type of issue. For
example, if almost everyone who wants to create or search for a
qubesos-contrib issue would identify the separate issue tracker in
that repo as the correct place, then it'd be worth branching it off.
Perhaps everyone thinks "I want to add (or I want someone else to add)
a brand new feature to Qubes" (as opposed to "I want this thing that
already exists in Qubes to be improved"), so their mental model leads
them to that separate issue tracker. On the other hand, I could see
this failing, since many people who wish to request a new feature
won't know whether it would be implemented by the core devs or
community devs. (We often don't know ourselves.)

> Now back to bugs categories.
>
> Hardware support related issues are tricky to track - some are
> really about generic Linux support (which then falls into category
> 1.4), but some are triggered by a specific configuration we use in
> Qubes OS (which also includes Xen). In some cases, it is hard to
> tell, especially as reproducing an issue requires a specific
> hardware. Because of this, generally we do not track problems with
> very specific hardware in qubes-issues, unless affect significant
> user base (popular hardware, affects a whole family of devices
> etc). Those more localized issues we prefer to discuss just on
> qubes-users ML.

Yes, although lately I've been leaving more of these issues open (with
the "hardware support" label), for a few reasons:

- In many cases, your expertise is required to determined whether it is
actually an installer bug or a localized hardware problem.
- In many cases, addressing these issues seems to lead to general-good
bug fixes or installer improvements.
- Since broader hardware compatibility is one of our goals, I figure
that we probably want these problems to be fixed eventually.

However, I'm happy to implement whatever policy you think would be
best regarding these issues.

> My main issue with this approach is lack of convenient method to
> get steps required to get Qubes working on a specific machine.
> Hardware Compatibility List serves as a place to get broad support
> status (works/doesn't work/something in between), but if any extra
> steps are needed, one needs to dig through (potentially lengthy)
> thread on a mailing list. Or multiple of them. It would be useful
> to have a convenient place for a summary of such thread(s), that
> can be easily updated. A wiki page perhaps (either in github or
> somewhere else)? This is IMO very similar to community
> documentation, so maybe some relation to
> https://github.com/Qubes-Community/ could work?
>

That's a good idea. Perhaps we could have a written template that the
owner of the hardware could fill out. We could use a GitHub Issue
template for this and put them in a separate repo's issue tracker so
that they don't clutter up qubes-issues.

> Bugs in 3rd-party software generally we forward to upstream bug
> tracker and close in qubes-issues. But it is still helpful to track
> that upstream issue, especially when it affects some important
> function in Qubes (like recently salt issues in Fedora 32,
> affecting template updates[3]). Some issue trackers (bugzilla for
> example) have a feature of tracking external bug references, but
> not github.

Ok, I agree that this is a good example of a limitation of the issue
tracker itself.

> Note this is just about issue tracking, not necessary providing
> actual fixes - in many cases we do help upstream projects fixing
> issues that affect Qubes.
>
> Bugs that are not diagnosed enough to be actionable also noticeable
> part of github-issues. This makes it harder to find issues that can
> be worked on.
>

I think the main bottleneck here is the limited time of those with the
expertise required to perform such diagnoses.

> ## Specific questions/ideas ##
>
> I think some of the above problems can be solved with combination
> of more labels ("more-info-needed"? "diagnosed"? using more "help
> wanted"?) and some predefined filters. Filters like "show bugs
> affecting current stable release that have all the info required to
> fix it", or "show features planned for the next release".
>

Good idea! I've created these and will start using them:

- "more-info-needed"
- "needs-diagnosis"
- "diagnosed"

Let's experiment with them a bit and see how they work out in
practice. If anyone has suggestions for more like these, let me know.

"show features planned for the next release" we can already do by
searching for the label "T: enhancement" + the next release's milestone.

> While preparing such filters isn't really hard, I have no idea how
> to make them discoverable and easy to use. Any ideas?
>

I could add a list of pre-defined filter searches to:

https://www.qubes-os.org/doc/reporting-bugs/

> Also, maybe someone have some ideas for better labels that would
> allow easier filtering? For example I find priorities label hard to
> use, because it doesn't allow listing for example "issues with
> priority major or higher".
>

It's a bit hacky, but technically doable. GitHub Issues doesn't allow
searching with the logical operator "OR", but it does allow excluding
labels by prepending a minus sign ("-"). Since every open issue that
I've triaged should have a priority label ("P:[...]"), we can exclude
all the priority labels we don't want. So, to use your example, the
search for "issues with priority major or higher" would be:

is:open is:issue -label:"P: minor" -label:"P: default"

> There is also "projects" feature on github. Some of us use it for
> medium size task groups (too big for a single issue, but still
> small enough to not result in unusable long list). But I'm not sure
> if it helps with anything I mentioned in this email.
>
>
> [1] https://github.com/QubesOS/qubes-issues/issues/5863 [2]
> https://github.com/QubesOS/qubes-issues/issues/5875 [3]
> https://github.com/QubesOS/qubes-issues/issues/5791
>
>

- --

Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAl7Y5noACgkQ203TvDlQ
MDBzDw//UdNbijAUJZ2mF38XJtCxSLPQBCucoaXrBGXIsxcCRb/yR94IQLcfmpa9
9B84veITZxCwji4rh64t3hGi54vqDLWV+yMhBRjh3YfuCPl3cEs1cZJqb7LtMF6J
IaSAiID2XmQtUToEH0gZbOIm3vEJpRv4YG3Ry8QTS5kz67lPjqfB7iXZdkLxMMpw
Zmw8dIOghuXBaFvswAEfzZGwNY7EwT6gofN2j8BuLCOJrFcmX89zygu9cfU3AI8U
1cJuSXCndks3uiHVsTomStcs5SgBNSMY2skIoCVdaX2xzThksKJGdqDmrps3KkhF
x+Zp7lH6NX+Iuw+OxZkYJSnHryvkM58jEph+7PLVPqlVObnlHyT7GwHtE4EllrDl
GVzhfAeuCWQarST85w73nqiXcZC5cw1Mu7/uYwlZMoPHHPq6U9nFOtLkHfmACpsc
/T1gQnyOZ19W+eRqSU4Ihk+pYrIgvysqs0tC2vhg0PuHF0AqX77n4zkjZJEn/fDu
btJv2jpsY5CHJEUz4TVbxGw7N5tnh5SphaQFQR3ktt8dp+JQw2ydeKzuM8ju1IE4
KYZcVyTJsFItMFwQDRKiulENgUITrEMWvhWlXZRzDbGWClGc+LOfcpmDcjPMeiiM
SjRXK80Fxz97goTH3D9gYQEopIt1a6ogbu6ReyTT2gJDxT+HOF4=
=YLTD
-----END PGP SIGNATURE-----