Announcing Project Bugzilla Harmony

[Dylan Hardison]

This is my long-planned plan for improving the Bugzilla project;
Bugzilla Harmony is a boring thing that we've got to do to reverse the stagnation of development against upstream.
Later you may hear more about something called Bugzilla Quantum, which is set of ambitious UX changes to BMO.
This should make it even more obvious why we seek to harmonize the two codebases.

I'm not going to go into any low-level details about code, but about process.
I hope that I can get many people to make very small contributions towards this goal.

Goal:
Harmonize the upstream Bugzilla code to track bugzilla.mozilla.org

Requirements:

1. Significant changes to BMO must *not* be required.
1.1. No requiring perl 5.14, though nothing prevents us from working *better* on modern perls.
1.2. The usernames != login patch has to be backed out (someone started this, and I've asked gerv to review it on GitHub)
1.3. An exception to this rule is for SQL queries that only work on MySQL; BMO will change as is required to support all our supported backends.
2. We are going to follow https://rfc.zeromq.org/spec:42/C4/ with the understanding that "Platform issue tracker" refers to bugzilla.mozilla.org
3. All this work will happen to the "harmony" branch, and we'll continue to not touch 'master' to not break anyone that happens to be using it.


Packaging / Release:

We are not in the business of packaging, there are too many combinations to worry about all of them.
People are welcome to help with packaging Bugzilla 6 / Harmony on anything, and nothing drastic will change
there. But as for releasable products, there are two channels that we must support:

1. A runnable docker container, based on Alpine linux
2. A checkout from a tagged commit from GitHub.

The release process for Bugzilla Harmony will amount to tagging a commit (from the Harmony branch, more on that later).
The release notes should be exactly the commit messages. Part of reviewing a PR is making sure the commit messages tell the appropriate story.


Commits / Reviews / Pull Requests:

Read the C4 document, that is what contributing to Bugzilla's harmony branch should be like.

Tasks:
Some of these might even be fun.

1. Revert login_name != email (in favor of a much more general approach that we'll be making in BMO)
2. Revert multiple bug aliases (which are identical to keywords in practice)
3. Keep PSGI/Plack support -- one of the greatest features of the master branch
4. Revert the REST API in favor of the (older) system that BMO is still using.
5. Ensure a database created by 4.2, 5.0, or the master branch can be "upgraded" to from the harmony branch.
6. Move all tests to Circle CI
7. Make it easier to write selenium / web driver tests
8. Ensure all the fixes for memory leaks in BMO get ported over
10. Port GitHubAuth (this has already been done!)
11. Port MFA (two-factor authentication)
12. Port docker / containerization code
13. Change above container code to be based on Alpine instead of CentOS 6.9
14. Make bugzilla deployable to multiple cloud infrastructures.



-
To view or change your list settings, click here:
<https://lists.bugzilla.org/cgi-bin/mj_wwwusr?user=dev-apps...@lists.mozilla.org>

[Gervase Markham]

Hi Dylan,

Thanks for this - it's good to see some progress on defining what we are
going to be doing :-)

On 30/11/17 20:36, Dylan Hardison wrote:
> 1. Significant changes to BMO must *not* be required.
> 1.1. No requiring perl 5.14, though nothing prevents us from working *better* on modern perls.

What are the current Perl requirements for stable, trunk and BMO?

What is the objection to updating the version of Perl we require?

> 1.2. The usernames != login patch has to be backed out (someone started this, and I've asked gerv to review it on GitHub)

Remind us all why this needs to happen?

Have we shipped this code to anyone?

> 2. We are going to follow https://rfc.zeromq.org/spec:42/C4/ with the understanding that "Platform issue tracker" refers to bugzilla.mozilla.org

What is your goal here? What about our current development model is
broken and needs fixing? Are there specific bits of this rather long
document which you are particularly concerned we adopt?

> 3. All this work will happen to the "harmony" branch, and we'll continue to not touch 'master' to not break anyone that happens to be using it.

I assume "harmony" will be a branch in http://github.com/bugzilla/bugzilla?

Will "harmony" start from master, stable, or from the BMO codebase?

I was rather expecting to start with the BMO code and work back towards
Bugzilla compatibility, rather than starting with master and working
towards BMO compatibility. The former certainly seems like it would be
easier to do...

> We are not in the business of packaging, there are too many combinations to worry about all of them.
> People are welcome to help with packaging Bugzilla 6 / Harmony on anything, and nothing drastic will change
> there. But as for releasable products, there are two channels that we must support:
>
> 1. A runnable docker container, based on Alpine linux

I have no particular objection, but why Alpine Linux? What is that?

> 2. A checkout from a tagged commit from GitHub.
>
> The release process for Bugzilla Harmony will amount to tagging a commit (from the Harmony branch, more on that later).

By "release process", do you mean "the process which result in the thing
we call e.g. Bugzilla 6.0"? Or are you talking about some testable
milestones we might ship along the way?

> The release notes should be exactly the commit messages. Part of reviewing a PR is making sure the commit messages tell the appropriate story.

I don't think this particular point is going to fly. It means you can't
really tell any story which spans multiple commits, and it means you
won't be picking out particular features as worth promoting. Doing
things this way will just result in pretty useless release notes. I
understand and agree with your desire to reduce the work involved in
shipping a release, but I think this is an over-economy.

> Commits / Reviews / Pull Requests:
>
> Read the C4 document, that is what contributing to Bugzilla's harmony branch should be like.

That is a somewhat intimidating statement; see my question above. We
have been discussing a significant change to _what_ we are building; a
significant change to _how_ we are building it is somewhat of a bolt
from the blue. I would very much appreciate it if you were to explain
all this a bit more.

> 4. Revert the REST API in favor of the (older) system that BMO is still using.

Why is that necessary? Cannot a new, improved API and the old one live
in parallel?

> 5. Ensure a database created by 4.2, 5.0, or the master branch can be "upgraded" to from the harmony branch.

I think you mean the other way around :-)

> 10. Port GitHubAuth (this has already been done!)

Surely the harmony branch needs eventually to support all BMO's extensions?

Gerv

[Denis Roy]

This is great news! I will follow along and contribute however I can,
with the very few cycles that I have :(

Some comments below:

On 12/01/2017 08:20 AM, Gervase Markham wrote:
>
>> Commits / Reviews / Pull Requests:
>>
>> Read the C4 document, that is what contributing to Bugzilla's harmony branch should be like.
> That is a somewhat intimidating statement; see my question above.

+1 Not sure I follow.

>> 4. Revert the REST API in favor of the (older) system that BMO is still using.
> Why is that necessary? Cannot a new, improved API and the old one live
> in parallel?

I agree, we're thrilled to see the REST API and plan on building
components around it.


Thanks for pushing this forward!

Denis

[Gervase Markham]

On 01/12/17 13:20, Gervase Markham wrote:
> I assume "harmony" will be a branch in http://github.com/bugzilla/bugzilla?
>
> Will "harmony" start from master, stable, or from the BMO codebase?
>
> I was rather expecting to start with the BMO code

It seems like the harmony repo does do this, which is great. I
definitely think it's the right way to go. But a lot of the things you
listed as "needing doing" need doing from the perspective of converting
trunk into BMO. E.g. "back out the login name != email address patch".
Do we need to make a new list from the reverse perspective?

Gerv

[Dave Miller]

Here's another one I found stuck in the moderator queue when I went
looking to see what happened to Dylan's other post:

Subject:
Re: Announcing Project Bugzilla Harmony
From:
Dylan Hardison <dy...@mozilla.com <mailto:dy...@mozilla.com>>
Date:
Fri, 1 Dec 2017 10:23:57 -0500

To:
devel...@bugzilla.org <mailto:devel...@bugzilla.org>

> On Dec 1, 2017, at 09:56, Denis Roy <deni...@eclipse-foundation.org> wrote:
>
> This is great news! I will follow along and contribute however I can,
> with the very few cycles that I have
>

> Some comments below:
>
>
> On 12/01/2017 08:20 AM, Gervase Markham wrote:

>>> Commits / Reviews / Pull Requests:
>>>
>>> Read the C4 document, that is what contributing to Bugzilla's harmony branch should be like.
>> That is a somewhat intimidating statement; see my question above.

> +1 Not sure I follow.

>>> 4. Revert the REST API in favor of the (older) system that BMO is still using.
>> Why is that necessary? Cannot a new, improved API and the old one live
>> in parallel?

The "new" one is just a code migration from the existing one. It is "mostly" API compatible,
but we couldn't use it on BMO as there were too many differences. It is required for harmony to leave the REST API that is used on BMO alone.

To support the work Kohei is doing on the UX side, I will add an /api/ or /rest2/ endpoint
and provide new features there -- completely separate from the JSONRPC/XMLRPC/REST framework that exists now.

It is really easy to just drop in a new .cgi and support REST directly, and leave the (BMO/5.0/etc) REST system in place.

All work on APIs follows this guideline from the C4 process:

2.6. Evolution of Public Contracts

• All Public Contracts (APIs or protocols) SHALL be documented.
• All Public Contracts SHOULD have space for extensibility and experimentation.
• A patch that modifies a stable Public Contract SHOULD not break existing applications unless there is overriding consensus on the value of doing this.
• A patch that introduces new features SHOULD do so using new names (a new contract).
• New contracts SHOULD be marked as "draft" until they are stable and used by real users.
• Old contracts SHOULD be deprecated in a systematic fashion by marking them as "deprecated" and replacing them with new contracts as needed.
• When sufficient time has passed, old deprecated contracts SHOULD be removed.
• Old names SHALL NOT be reused by new contracts.

[Dave Miller]

This got forwarded to me, I don't see that it ever made it to the
mailing list, not sure where it ended up. But need to get these replies
out there.
*
From: *Dylan Hardison <dy...@mozilla.com <mailto:dy...@mozilla.com>>
*Subject: **Re: Announcing Project Bugzilla Harmony*
*Date: *December 1, 2017 at 10:50:29 EST
*To: *devel...@bugzilla.org <mailto:devel...@bugzilla.org>

>
> On Dec 1, 2017, at 08:20, Gervase Markham <ge...@mozilla.org

> <mailto:ge...@mozilla.org>> wrote:
>
> Hi Dylan,
>
> Thanks for this - it's good to see some progress on defining what we are
> going to be doing :-)
>
> On 30/11/17 20:36, Dylan Hardison wrote:
>> 1. Significant changes to BMO must *not* be required.
>> 1.1. No requiring perl 5.14, though nothing prevents us from working
>> *better* on modern perls.
>
> What are the current Perl requirements for stable, trunk and BMO?
>
> What is the objection to updating the version of Perl we require?
>

We declare a min version in our code, and this declaration changes the
syntax to some degree.
For instance, in trunk we have use 5.14.0, which also enables strict
mode too (so use strict is not needed).

BMO is targeting 5.10.1. It also runs under 5.16 quite well, because
5.16 is used by version-control-tools,
and it also runs under 5.24 and 5.26 just fine. But we can't rely on
features from those until BMO production is migrated.
In about six months I suspect BMO will be running on alpine linux and
perl 5.26.

I very much want to use only the supported (by the perl community)
versions of perl, which is just "the last two versions",
but the reality is that I can't upgrade overnight.

So for harmony, I'd like to have the minimum version be 5.10.1.

I plan on porting https://github.com/mozilla-bteam/bmo-systems
<https://github.com/mozilla-bteam/bmo-systems> to upstream bugzilla as
well, and

>> 1.2. The usernames != login patch has to be backed out (someone
>> started this, and I've asked gerv to review it on GitHub)
>
> Remind us all why this needs to happen?
>
> Have we shipped this code to anyone?

People running the master branch are definitely using it.

The problems are a few:

it's possible for the DB to be out of sync with the parameter,
and if you're registering users when out of sync "strange" things could
happen.

For BMO the plan is to merely lax the requirement on the login_name
field, after adding (multiple) email addresses to the account.
This is similar to what many other platforms do.

https://bugzilla.mozilla.org/show_bug.cgi?id=1372631
<https://bugzilla.mozilla.org/show_bug.cgi?id=1372631>

>
>> 2. We are going to follow https://rfc.zeromq.org/spec:42/C4/
>> <https://rfc.zeromq.org/spec:42/C4/> with the understanding that
>> "Platform issue tracker" refers to bugzilla.mozilla.org

>> <http://bugzilla.mozilla.org>

>
> What is your goal here? What about our current development model is
> broken and needs fixing? Are there specific bits of this rather long
> document which you are particularly concerned we adopt?
>
>> 3. All this work will happen to the "harmony" branch, and we'll
>> continue to not touch 'master' to not break anyone that happens to be
>> using it.
>
> I assume "harmony" will be a branch in
> http://github.com/bugzilla/bugzilla?
> <http://github.com/bugzilla/bugzilla?>
>
> Will "harmony" start from master, stable, or from the BMO codebase?
>
> I was rather expecting to start with the BMO code and work back towards
> Bugzilla compatibility, rather than starting with master and working
> towards BMO compatibility. The former certainly seems like it would be
> easier to do...
>
>> We are not in the business of packaging, there are too many
>> combinations to worry about all of them.
>> People are welcome to help with packaging Bugzilla 6 / Harmony on
>> anything, and nothing drastic will change
>> there. But as for releasable products, there are two channels that we
>> must support:
>>
>> 1. A runnable docker container, based on Alpine linux
>
> I have no particular objection, but why Alpine Linux? What is that?

A very small linux distro and the de-facto base image used by
high-quality docker containers.

>
>> 2. A checkout from a tagged commit from GitHub.
>>
>> The release process for Bugzilla Harmony will amount to tagging a
>> commit (from the Harmony branch, more on that later).
>
> By "release process", do you mean "the process which result in the thing
> we call e.g. Bugzilla 6.0"? Or are you talking about some testable
> milestones we might ship along the way?
>

Only the former. I'm going for continuous releases. I'm never going to
have time for the other release process.

>> The release notes should be exactly the commit messages. Part of
>> reviewing a PR is making sure the commit messages tell the
>> appropriate story.
>
> I don't think this particular point is going to fly. It means you can't
> really tell any story which spans multiple commits, and it means you
> won't be picking out particular features as worth promoting. Doing
> things this way will just result in pretty useless release notes. I
> understand and agree with your desire to reduce the work involved in
> shipping a release, but I think this is an over-economy.

You can ignore a subset of commits -- I use this to build the
announcements for BMO for the last 4 years.
I also think with a continuous release process it is less interesting.

We can say "pushing built images is not a release", but these
non-releases are the only thing I'm committed to doing or supporting.

>
>> Commits / Reviews / Pull Requests:
>>
>> Read the C4 document, that is what contributing to Bugzilla's harmony
>> branch should be like.
>
> That is a somewhat intimidating statement; see my question above. We
> have been discussing a significant change to _what_ we are building; a
> significant change to _how_ we are building it is somewhat of a bolt
> from the blue. I would very much appreciate it if you were to explain
> all this a bit more.
>

The existing process has not worked, it's burdensome. The C4 process is
effective
for a variety of successful projects.

That document gives clear expectations for a lot of situations and
encourages patches to actually land.
I think that's something important.

>> 4. Revert the REST API in favor of the (older) system that BMO is
>> still using.
>
> Why is that necessary? Cannot a new, improved API and the old one live
> in parallel?
>

(explained upthread)

>> 5. Ensure a database created by 4.2, 5.0, or the master branch can be
>> "upgraded" to from the harmony branch.
>
> I think you mean the other way around :-)

yeah.

>
>> 10. Port GitHubAuth (this has already been done!)
>
> Surely the harmony branch needs eventually to support all BMO's
> extensions?
>

Absolutely!

The reason harmony is starting from 5.0 is that the other major
stake-holder is running (or will be shortly) 5.0 (RedHat).
I hope they open up their fork's repo, and I will continue to hope for that.
I'm already going to be asking them if they can back-out on multiple bug
aliases.

Handling the conflicts between BMO and 5.0 is a burden I am able to
bear, so I won't ask my RedHat friends to shoulder it.