Forward-porting vs back-porting fixes

[Apollon Oikonomopoulos]

Hi all,

Now that we've officially released 3.0, it's time to look a bit at
stable release management. What I'd like to discuss in particular is the
possibility of switching from a forward-porting workflow for stable
updates to a back-porting one.

Ganeti traditionally relied on a forward-porting, merging workflow for
the stable branches: a bugfix would always target the *earliest* stable
branch it should be applied on, and stable branches would subsequently
be merged upwards and finally into master¹. So, to fix a bug that was
introduced for instance in 2.15, we would commit the fix on stable-2.15,
and then successively merge stable-2.15 into stable-2.16, stable-2.16
into stable-3.0 and stable-3.0 into master.

¹ This the workflow recommended by gitworkflows(7) under "Merging upwards".

This workflow has a couple of advantages. For a start, it makes sure
that nothing is left out of intermediate stable branches: by
consistently merging upwards, the fix will end up in all intermediate
stable releases. Additionally, the change is committed exactly once,
with a single commit ID, so it's really easy to perform queries of the
form "show me the earliest release that contains fix X from each stable
branch".

That said, it also comes with a couple of disadvantages. First of all,
it adds a lot of overhead on regular development by coupling development
with stable release engineering: you can't just fix things on master
right here and right now, you have to figure out when they were first
broken, commit there and up-merge and resolve conflicts many times
before your fix lands on master. With it comes that fact that often
changes to master will be introduced via merge commits, which eventually
breaks commit atomicity on master. Finally, it assumes that you know
your fix is complete and that it won't need further testing after
committing to a stable branch, which is of course not always the case.

The alternative workflow is to fix things on master, and then use `git
cherry-pick` to backport the fixes to the stable branches they are
needed on. This allows for relatively fast-paced development on master,
produces a more linear and easy-to-understand history without
introducing changes on merge commits, and allows for the changes to be
tested on master/CI before hitting a stable branch. On the other hand,
the same fix will be committed multiple times to the repository (once
for each branch it is included on).

The real question here is not about merging vs cherry-picking: the
answer to that is almost always "merge whenever you can" exactly because
you can keep better track of fixes. The question is really about
forward-porting vs back-porting, with the added note that back-porting
cannot be done via merging (because a merge from master would include
all newer changes along with the fix) and has to use cherry-picking
instead.

Most of the projects I know of out there have settled on back-porting
strategies; this includes the Linux kernel and HAProxy to name a few, as
well as Debian (where fixes in stable must have been tested in unstable
beforehand). This is done for various reasons, but I think the biggest
advantages are that it allows rigorous testing to happen once on master,
and it also decouples regular development from stable release
management. The latter is important not only because it reduces
development friction, but also because it provides the ability to
*delegate* stable release management and have stable development be
performed asynchronously.

I think we could try a back-porting strategy for the 3.0 stable release
cycle, in the hope that it allows for faster development on master and
simpler PR workflows. If we end up not liking it, we can always revert
to the forward-porting strategy without a problem. What do you think?

Regards,
Apollon

[Sascha Lucas]

Hi Apollon,

On Wed, 30 Dec 2020, Apollon Oikonomopoulos wrote:

> Now that we've officially released 3.0, it's time to look a bit at
> stable release management. What I'd like to discuss in particular is the
> possibility of switching from a forward-porting workflow for stable
> updates to a back-porting one.

Thanks a lot for your extensive comments, showing the pros and cons of
both possibilities. I can understand your arguments to prefer back-porting
and I trust your judgment. So from my point of view, let's go with
back-porting.

Thanks, Sascha.

[Iustin Pop]

On 2020-12-30 11:50:10, Apollon Oikonomopoulos wrote:
> Hi all,
>
> Now that we've officially released 3.0, it's time to look a bit at
> stable release management. What I'd like to discuss in particular is the
> possibility of switching from a forward-porting workflow for stable
> updates to a back-porting one.
>
> Ganeti traditionally relied on a forward-porting, merging workflow for
> the stable branches: a bugfix would always target the *earliest* stable
> branch it should be applied on, and stable branches would subsequently
> be merged upwards and finally into master¹. So, to fix a bug that was
> introduced for instance in 2.15, we would commit the fix on stable-2.15,
> and then successively merge stable-2.15 into stable-2.16, stable-2.16
> into stable-3.0 and stable-3.0 into master.

Minor correction: Ganeti originally used back-porting, then switched to
forward-porting during the 2.1 release or so. It is not technically
relevant, but since we're discussing history anyway…

Yes, back-porting was very painful during the 1.2/2.0 era.
Forward-porting was not chosen arbitrarily, but rather to solve a real
problem (that resulted in many production bugs).

[snip rest]

> I think we could try a back-porting strategy for the 3.0 stable release
> cycle, in the hope that it allows for faster development on master and
> simpler PR workflows. If we end up not liking it, we can always revert
> to the forward-porting strategy without a problem. What do you think?

I think back-porting is more practical, but if we do it, we should
ensure that the original commit ID is recorded (`git cherry-pick -x`),
so that at least we have a weak trail of record.

regards,
iustin