Jenkins 2.217 out-of-order release post-mortem

[Oleg Nenashev]

Hi all,

Today we had an out-of-order release of Jenkins 2.217. This release fixes the major regression in localization files (JENKINS-60822), and it also includes the extension finder regression fix (JENKINS-60816). Thanks a lot to Jon Brohauge and Jesse Glick for the fixes!

You may have noticed that the release includes a number of other changes including the beta Websocket support and a major rework of the JavaScript packaging (now we use Webpack!). Basically we released all the changes integrated into the master branch. It is not a common approach for out of order releases when we usually release regression fixes.I would like to explain the reasons:

• There might be a security release on Jan 29th (TBA by the security team if it happens), usually such releases target .2 LTS which is expected to land on that date. We want to have a stable Weekly release by this time
• There were important changes like Websocket Beta support in Remoting (JEP-222), and we would rather want to get some user feedback before the next LTS baseline selection on Feb 26th
• There is FOSDEM on Feb 01-02. Daniel Beck, Mark Waite and me will be traveling, And the next common weekly dates (Jan 26 and Feb 02) are not the best time for major changes. Delaying releases till Feb 09 drastically reduces the window before the new LTS baseline selection
• Last but not least, the changes were already in master and set expectations about a soon release. It is on me, because over the past months I switched from doing merges on Thursday/Friday to a more CI-alike approach with merging changes after getting sufficient reviews and 24hrs merge timeout. It caused mis-alignment with the security release process which expected merges later on the week/

Given all these factors, I asked Kohsuke to release a standard weekly ahead of the schedule. It will give us enough time to react if any new regressions happen. The release looks good so far! I will also make sure to document the current weekly integration process and to follow-up on aligning it with the security release. Since we onboard more maintainers to the Jenkins core, we need to move many established practices to the documented form.

I also suggest skipping the weekly on Jan 26 or at least minimizing the scope of the changes to low-risk changes and improvements. If you have write access to the Jenkins core repo, I kindly ask to be careful with merges. JENKINS-57888 is one of the fixes I would consider for it, but the PR is yet to be submitted. If you see any other changes which should land in the weekly, let's discuss them in this thread.

Sorry for any inconvenience,

Oleg

[Jesse Glick]

On Thu, Jan 23, 2020 at 6:03 PM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> If you see any other changes which should land in the weekly, let's discuss them in this thread.

https://issues.jenkins-ci.org/browse/JENKINS-60857 : certificates in Jetty