Retrospective: Regressions in Jenkins 2.204.3...2.204.5

[Oleg Nenashev]

Dear all,

We had serious regressions in the 2.204.x LTS baseline starting from 2.204.3 which was released on Feb 28 (changelog). After that we had 2 out-of-order releases over a week, with 3 major regressions reported by Jenkins LTS users. These regressions were largely related to the embedded Jetty web container updates in the LTS baseline. In 2.204.5 we also had to reintroduce a defect in the LTS release, and it is also far from business as usual.

I have started a public retrospective document to restore the sequence of events and to discuss what we could do better in the next releases. Anyone is welcome to comment in the doc and to make suggestions there. If you are a Jenkins user affected by the issues, your feedback will be much appreciated as well. Thanks in advance for your time!

I also suggest organizing a recorded video call next week to discuss the retrospective feedback. If you are interested to participate, please vote for the meeting times here: https://doodle.com/poll/xrhpmsk9xpba2tfh

Just to emphasize, the purpose of the retrospective  is to improve our processes and to ensure better quality of LTS and weekly releases. The regressions were caused by numerous causes and they passed though all our quality gates, there is no sense to put blame on any Jenkins team. There are many contributors involved in the core maintenance, and we do our best to provide stable releases, including weekends and burning the midnight oil. Please refrain from putting a blame and let's focus on making Jenkins releases more stable.

P.S: We are always looking for more contributors and reviewers/maintainers in the Jenkins core. If you are interested to help with maintaining the kernel of the project with millions of developers using it daily, please let us know!

Best regards,

Oleg Nenashev

Jenkins Core Team

[Oleg Nenashev]

I plan to close the Doodle poll tomorrow.

If you want to participate, please vote. https://doodle.com/poll/xrhpmsk9xpba2tfh

Best regards,

Oleg

[Oleg Nenashev]

After some votes the winner is March 18th, 10AM UTC. I have added a meeting to the Jenkins calendar.

Join Zoom Meeting : https://zoom.us/j/875768974 

Find your local call-in number: https://zoom.us/u/afxZRrxiW 

The meeting will be recorded, and we will make sure to do meeting notes and to update the Google Doc.

Best regards,

Oleg

[Oleg Nenashev]

Hi all,

Today we had a retrospective call where we processed feedback and defined some action items. Thanks to everyone who participated in the retrospective and in the meeting! You can find all notes in the Google Doc, meeting video is published here: https://youtu.be/V1jD5XZ8_Jc

Key decisions we agreed on:

• Jenkins LTS backporting will be happening through pull requests. We will start from creating a single pull request per LTS baseline and may alter the approach later if needed
• We will be doing LTS baseline selection 2 weeks earlier in the LTS cycle to reduce overlap between LTS X.3 bacporting/testing and next baseline merge windows
• We will improve visibility of non-trivial backporting cases in Jenkins Jira. non-trivial-lts-backporting label was added to highlight such cases. Additionally, we will consider issue relationships ("causes" but also others), so we ask people to add these.
• We will improve visibility of release candidates by tagging release candidates through GitHub Releases. We will also use social media and Jenkins Jira comments to facilitate testing by users and contributors
• We will use a new dashboard for Jenkins core maintainers so that they can easily monitor regressions which may impact next LTS baselines

There are more smaller decisions and action items which are listed in the Google doc. We did not discussed all retrospective feedback, and we will discuss the rest in the follow-up mailing list threads. I will organize that over this and the next week. If you have any addition feedback or proposals, please do not hesitate to add them to the document.

Best regards,

Oleg