Jenkins Core Release automation

75 views
Skip to first unread message

Olblak

unread,
Apr 14, 2020, 10:32:01 AM4/14/20
to Jenkins Developers ML, 'Gavin Mogan' via Jenkins Infrastructure
Hi Everybody,

This project is getting to the end and I am looking for a last round of reviewers and testers before planning a weekly release with the new process.
I opened a pull request (link) with instructions about how to trigger a release but currently, it only covers weekly releases.
Our first objective is to successfully release weekly versions from https://release.ci.jenkins.io, then we'll focus on stable and security releases.

For security releases, the way I see it:
1) We add two security profile, one for weekly and the second one for the stable in order to trigger a release from jenkinsci-cert/jenkins:master  to https://repo.jenkins-ci.org/releases/
2) Trigger packaging job(weekly or stable profile),
3) We manually merge from jenkinsci-cert/jenkins:master to jenkinsci/jenkins:master

For stable releases:
1) We update the stable profile branch to 'stable-<version>', variable JENKINS_GIT_BRANCH)
2) We trigger stable release job
3) We trigger stable packaging job

Another topic that needs clarification is who should be able to trigger a release from release.ci.jenkins.io and who will trigger the different releases?

Cheers

Oleg Nenashev

unread,
Apr 14, 2020, 10:44:56 AM4/14/20
to jenkin...@googlegroups.com, Jenkins Developers ML
Thanks a lot to Olivier and all other contributors who invested lots of time to get it done! I am looking forward to finally get it over the line, at least for weeklies.

FWIW, the Sunday's weekly got delayed (due to Easter). We could make an attempt to cut this release using the new flow without waiting for the next weekly.

Best regards,
Oleg
 

--
You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com.

Olblak

unread,
Apr 14, 2020, 3:03:57 PM4/14/20
to Jenkins Developers ML, 'Gavin Mogan' via Jenkins Infrastructure
Today, during the infra meeting, we discussed creating a new weekly release from the new process this Thursday the 16th at 3PM UTC.
While there won't be major features, the purpose of this release is to validate that everything is working as expected in a real scenario.


I am proposing the following agenda


* 3PM(UTC), I start the release job and we release directly to the master branch and directly on repo.jenkins-ci.org/releases.
Currently the release job take approximatively 1h30


* 4:30PM(UTC), or early if ready, I start the packaging process in order to publish artifacts to pkg.jenkins.io
Currently, the packaging job take approximatively 15min to run and I estimate the script sync.sh to take 15min to run

* 5PM(UTC), we verify that the release is working correctly.

During the whole all process we'll be on IRC, #jenkins-infra channel

Again feel free to raise any concerns you may have.

Cheers
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Olblak

unread,
Apr 17, 2020, 5:32:47 AM4/17/20
to Jenkins Developers ML, 'Gavin Mogan' via Jenkins Infrastructure
Hi Everybody

Yesterday we released a weekly version from the new release environment and while we managed to ship our weekly version, we also faced unplanned issues that we had to solve so here a quick retro.

The process was done in two stages the release and then the packaging

Stage 1, Release:

The release part, was the easiest one. We only add two issues.
The first issue was due to the password used to authenticate on Maven repository, containing special characters that couldn't be parsed by Maven release plugin. We just changed the password.
The second problem was a permission issue on repo.jenkins-ci.org which was easily fixed.
In the result of this stage, as expected every artifact was pushed to repo.jenkins-ci.org

Stage 2, Package:

Well, this stage was more tricky as we heavily refactored that part while also heavily relying on existing service. We discovered multiple side cases that we didn't catch during the testing period and I will describe here the most important ones.

*  GPG: While this was something that we discussed a long time ago we forgot to communicate in advance but the GPG key used to sign releases changed for weekly release in order to not use Kohsuke gpg key. In a result of this, people who use RedHat/Debian/Suse repository need to reimport the new key. Remark: this is something that will also have to be done for Lts once we also release Lts with the current process

New Key:                                                                                                                                                                               
pub   rsa4096 2020-03-30 [SC] [expires: 2023-03-30]
      62A9756BFD780C377CF24BA8FCEF32E745F2C3D5
uid           Jenkins Project <jenkins...@googlegroups.com>
sub   rsa4096 2020-03-30 [E] [expires: 2023-03-30].

Old Key:
pub   dsa1024 2009-02-01 [SC]
      150FDE3F7787E7D11EF4E12A9B7D32F2D50582E6
uid           Kohsuke Kawaguchi <k...@kohsuke.org>
uid           [jpeg image of size 3704]
uid           Kohsuke Kawaguchi <kohsuke....@cloudbees.com>
uid           Kohsuke Kawaguchi <koh...@cloudbees.com>
uid           Kohsuke Kawaguchi <kkawa...@cloudbees.com>
sub   elg2048 2009-02-01 [E]
sub   rsa4096 2016-11-01 [S]

* mirror.jenkins-ci.org: This service has now additional Html files `HEADER.html` and `FOOTER.html` per directory. Those two files are used by apache directory listing to prepend and append Html information to the directory listing. The main advantage of this is to provide information from mirrors about how to install those packages from your operating system from every mirror. At least apache with the correct configuration, an example of this is


Please note that the instruction to configure os package manager is referencing the wrong URL and it should be pkg.jenkins.io. A fix needs to be provided, other remark CSS needs to align with other Jenkins website if someone is willing to work on this, feel free to do, it's defined here https://github.com/jenkinsci/packaging/blob/INFRA-910-core-release-automation/templates/base.html

* pkg.jenkins.io: this service is used by RedHat/Debian/Suse package manager to know which version can be installed, previously a list of packages that could be download from pkg.jenkins.io was generated for each release, this won't be the case anymore instead a link to mirrors should be provided. As a result of this, the latest weekly version is not listed even though you can easily craft an URL.

We still have improvement coming and the next release can only be smoother.

Thanks to Tim Jacob, Marky Jackson, Oleg Nenashev, Alex Earl, Mark Wait, Daniel Beck, Kohsuke, Oliver Ogondza who helped me to ship this first release yesterday but also to all the people who helped me, over the last two years to make this project a thing.

Have a good day,

Cheers

On Wed, Apr 15, 2020, at 10:53 PM, Olblak wrote:
After some discussion with the people involved in the process, I'll start at 11AM UTC instead of 3PM UTC, so we have more time to fix any unplanned issues.

On Wed, Apr 15, 2020, at 9:51 AM, Oleg Nenashev wrote:
I confirm my availability for tomorrow. Will also spend some time today to merge changes into the Jenkins core so that the release includes some real enhancements and fixes for Jenkins users.

Regarding the schedule, it would be great to start the release process a bit earlier so that we have more time to fix issues if any. E.g. we could start the release Pipeline early so that artifacts are ready to packaging when US wakes up.

Best regards,
Oleg

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Kohsuke Kawaguchi

unread,
Apr 17, 2020, 9:00:56 AM4/17/20
to jenkin...@googlegroups.com, Jenkins Developers ML
Wow, congratulations!!

After 15 years and ~900 releases, this process has finally left my hands...


To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/649dfada-eaa8-4169-9f96-e6173ee8e216%40www.fastmail.com.


--
Kohsuke Kawaguchi

Jesse Glick

unread,
Apr 17, 2020, 9:59:40 AM4/17/20
to jenkin...@googlegroups.com
On Fri, Apr 17, 2020 at 9:00 AM Kohsuke Kawaguchi <k...@kohsuke.org> wrote:
> After 15 years and ~900 releases, this process has finally left my hands...

Sweet. You can turn off that computer in your basement now and save a
bit on your electric bill!

Tracy Miranda

unread,
Apr 17, 2020, 12:36:33 PM4/17/20
to jenkin...@googlegroups.com
A huge milestone! 🎉

Thanks Olivier for the detailed summary - love it and how it is a true community effort. 
Can I start planning a party to celebrate this now?

Tracy

--
You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CANfRfr3pJqfrKQ61k_qhgOO0gPFXExoU%2B%3DGbiVpPKp2GxhwxsA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages