[Gerrit-CI] Security incident - CVE-2024-23897

[Luca Milanesio]

Dear Community,

As Gerrit Release Manager and Gerrit-CI maintainer, I must let you know about a security incident that happened on the 24th of January 2024; see the timeline of events and the details at [1].

Please read carefully the details of the incident and the actions that we have put in place to rotate all the secrets that could have been potentially compromised. Also, look at the actions we have put in place to make the CI more resilient to future incidents.

The most noteworthy step we are taking is the restriction of access to anonymous users: the main UI that all the contributors should see is the Gerrit, which also includes the status of the builds in the "Checks" tab of each change, we do not foresee any reason why anyone should need to navigate the Jenkins UI and calls its API if not coming from the Gerrit Checks tab links.

In a nutshell, these are the new restrictions to the https://gerrit-ci.gerritforge.com site:

1. The UI is no longer visible or browseable by unauthenticated users. The only visible element is a big button for login. At the moment, any GitHub user can log in, but we will also reinforce the authentication layer in the next few weeks and allow only trusted contributors and maintainers.
2. The hyperlinks from the Gerrit changes are working as before and give access to the build logs without the need to log into Jenkins
3. The artifacts, JSON details and listing of jobs and views from the Gerrit plugin manager are working as usual, without the need to login to Jenkins
4. The Jenkins version has been updated to v2.375.4, with the next upgrade to v2.440.1-lts in review at [2].
5. The new Gerrit-CI does not contain or expose any jobs that could potentially be using sensitive credentials (e.g. Gerrit homepage publishing Job and Gatling tests Job on AWS are not accessible anymore over the internet). We are running a secure instance running those jobs in a more protected and inaccessible environment.

With this upgrade, we have also moved the Gerrit build executions from Google Cloud / RBE (old Gerrit-CI) to new on-premises dedicated hardware for remote builds through BuildBuddy, see [4]. Alvaro has carried over the endeavour over the past few months and has demonstrated increased throughput, lower build verification latency, and higher stability; see the full report in [3].

I have performed a sanity check on the changes validations, and it is all working as expected: incoming changes are built and verified in around 5 minutes, which is over two times faster than before, thanks to the BuildBuddy RBE caching.

If you notice anything unusual or have any issues with the new setup, please feel free to answer this discussion thread.

Luca Milanesio

Gerrit Maintainer

Gerrit Release Manager

Gerrit CI Manager

[1] https://docs.google.com/document/d/e/2PACX-1vQ6oF1Bp5voqymlV2U0UReQ7LOpMU_7HT7Wmz-GsD5uWb-arLrCNu_h-yQTGi-Q-0BWGqv7nwSIrsvZ/pub

[2] https://gerrit-review.googlesource.com/c/gerrit-ci-scripts/+/410726

[3] https://groups.google.com/g/repo-discuss/c/jQPgaKmaNQA/m/joX6z7JFAAAJ

[4] https://www.buildbuddy.io/

[Jacek Centkowski]

Kudos Luca, Alvaro and the team!

[Luca Milanesio]

I have also published a “minimalistic UI” for browsing plugins (see [1]), taking the HTML / JS / CSS from the plugin-manager. The page does not expose any Jenkins UI elements, does not require any logins and still allows searching for plugins and accessing their download links.

See below a screenshot below:

Hopefully the above should fill the gaps for all of those who were using Gerrit-CI as source of plugins binaries download.

Luca.

[1] https://gerrit-ci.gerritforge.com/plugin-manager/