Dear all,
As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread and in this Dev List thread.
Current status:
Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.
Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.
Quick summary:
Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.
Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.
Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.
Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.
Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries
Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.
Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking
Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts.
Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link
Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.
Best regards,
Oleg Nenashev
Dear all,
We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.
Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.
Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.
Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in Jenkins Jira (project=INFRA, component=account).
Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the vulnerability reporting guidelines.
Best regards,
Olivier Vernin && Jenkins Infrastructure Team
"Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.
--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4547a00e-e223-4075-a2a1-9162b4634b5bo%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/58a6e193-9231-408c-a783-07bddfc23029n%40googlegroups.com.
Hi,thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YXJzCV7jFGLz18BnjePsTVjbf5ch%2B9nVBcjC-QUO5z2T8Q%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3BieEJ4aA5zax1SqVf%2B_EuNL9C3-qEhCjxkiw6PRPXXakEg%40mail.gmail.com.
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YX%2BC_yqey%2B8Da5q7oj-grWh15Hz4-JmVY_GTxynYsk%2B7bg%40mail.gmail.com.
PS. I am unable to reset my password on Jira/accounts.jenkins.io ATM - no password reset email is coming to me.
I don't know what (tools/process) Linux Foundation is using
- we need to know what are the limitations (would they allow for managing groups for different access levels e.g. plugin maintainers vs infra maintainers vs normal reporters)
--You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPe2pWgGGCMxSWbm7z_v9dqwqez1%3DMWBJMJCyOkZTYOksnr9cA%40mail.gmail.com.