ANN: Jenkins release artifacts uploads blockage on June 09, and a temporary downloads issue

[Oleg Nenashev]

Dear all,

As you may have noticed, the release artifact uploads are currently blocked in the Jenkins Artifactory instances (https://repo.jenkins-ci.org/). We are doing a security investigation due to a partial user database loss on June 02. Today we blocked releases to the Jenkins artifactory, and there also was a temporary outage of the Artifactory downloads which was a collateral damage of the temporary permissions. You can find more details about it in this Jenkins Infra Thread and in this Dev List thread.

Current status:

• Downloads are restored for all artifacts on https://repo.jenkins-ci.org/, Jenkins core historical releases, Remoting library and Windows Service Wrapper which were among ones reported by Jenkins users.

• Uploads: Jenkins artifact uploads are blocked for the most of Jenkins plugin maintainers and contributors. It affects releases of Jenkins plugins, Jenkins core and modules, developer tools and all libraries hosted on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments are not affected.

Quick summary: 

• Jun 02 - There was a Kubernetes Cluster outage on June 02. During this outage we had to rebuild the cluster from scratch to get some services working again.

• Jun 02 - After the recovery we lost three months of LDAP changes. It has happened due to the broken backup of the LDAP database.

• Jun 02 - We identified a number of potential security risks which may be caused by the LDAP outage. Account overtake and malicious upload was one of the identified risks. FTR this issue is tracked as SECURITY-1895 as a follow-up to these discussions. Only the Security team members have access to it, so I am not sharing a link here.

• Jun 09 - After the security risk was independently reported in public by a plugin maintainer in the dev list thread, we decided to block uploads of release artifacts to the Jenkins Artifactory instance.

• Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked (plugins, Jenkins core and modules, developer tools, etc.). Downloads of some binaries were also blocked as an unexpected collateral damage. Jenkins core historical releases, Remoting library and Windows Service Wrapper are among the affected binaries

• Jun 09, 10AM UTC - We finished reviews of all artifact releases to https://repo.jenkins-ci.org/, which happened between the infra outage on June 02 and the blockage of the releases. There are no maliciously uploaded artifacts. Note that the common plugin release flow requires access to GitHub in order to push the release commits, so a malicious attacker would need to overtake both Jenkins and GitHub accounts of a single user to submit a legitimately-looking release.

• Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch in the Repository Permission Updater was applied to prevent uploads. Artifact uploads are still blocking

• Jun 09, 2PM UTC, based on repo.jenkins-ci.org and issues.jenkins-ci.org data, we restored maintainers accounts.

Our next steps would be to communicate the issue to all maintainers and contributors who might have been affected by the LDAP history loss. We will likely need to perform additional user verification steps for plugin maintainers to ensure that there are no contributors affected by the issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure team meeting where this issue will be discussed in more detail. This is a public meeting, and everyone is welcome to join. Calendar link

Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and Security team members who contributed to this investigation.

Best regards,

Oleg Nenashev

[Dmitry Sotnikov]

Thank you, Oleg and the Security Team!

Dmitry

[Olblak]

Dear all,

We are ready to proceed with restoration of the Jenkins account database. Today we are going to restore user LDAP accounts that were created since the First of February 2020 based on the data from Jenkins Jira and the repository Permission Manager metadata data. We will also reset passwords for all users registered in the database.

Step 1. All users who lost their account will receive an email saying that their accounts were re-created. There will be no temporary password in these emails, but there will be information pointing to this thread.

Step 2. We’ll reset every user password from the LDAP database, it is more than 100 000 users. Once done, you’ll receive an email telling you that your password was reset with a reason containing a link to this mail thread.

Step 3. We will delete accounts of users who requested such deletion between February and June 2020. These users were restored from the backup, so we have to delete them again.The list of users is based on Jira tickets and private messages to the Jenkins Infra officer. If for some reason you notice that your account still exists, feel free to raise a ticket in Jenkins Jira (project=INFRA, component=account).

Please do not hesitate to contact us using the #jenkins-infra channel on Freenode IRC or the Jenkins Infrastructure mailing list if you have any questions or suggestions. If you see a security issue related to the accounts, please follow the vulnerability reporting guidelines.

Best regards,

Olivier Vernin && Jenkins Infrastructure Team

[Oleg Nenashev]

June 12th update:

• We are still working on the account migration
• Step 1 is completed, all users have been restored in the database based on the data from Jenkins Jira and repository permissions updater.
• Step 2 is in progress. Tens of thousands users have already received the password reset notifications, we had 2 batches of password resets today. We will continue the migration tomorrow
• Step 3 - not started
• Plugin uploads are still blocked at the moment
• Tomorrow we plan to double-check the account resets for plugin maintainers, and we will consider reenabling uploads after that

Best regards,

Oleg

[Dave Pedu]

Hello,

I have received an email linking to this thread. However, it contains a plaintext password for my account, despite this:

> There will be no temporary password in these emails, but there will be information pointing to this thread.

Is this email legitimate or am I being phished? Screenshot attached.

Thanks,

Dave

[Oleg Nenashev]

Hi Dave,

This is an email from the Step 2. We’ll reset every user password from the LDAP database. This one includes a temporary password, and we expect users to change it after they login into the system.

For those who wonder: Yes, the temporary password is sent in plain text as mentioned above. This is how our current password reset system is designed. As other projects, we have a decent amount of technical debt in our infrastructure which we gradually resolve. I have already added changing the account password reset flow to the outage retrospective list, an we will be reviewing what to do there after the outage is fully resolved. Apart from fixing it, migrating to a 3rd-party identity service is on the table for me (Linux Foundation or GitHub).  If anyone is interested to participate and to improve the project, the Jenkins infrastructure team is always looking for more contributors!

If anyone has concerns about such method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way.

Best regards,

Oleg

[Oleg Nenashev]

Dear all,

June 12 update: 

• We continue to work on the accounts migration and will share the next update on Monday
• Jenkins releases are still blocked. If there are any emergency releases you need to perform, please reply in this thread.

Best regards,

Oleg Nenashev

[Liam Newman]

I have two releases that I consider high-priority: github-branch-source and github-api .

Users have been able to rollback to the previous release to unblock themselves, but people who cannot rollback (new installations) remain blocked.  

[Oleg Nenashev]

Ack. I will make sure we have a workaround applied on Monday if the user update is not finished

[Oleg Nenashev]

Dear all,

We have reset all plugin maintainer accounts, and we have reenabled plugin uploads in the Repository Permission Updater. By now all upload permissions should be restored, except a few new user registrations in the Jenkins Artifactory instance over the past week. All Artifactory API tokens were revoked. If you experience any issues with plugin and component releases, please let us know in this thread.

For a list of disabled user accounts, please see this pull request: https://github.com/jenkins-infra/repository-permissions-updater/pull/1574. The disabled users need to login to https://repo.jenkins-ci.org/ again, and then to submit a pull request restoring their permissions. If you use Artifactory API tokens for uploads, you need to login to Artifactory and to reconfigure them.

We apologize for any inconvenience the restrictions caused, and we will have a retrospective to discuss what we could do better to prevent it in the future. If you want to share any feedback, please send it to this thread. If you want to share a private feedback, please send it to my email.

Best regards,

Oleg Nenashev

[Steve Springett]

"Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.

[Oleg Nenashev]

Hi Steve,

Duly noted. Note that we offered an alternate way for maintainers to get their password delivered if they are not fine with the current delivery method. In my message from Jub 12: If anyone has concerns about such a method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way. You did not contact us, and hence you got your password reset with the standard process. If you want to get your password reset in a secure way, please feel free to use this process.

Again, we operate with resources and tools we have. The Jenkins project and its infrastructure are driven by volunteers, and we have limited capacity when it comes to fixing urgent things due to uncoordinated disclosures. You may call it insane, but it was the solution we delivered with given circumstances. Contributors have families and other commitments, and please know that the situation has taken a high toll on them. Everybody is welcome to contribute and to contribute to the infrastructure. I am cordially grateful to those several contributors who stepped up and helped to get the issue fixed, or offered to help, or sent kind words over different channels. This is an example to follow.

Everyone is welcome to join the team and to work together on a better solution for user management so that we can prevent a similar situation in the future.

Best regards,

Oleg

On Mon, Jun 15, 2020 at 5:02 PM Steve Springett <Steve.S...@owasp.org> wrote:

"Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4547a00e-e223-4075-a2a1-9162b4634b5bo%40googlegroups.com.

[Steve Springett]

Security best practices should not be opt-in. I receive the manifest (daily) emails and did not see this topic. Many others likely did not either.

Jenkins is viewed by many as Critical Cyber Infrastructure and plays an important role in the global software supply chain. That supply chain was just weakened today, on purpose.

I understand the volunteer perspective. I lead multiple OWASP projects and spend a considerable amount of time doing open source projects. I get it. Just know that the software supply chain affects everyone including CloudBees Enterprise customers and every downstream consumer of projects built using Jenkins.

[Matt Sicker]

No complaints about the lack of a DNSSEC record or other ways to avoid
an initial MitM attack when connecting to jenkins.io for the first
time. Or numerous other theoretical points of failure. It might be
better to offer constructive advice rather than declaring everything
broken.

On Mon, Jun 15, 2020 at 1:20 PM Steve Springett

<Steve.S...@owasp.org> wrote:
>
> Security best practices should not be opt-in. I receive the manifest (daily) emails and did not see this topic. Many others likely did not either.
>
> Jenkins is viewed by many as Critical Cyber Infrastructure and plays an important role in the global software supply chain. That supply chain was just weakened today, on purpose.
>
> I understand the volunteer perspective. I lead multiple OWASP projects and spend a considerable amount of time doing open source projects. I get it. Just know that the software supply chain affects everyone including CloudBees Enterprise customers and every downstream consumer of projects built using Jenkins.
>
>
> On Monday, June 15, 2020 at 10:27:01 AM UTC-5 Oleg Nenashev wrote:
>>
>> Hi Steve,
>>
>> Duly noted. Note that we offered an alternate way for maintainers to get their password delivered if they are not fine with the current delivery method. In my message from Jub 12: If anyone has concerns about such a method and wants to use alternate channels for encrypted password transfer, please send us a message through the Jenkins Infrastructure mailing list from your email registered in Jenkins. In this email please provide your public GPG key so that we can reset a password again in a secure way. You did not contact us, and hence you got your password reset with the standard process. If you want to get your password reset in a secure way, please feel free to use this process.
>>
>> Again, we operate with resources and tools we have. The Jenkins project and its infrastructure are driven by volunteers, and we have limited capacity when it comes to fixing urgent things due to uncoordinated disclosures. You may call it insane, but it was the solution we delivered with given circumstances. Contributors have families and other commitments, and please know that the situation has taken a high toll on them. Everybody is welcome to contribute and to contribute to the infrastructure. I am cordially grateful to those several contributors who stepped up and helped to get the issue fixed, or offered to help, or sent kind words over different channels. This is an example to follow.
>>
>> Everyone is welcome to join the team and to work together on a better solution for user management so that we can prevent a similar situation in the future.
>>
>> Best regards,
>> Oleg
>>
>>
>> On Mon, Jun 15, 2020 at 5:02 PM Steve Springett <Steve.S...@owasp.org> wrote:
>>>
>>> "Technical debt" is not an excuse to reset plugin maintainers accounts and include a clear-text email containing their username AND password. That's insane. As a security professional I will not stand for that. I will no longer be maintaining Jenkins plugins and will attempt to find new maintainers for the ones I do. No guarantees.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4547a00e-e223-4075-a2a1-9162b4634b5bo%40googlegroups.com.
>
> --

> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/da467d52-3ad4-41f5-8d8b-2f86a68487a1n%40googlegroups.com.



--
Matt Sicker
Senior Software Engineer, CloudBees

[Olivier Lamy]

Awesome.

Thanks a lot for the hard work!

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/58a6e193-9231-408c-a783-07bddfc23029n%40googlegroups.com.

--

Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

[Karsten Jeschkies]

Hi,

thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?

Best.

Karsten.

[Mark Waite]

On Wed, Jun 17, 2020 at 6:44 AM Karsten Jeschkies <kjesc...@d2iq.com> wrote:

Hi,

thanks for you hard work. I reset my password successfully but cannot upload a release for the Mesos plugin. Are releases still blocked?

Releases are not blocked but a password reset will also reset your password to the artifact repository.  If you're receiving an HTTP 401 when you try to `mvn release perform` you may need to update your password in the ~/.m2/settings.xml.

I had to do that in order to release a new version of a plugin yesterday.  I logged into the Jenkins Artifactory instance and had it generate an encrypted password from my profile page on that server.  I inserted that encrypted password into my ~/.m2/settings.xml file.  I'm not sure if that is the preferred way to do it, but it worked for me.

Mark Waite

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ea5483fb-6873-41dd-a82c-d5518c7de106o%40googlegroups.com.

[Tim Jacomb]

Apparently I use an artifactory API key to release, so I had to go into my artifactory settings (https://repo.jenkins-ci.org/webapp/#/home)

and generate a new API key

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.

[kjesc...@d2iq.com]

Hi,

thanks for the advice. Hm, my ~/.m2/settings.xml had my encrypted password. The docs (https://wiki.jenkins.io/display/JENKINS/Hosting+Plugins#HostingPlugins-Releasingtojenkins-ci.org) don’t mention the API key. How can I configure Maven to use the API key instead?

Many thanks.

Karsten.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGqXd-FwrxzgVtVhJ0nki1BOwgGawSuE%3Dc4%2B940sh07XQ%40mail.gmail.com.

[Tim Jacomb]

it's just the same as a password to maven, so use the api key instead of a password.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YXJzCV7jFGLz18BnjePsTVjbf5ch%2B9nVBcjC-QUO5z2T8Q%40mail.gmail.com.

[kjesc...@d2iq.com]

Hm, that does not work. I am using the Gradle JPI plugin. It does not seem to pick up ~/.m2/settings.xml nor ~/.jenkins-ci.org.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3BieEJ4aA5zax1SqVf%2B_EuNL9C3-qEhCjxkiw6PRPXXakEg%40mail.gmail.com.

[Oleg Nenashev]

Duly noted about the Documentation.

I will migrate https://wiki.jenkins.io/display/JENKINS/Hosting+Plugins to jenkins.io and extend it to cover the use-case tonight

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3UvrCTflXGk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKs8YX%2BC_yqey%2B8Da5q7oj-grWh15Hz4-JmVY_GTxynYsk%2B7bg%40mail.gmail.com.

[Olblak]

Hi Everybody,

Some updates regarding this topics, while almost everything is back to normal, we decided to not pursue with the "every user password reset" as we announced initially but instead we focused on maintainers and administrators access.
The reason for that is because we don't have ready to use tooling so it requires us to write custom scripts.
While we initially tried to go down that path, we reset +-30% of the database, we realized that because of the amount of garbage we have in that database, it was hard and time-consuming to finish this so we decided to look after alternatives for accounts.jenkins.io.

At the moment we have two promising alternatives:

Keycloak as a replacement for accounts.jenkins.io.
keycloak is an opensource identity management tool, which supports many integrations like Github SSO or LDAP.
It's deployed and only available from our VPN at the moment, configuration is defined here.
It uses a RDS PostgreSQL database running on AWS and containers are running on our AKS cluster.
It was easy to deploy, configured, *seems* easy to maintain, and its database is running on a managed service.
It sounds very promising as it does exactly everything that accounts.jenkins.io do with a lot more like:
* Enforce email verification
* OTP
* Safe reset password workflow
* Using your social account like Github for login
* And many more
So we could stop losing our time patching our custom identity management tool.

The second option would be to totally or partially delegate identity management to the Linux Foundation infrastructure team.

We had a first exploratory meeting with them this week and we have another one planned next week

The whole idea is the vast majority of Jenkins account are used to report/update issues while the smallest amount of accounts are used by plugin maintainers (+-1700)and Jenkins administrators(+-20).
So if we can delegate the management of Jira to them, we wouldn't need to maintain an identity management tool anymore.
While implementation details still need to be discussed with them what seems to be clear at the moment are:
* Identify management would be a black box, as it would also contain other Linux Foundation accounts.
* We could use it for Artifactory (repo.jenkins-ci.org) as they are already doing the same for other communities that they are managing.

While we would lose flexibility on this, we wouldn't have to maintain it or care about GDPR.

Therefore it will give us more time to focus on other initiatives.

If you have any advice, questions, concerns on this topic, feel free to raise them.

Thanks for your patience

Olivier

[Radosław Antoniuk]

Hi Olivier,

Thanks a lot for thorough update, both options sound really interesting. 

I think it would be easier to rely on an external identity provider and each of the Cloud providers now provide this (e.g. AWS SSO, Azure AD etc.). I believe that most people using Jenkins have GH or FB accounts. Sounds like this is almost the same as running keycloak so assuming that we are using k8s and rds anyway, but I understand that GDPR/CCPA would need to be handled by us in such case.

I don't know what (tools/process) Linux Foundation is using, but the important parts from my perspective:

- all the tools, not only Jira, but also blog, artifactory and ci should use it for Auth/AuthZ

- we need to know what are the limitations (would they allow for managing groups for different access levels e.g. plugin maintainers vs infra maintainers vs normal reporters)

- would it still be possible to use GitOps for open permission management

For me both solutions are a great step forward from the current solution and I would personally choose the one that allows the least maintenance but that would address all the requirements (i.e. being the SSO solution) - for me GH is the natural choice as the SSO provider as Jenkins code is hosted there.

PS. I am unable to reset my password on Jira/accounts.jenkins.io ATM - no password reset email is coming to me.

Cheers,

Radek

[Olblak]

Hi Radek,

Thanks for your feedback

PS. I am unable to reset my password on Jira/accounts.jenkins.io ATM - no password reset email is coming to me.

If you send me your username and email address, I could have a look to it.

I don't know what (tools/process) Linux Foundation is using

They are using https://identity.linuxfoundation.org/ and migrating to Auth0 but in both cases they allow you to use social accounts like Github

- we need to know what are the limitations (would they allow for managing groups for different access levels e.g. plugin maintainers vs infra maintainers vs normal reporters)

This is something that need to be clarify indeed, as far as I understood it at the moment LF identity management can only be used for services that they manage on the opposite is true as well if we ask they to manage our Jira they will probably need to also manage the identity, but indeed this is something that need clarification.

Ps: We continued the discussion about keycloak on the jenkins infrastructure mailing list here

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPe2pWgGGCMxSWbm7z_v9dqwqez1%3DMWBJMJCyOkZTYOksnr9cA%40mail.gmail.com.

[Robert St. John]

Hello,

I have successfully reset my password on accounts.jenkins.io, and I can login.  However, I am now unable to login to Jira.  Is anyone else having this problem?  My user name is restjohn.

Thanks.

Robert

[nigel.a...@braincorp.com]

I am unable to reset my password or login. This seems to be expected give that not all users were reset. 

What is the plan for users whose accounts were not reset? Should I create a new account?

-Nigel

[Oleg Nenashev]

Hi all,

If you experience issues with login, please bring them up issues in the Jenkins Infrastructure mailing list: https://groups.google.com/forum/#!forum/jenkins-infra

Thanks for understanding,

Oleg