[JIRA] (JENKINS-52232) Credentials not usable after upgrade to 1.14

117 visualitzacions
Ves al primer missatge no llegit

cbley@exa-online.de (JIRA)

no llegida,
28 de juny 2018, 4:18:0228/6/18
a jenkinsc...@googlegroups.com
Claudio B created an issue
 
Jenkins / Bug JENKINS-52232
Credentials not usable after upgrade to 1.14
Issue Type: Bug Bug
Assignee: Devin Nusbaum
Components: ssh-credentials-plugin
Created: 2018-06-28 08:17
Priority: Major Major
Reporter: Claudio B

I have upgraded to ssh-credentials version 1.14 which fixes SECURITY-440 / CVE-2018-1000601.

After upgrading from version 1.13, no job could authenticate to Github, since the credentials was using a "private key file on master".

According to the announcment:

> Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

This seems not to work for me. I do not see `SECURITY-440: Migrating FileOnMasterPrivateKeySource to DirectEntryPrivateKeySource` message in the logs and the "private key" input box of the credentials is just empty.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

wfollonier@cloudbees.com (JIRA)

no llegida,
28 de juny 2018, 10:57:0128/6/18
a jenkinsc...@googlegroups.com
Wadeck Follonier commented on Bug JENKINS-52232
 
Re: Credentials not usable after upgrade to 1.14

Claudio B do you have a backup of that instance, in order to try to reproduce the behavior ?

If it's the case, could you try to restart the instance after the plugin update ?

wfollonier@cloudbees.com (JIRA)

no llegida,
28 de juny 2018, 11:29:0228/6/18
a jenkinsc...@googlegroups.com
Wadeck Follonier edited a comment on Bug JENKINS-52232
[~jenkey] do you have a backup of that instance, in order to try to reproduce the behavior ?

If it's the case, could you try to restart the instance after the plugin update ?


Could you also give us the version of your credentials plugin ?

In order to reproduce also ourself the problem, did you upgrade the plugin using the update-center ? and then you restarted your instance ?

cbley@exa-online.de (JIRA)

no llegida,
2 de jul. 2018, 10:43:022/7/18
a jenkinsc...@googlegroups.com

Wadeck Follonier, sorry I don't have a backup for this instance. In the meantime, I simply copied the private key from the file, pasted it into the input box and saved the credentials.

I did install the plugin using the update-center, and I restarted Jenkins using the "Restart Jenkins if installation complete and no jobs are running" checkbox. Is that enough or should I ensure to restart the service next time?

After the upgrade and realizing that all my jobs did not work anymore, I downgraded the ssh-credentials-plugin to version 1.13, restarted and all was working again.

BTW, I am using the jenkins RPM from pkg.jenkins.io/redhat-stable.

The credentials-plugin is at version 2.1.17.

stuart@somepointinthefuture.co.nz (JIRA)

no llegida,
2 de jul. 2018, 23:54:022/7/18
a jenkinsc...@googlegroups.com

We had exactly the same issue, we are on jenkins 2.107.3, we upgraded the plugin to 1.14 and found the only option we had was 'enter directly'. Downgrading the plugin to 1.13 restored the functionality.

jnz@topdanmark.dk (JIRA)

no llegida,
3 de jul. 2018, 3:56:023/7/18
a jenkinsc...@googlegroups.com

In [SECURITY-440] It is stated that 

SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

Thus you need to enter your ssh-key directly. We fixed this by entering our ssh-key into an Environment Variable and loading that into the credentials at boot time.

jnz@topdanmark.dk (JIRA)

no llegida,
3 de jul. 2018, 4:22:023/7/18
a jenkinsc...@googlegroups.com
Jon Brohauge edited a comment on Bug JENKINS-52232
In [[SECURITY-440|https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440 |https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440 ]] It is stated that 

{quote}

SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor {{~/.ssh}}. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

{quote}

Thus you need to enter your ssh-key directly. We fixed this by entering our ssh-key into an Environment Variable and loading that into the credentials at boot time.


BTW: The scope of the Credentials need to be SYSTEM not GLOBAL for this version (1.14) to work in "GitHub Organization" type jobs. Maybe this is required on other types of jobs as well.

dnusbaum@cloudbees.com (JIRA)

no llegida,
19 de jul. 2018, 13:10:0219/7/18
a jenkinsc...@googlegroups.com
Devin Nusbaum closed an issue as Cannot Reproduce
 

Wadeck Follonier took a look but was not able to reproduce this issue. As far as we know the migration should work correctly. If anyone can reproduce the migration not working please reopen the ticket and comment with the steps to reproduce the issue.

Change By: Devin Nusbaum
Status: Open Closed
Resolution: Cannot Reproduce

dnusbaum@cloudbees.com (JIRA)

no llegida,
7 de febr. 2019, 9:49:027/2/19
a jenkinsc...@googlegroups.com
Devin Nusbaum commented on Bug JENKINS-52232
 
Re: Credentials not usable after upgrade to 1.14

Looking at JENKINS-54746 I think the exception in this comment (users report seeing this in the old data monitor) is the root cause. The migration is gated by the RunScripts permission being active when that code runs, and that code is expected to run as ACL.System, but for some reason it ran as ACL.anonymous. When the exception is thrown in readResolve, perhaps the effect is that the serialized data is totally lost and it is as if no private keys were ever entered. Perhaps this code should be modified to instead just return if that permission is not found to avoid the conversion exception.

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

dnusbaum@cloudbees.com (JIRA)

no llegida,
7 de febr. 2019, 9:49:037/2/19
a jenkinsc...@googlegroups.com
Devin Nusbaum reopened an issue
 
Change By: Devin Nusbaum
Resolution: Cannot Reproduce
Status: Closed Reopened

dnusbaum@cloudbees.com (JIRA)

no llegida,
7 de febr. 2019, 9:53:027/2/19
a jenkinsc...@googlegroups.com
Devin Nusbaum assigned an issue to Unassigned
Change By: Devin Nusbaum
Assignee: Devin Nusbaum

wfollonier@cloudbees.com (JIRA)

no llegida,
11 de febr. 2019, 8:41:0511/2/19
a jenkinsc...@googlegroups.com
Wadeck Follonier commented on Bug JENKINS-52232
 
Re: Credentials not usable after upgrade to 1.14

After the new tickets opened, I re-try to reproduce the case with the new information.

The migration of the credentials keys was meant to be done after InitMilestone.JOB_LOADED is triggered, from credentials, in SystemCredentialsProvider.forceLoadDuringStartup(). At this point the running user should be SYSTEM. I discovered that the migration could be triggered before, when the credentials are stored in folder, or when they are used in the configuration of an agent (like ssh-agents). But at this point, both migration were done using SYSTEM in my case and so, passing with success the permission check on RUN_SCRIPTS that was added especially for the security patch.

So now, to go further, I need to have the logs from people that encountered the issues. Especially if they can reproduce the case, perhaps they could provide more information about all the plugins, the log file, the config file of their Jenkins, or anything else that could be useful (be careful to not upload credentials in plain text). What are you using as AuthorizationStrategy ?

My hypothesis is something force the current running user to be anonymous instead of System during the startup/migration, (even temporally) and call the credentials while loading.

"Call for witnesses"
=> People from this: Claudio B, Jon Brohauge, Stuart Whelan
=> People from JENKINS-54746: Florian Bäuerle, Darryl Pogue, Pieter-Jan Busschaert, Tom Ghyselinck, Makson Lee, Michael Mrozek, Torsten Landschoff, Matthew Stewart, Johannes Rudolph, Aaron Marasco, Beat Guggisberg
=> People from JENKINS-55971: Steve Graham

nneul@neulinger.org (JIRA)

no llegida,
11 de febr. 2019, 8:46:0211/2/19
a jenkinsc...@googlegroups.com

I had the same symptom - just been watching the issues here, have not tried re-updating since the last failure. In the first iteration, I didn't pay any active attention to log during upgrade itself, only noticed the resolve errors afterwards. 

In my case, using project matrix auth. Will try to see if symptom reproduces and capture some additional logs for you if so. 

wfollonier@cloudbees.com (JIRA)

no llegida,
11 de febr. 2019, 8:50:0211/2/19
a jenkinsc...@googlegroups.com

Nathan Neulinger That could be very very appreciated! I tried with project matrix auth without success (=migration was smooth). Could you give me insights about the structure of your instance ? Multiple folders ? restricted ones that are hiding some projects ? Credentials used at folders level ? Do you use internal credential provider or external ones ? Any information about your setup that is not "out of the box" will be useful

nneul@neulinger.org (JIRA)

no llegida,
11 de febr. 2019, 9:01:0111/2/19
a jenkinsc...@googlegroups.com

Gotta love when failure = working fine.

 

No folders, all jobs visible to anonymous, but have a handful of jobs that are restricted from execution. Didn't look at credentials related to anything within the jobs - only for the agent connections. It's possible there was impact inside the jobs as well - didn't get that far when I saw the upgrade fail. 

Aside from sitting behind apache, and being in a different directory structure the instance should be pretty vanilla. 

nneul@neulinger.org (JIRA)

no llegida,
11 de febr. 2019, 9:03:0211/2/19
a jenkinsc...@googlegroups.com

For the credentials for me - it's almost entirely pre-existing use of ~jenkins/.ssh/id_rsa that was affected. Very little use of built-in creds - and none at all of that for agent connections. 

jenkins@mrozekma.com (JIRA)

no llegida,
11 de febr. 2019, 22:28:0211/2/19
a jenkinsc...@googlegroups.com

This no longer happens for me, so I assumed it was found and fixed; I didn't do anything to try to fix it. You keep mentioning SYSTEM, so I guess I should point out that I was only seeing this on Linux nodes; the SSH private key used to login to them wasn't getting migrated correctly, so none of them would come up

Aaron.Marasco@BIA-Boeing.com (JIRA)

no llegida,
13 de febr. 2019, 7:40:0213/2/19
a jenkinsc...@googlegroups.com

Wadeck Follonier sorry I don't have much to contribute, as noted in the other ticket, I got it working and went on my merry way. As Nathan Neulinger noted above - my setup was pretty much the same. The Jenkins user on the Linux server had the ssh keys outside of Jenkins itself (in a standard Unix manner) and I had to manually copy them into the GUI.

Aaron.Marasco@BIA-Boeing.com (JIRA)

no llegida,
13 de febr. 2019, 7:42:0213/2/19
a jenkinsc...@googlegroups.com
Aaron Marasco edited a comment on Bug JENKINS-52232
[~wfollonier] sorry I don't have much to contribute, as noted in the other ticket, I got it working and went on my merry way. As [~nneul] noted above - my setup was pretty much the same. The Jenkins _user_ on the Linux server had the ssh keys _outside of Jenkins itself_ (in a standard Unix manner) and I had to manually copy them into the GUI.

 

Edit: For some reason it stripped the link from "other ticket" to https://issues.jenkins-ci.org/browse/JENKINS-54746?focusedCommentId=357252

jnz@topdanmark.dk (JIRA)

no llegida,
15 de febr. 2019, 2:23:0415/2/19
a jenkinsc...@googlegroups.com

We usually don't do upgrades.
By leveraging Docker, and JCasC, we rebuild from scratch, every time we need a new version Jenkins or a plugin. Having no state inside Jenkins, we can treat the containers as cattle. If it gets sick, here comes the bolt-pistol. As mentioned in my previous comment comment-343088, we fixed our issue by entering the SSH key "directly" and setting the proper scope.

nneul@neulinger.org (JIRA)

no llegida,
16 de febr. 2019, 22:14:0216/2/19
a jenkinsc...@googlegroups.com

Had a chance to try this again, and cannot reproduce now - upgrade processed smoothly with no issues. Sorry I can't provide anything further. 

vazhnov@yandex.ru (JIRA)

no llegida,
11 de maig 2019, 17:43:0211/5/19
a jenkinsc...@googlegroups.com

I've just installed fresh Jenkins and found I can't use SSH private key from Jenkins home directory, ~/.ssh/id_rsa. As workaround, I put SSH key into Jenkins Credentials, it works.

  • SSH Slaves v1.29.4,
  • SSH Credentials Plugin v1.16,
  • Jenkins v2.164.3,
  • host and slave OS: Ubuntu 18.04.2 with all updates,
  • OpenSSH v7.6p1.

vazhnov@yandex.ru (JIRA)

no llegida,
11 de maig 2019, 17:53:0311/5/19
a jenkinsc...@googlegroups.com
Alexey Vazhnov edited a comment on Bug JENKINS-52232
I've just installed fresh Jenkins and found I can't use SSH private key from Jenkins home directory, {{~/.ssh/id_rsa}}. As workaround, I put SSH key into Jenkins Credentials, it works.
* SSH Slaves v1.29.4,
* SSH Credentials Plugin v1.16,
* Jenkins v2.164.3,
* host and slave OS: Ubuntu 18.04.2 with all updates,
* OpenSSH v7.6p1.


*Update*: found [this|https://jenkins.io/security/advisory/2018-06-25/]:
bq. SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.
Respon a tots
Respon a l'autor
Reenvia
0 missatges nous