[JIRA] [ssh-agent-plugin] (JENKINS-32120) With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"

124 views
Skip to first unread message

oded@geek.co.il (JIRA)

unread,
Dec 17, 2015, 6:30:05 AM12/17/15
to jenkinsc...@googlegroups.com
Oded Arbel created an issue
 
Jenkins / Bug JENKINS-32120
With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"
Issue Type: Bug Bug
Assignee: Unassigned
Components: ssh-agent-plugin
Created: 17/Dec/15 11:29 AM
Environment: Jenkins 1.640 running on docker from onesysadmin/jenkins-docker-executors
All plugins up to date
Labels: plugin
Priority: Blocker Blocker
Reporter: Oded Arbel

When running a local build (on the master), the build fails to load the ssh-agent:

Started by an SCM change
Building in workspace /var/jenkins_home/workspace/production - client (prod3)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Java/JNR ssh-agent
[ssh-agent] Started.
[ssh-agent] Using credentials git (Bitbucket checkout)
[ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
	at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
	at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
	at org.bouncycastle.openssl.jcajce.PEMUtilities.crypt(Unknown Source)
	at org.bouncycastle.openssl.jcajce.JcePEMDecryptorProviderBuilder$1$1.decrypt(Unknown Source)
	at org.bouncycastle.openssl.PEMEncryptedKeyPair.decryptKeyPair(Unknown Source)
	at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:98)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper$SSHAgentEnvironment.add(SSHAgentBuildWrapper.java:388)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentBuildWrapper.preCheckout(SSHAgentBuildWrapper.java:227)
	at jenkins.scm.SCMCheckoutStrategy.preCheckout(SCMCheckoutStrategy.java:76)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
	at hudson.model.Run.execute(Run.java:1738)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Caused by: java.security.NoSuchAlgorithmException: PBKDF-OpenSSL SecretKeyFactory not available
	at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:121)
	at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:159)
	at org.bouncycastle.jcajce.util.DefaultJcaJceHelper.createSecretKeyFactory(Unknown Source)
	... 14 more
[ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)

Then the build looks like its continuing normally, but the ssh key was not actually loaded and as a result the build fails:

+ ssh -o StrictHostKeyChecking=no dep...@xxx.xxx.xxx some command
Permission denied (publickey).
+ ...
+ exit 5
+ exit 5
Build step 'Execute shell' marked build as failure
[ssh-agent] Stopped.

Running the same job on a slave works fine:

Started by user Oded Arbel
Building remotely on trusty (i-5827fe82) (trusty docker) in workspace /srv/build/workspace/production - client (prod3)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Java/JNR ssh-agent
[ssh-agent] Started.
[ssh-agent] Using credentials git (Bitbucket checkout)
[ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)

Downgrading the ssh-agent plugin to version 1.8 worked around the problem. After downgrade:

Started by user Oded Arbel
Building in workspace /var/jenkins_home/workspace/production - client (prodhk)
[ssh-agent] Using credentials git (Bitbucket checkout)
[ssh-agent] Using credentials deploy (Deployment key to work with Minion Master)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Java/JNR ssh-agent
[ssh-agent] Started.

Also see here: https://www.botbot.me/freenode/jenkins/2015-12-16/?msg=56268453&page=4

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

jespmark@gmail.com (JIRA)

unread,
Jan 6, 2016, 6:28:02 AM1/6/16
to jenkinsc...@googlegroups.com
Jesper Markenstam commented on Bug JENKINS-32120
 
Re: With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"

The issue appears to be a combination of ssh credentials plugin 1.11 + ssh agent plugin 1.9. Downgrading both to 1.10 / 1.8 respectively, restored expected functionality on clean installation with rpm jenkins-1.625.3-1.1.noarch, centos 67, oracle jdk 1.8.0_60.

jespmark@gmail.com (JIRA)

unread,
Jan 6, 2016, 6:29:02 AM1/6/16
to jenkinsc...@googlegroups.com
Jesper Markenstam edited a comment on Bug JENKINS-32120
The issue  appears to be  is present with  a combination of ssh credentials plugin 1.11 + ssh agent plugin 1.9. Downgrading both to 1.10 / 1.8 respectively, restored expected functionality on clean installation with rpm jenkins-1.625.3-1.1.noarch, centos 67, oracle jdk 1.8.0_60.

twolfart@java.net (JIRA)

unread,
Jan 12, 2016, 9:45:03 AM1/12/16
to jenkinsc...@googlegroups.com
twolfart commented on Bug JENKINS-32120

Downgrading to ssh agent plugin 1.8 worked for me, ssh credentials plugin is still at 1.11 (latest).

twolfart@java.net (JIRA)

unread,
Jan 12, 2016, 10:07:01 AM1/12/16
to jenkinsc...@googlegroups.com
twolfart updated an issue
 
Change By: twolfart
Comment:
Downgrading to ssh agent plugin 1.8 worked for me, ssh credentials plugin is still at 1.11 (latest).

aik.bold@gmail.com (JIRA)

unread,
Jan 18, 2016, 5:33:02 AM1/18/16
to jenkinsc...@googlegroups.com
Alexander Obuhovich commented on Bug JENKINS-32120
 
Re: With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"

According to Freenode channel log the problem happens, when the SSH key protected with Passphrase is being used: https://botbot.me/freenode/jenkins/2015-12-16/?page=3

philipp@beta-interactive.de (JIRA)

unread,
Jan 29, 2016, 9:32:03 AM1/29/16
to jenkinsc...@googlegroups.com

Having the same problems and can confirm, that it is only when your key is passphrase protected. Without passphrase everything is working as expected.

Jenkins v1.625.3, Ubuntu 14.04, java version 1.7.0_91 OpenJDK Runtime Environment

stashu.pub@gmail.com (JIRA)

unread,
Feb 12, 2016, 1:04:02 PM2/12/16
to jenkinsc...@googlegroups.com

Hi. A fix, but very strange things going on for me.
JM=jenkins master, BS=linux bld slave, somehost=ssh to this host
BldUser=jenkins build user SshUser=ssh user

I think I fixed or found a way passed this problem.
with SshUser private key I did: openssl rsa -in id_rsa -check > id_rsa.stan

my guess is while this checks my private key it spits out a different version/type of the key??
If I use that private key for my ssh job my error goes away and it works.
I added a new private key credential for SshUser with passphrase, leaving other one still there.

But here is the weirdness or what I do not understand.
BS node connects JM with old version SshUser key.
on JM I have another ssh key for BldUser for the buildjob.
In BuildJob i changed ssh-agent from BldUser to SshUser new priv key. This made it work!
Job does ssh -q somehost "hostname; pwd; id"
id is for SshUser as I expect.

but if I change the buildjob ssh-agent back to BldUser - it still works and the ssh user it uses is
from the BS ssh-agent setup (SshUser)!

17:51:15 [ssh-agent] Looking for ssh-agent implementation...
17:51:15 [ssh-agent] Java/JNR ssh-agent
17:51:17 [ssh-agent] Started.
17:51:17 [ssh-agent] Using credentials BldUser <- from buildjob
17:51:17 [my-box] $ /bin/sh -x /tmp/hudson2794482700022152346.sh
17:51:18 ++ hostname
17:51:18 + ssh -q somehost 'hostname; pwd; id'
17:51:18 somehost
17:51:18 /home/SshUser <- why from BS config (SshUser) and not from buildjob(BldUser)??
17:51:18 uid=910(SshUser) gid=25(xxx) groups=25(xxx)
17:51:18 + rc=0

Is there some jenkins key caching and using going on here?
If I take out using ssh-agent key from build job it will not work, so it does need something in the buildjob.

jenkins 1.625.3
ssh agent plugin 1.8
ssh credentials plugin 1.11

stashu.pub@gmail.com (JIRA)

unread,
Feb 12, 2016, 1:56:01 PM2/12/16
to jenkinsc...@googlegroups.com
Stan Towianski edited a comment on Bug JENKINS-32120
ssh agent plugin 1. 8 9
ssh credentials plugin 1.11

stashu.pub@gmail.com (JIRA)

unread,
Feb 12, 2016, 1:59:02 PM2/12/16
to jenkinsc...@googlegroups.com
p.s. earlier I also added this line to java.security per plugin page but it did not fix it then:
security.provider.11=org.bouncycastle.jce.provider.BouncyCastleProvider

jenkins 1.625.3
ssh agent plugin 1.9
ssh credentials plugin 1.11

java 8

stashu.pub@gmail.com (JIRA)

unread,
Feb 12, 2016, 1:59:03 PM2/12/16
to jenkinsc...@googlegroups.com
security.provider. 11 10 =org.bouncycastle.jce.provider.BouncyCastleProvider


jenkins 1.625.3
ssh agent plugin 1.9
ssh credentials plugin 1.11
java 8

stashu.pub@gmail.com (JIRA)

unread,
Feb 15, 2016, 4:33:04 PM2/15/16
to jenkinsc...@googlegroups.com

so, it looks like what I did to get around the problem is create another version of our private key that stripped out the passphrase. It's a workaround but does not fix the problem of passphrase not working in jenkins credential plugin.

a@blacksmith.cc (JIRA)

unread,
Feb 24, 2016, 11:27:03 AM2/24/16
to jenkinsc...@googlegroups.com
Adrian G commented on Bug JENKINS-32120

I have the same issue and can help with troubleshooting if useful. The problem that I've encountered seems to be that the SSH Agent plugin interferes with the Git plugin. With "SSH Agent" checked in the project definition, the ssh agent loads first, and fails to load keys because of the SecretKeyFactory error. The Git plugin then looks like it tries to use the ssh-agent configuration instead of the SSH credentials directly, and since there are no credentials, the whole job fails.

How can we resolve the issue with SecretKeyFactory not being available within the environment? That seems like the place to start.

stashu.pub@gmail.com (JIRA)

unread,
Feb 26, 2016, 10:57:05 AM2/26/16
to jenkinsc...@googlegroups.com

[ssh-agent] Using credentials myUser (with private key and passphrase)


[ssh-agent] Looking for ssh-agent implementation...

[ssh-agent] Java/JNR ssh-agent
[ssh-agent] Unable to read key: exception using cipher - please check password and data.
org.bouncycastle.openssl.EncryptionException: exception using cipher - please check password and data.
at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source)
at org.bouncycastle.openssl.PEMUtilities.crypt(Unknown Source)
at org.bouncycastle.openssl.PEMReader$KeyPairParser.readKeyPair(Unknown Source)
at org.bouncycastle.openssl.PEMReader$RSAKeyPairParser.parseObject(Unknown Source)
at org.bouncycastle.openssl.PEMReader.readObject(Unknown Source)
at com.cloudbees.jenkins.plugins.sshagent.jna.JNRRemoteAgent.addIdentity(JNRRemoteAgent.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.SecurityException: JCE cannot authenticate the provider BC
at javax.crypto.Cipher.getInstance(Cipher.java:657)
... 21 more
Caused by: java.util.jar.JarException: Class is on the bootclasspath

These are are related as far as I can tell. A teammate figured out Bouncy Castle problem has to be done on a build slave if you are using those. I'm not sure if it needs to be done on the Jenkins master too, but I did it there first already.

I think java 6 does not work with newer Jenkins? I did not test fix with java 7. I used Java 8

add these 4 jar files here:

/…./java/jdk/jdk-1.8u66/jre/lib/ext

rw-r r - 1 root root 2070477 Feb 25 16:46 bcprov-ext-jdk15on-147.jar
rw-r r - 1 root root 1997327 Feb 25 16:46 bcprov-jdk15on-147.jar

/…./java/jdk/jdk-1.8u66/jre/lib/security

rw-r r - 1 root root 3023 Feb 25 16:46 US_export_policy.jar
rw-r r - 1 root root 3035 Feb 25 16:46 local_policy.jar

vi /…./java/jdk/jdk-1.8u66/jre/lib/security/java.security
add this line after lines like it:
security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider

A build slave may connect without these files in place on the build slave java, BUT
a build job using ssh-agent will not work !

after making these changes, you need to disconnect and reconnect the build slave.
Then build jobs using ssh-agent with a credential with a passphrase and the private key will work for things like:
ssh myUser@hostwhatever "hostname; pwd; id"

jglick@cloudbees.com (JIRA)

unread,
Mar 1, 2016, 2:02:02 PM3/1/16
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to stephenconnolly
 

stephenconnolly made various changes in 1.9 including to BouncyCastle (I think).

Best to do builds on slaves anyway.

Change By: Jesse Glick
Assignee: stephenconnolly

stephenconnolly@java.net (JIRA)

unread,
Mar 7, 2016, 7:16:08 AM3/7/16
to jenkinsc...@googlegroups.com
stephenconnolly commented on Bug JENKINS-32120
 
Re: With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"

So the issue here is that https://issues.jenkins-ci.org/browse/JENKINS-30110 required a lot of plugins upgrade the agreedupon bouncycastle implementation version... and bouncycastle is notorious for breaking compatibility.

So you really need to either stay all below or all above the bouncycastle version change

aheritier@apache.org (JIRA)

unread,
May 31, 2016, 9:46:02 AM5/31/16
to jenkinsc...@googlegroups.com

Should be fixed is using Jenkins core >= 1.648 and ssh-agent 1.9

aik.bold@gmail.com (JIRA)

unread,
May 31, 2016, 9:59:03 AM5/31/16
to jenkinsc...@googlegroups.com

These are great news. If only I can do "yum update jenkins", that currently doesn't work due INFRA-685.

aheritier@apache.org (JIRA)

unread,
May 31, 2016, 11:40:03 AM5/31/16
to jenkinsc...@googlegroups.com

Alexander Obuhovich take care that the upgrade of Jenkins core >= 1.648 upgrades the library Bouncycastle which fixes this issue BUT as mentioned by stephenconnolly in

JENKINS-30110 it may/will create incompatibilities with all plugins that may use directly bouncycastle and the recent APIs they broke (again).
Various jenkins developers are trying to identify all impacted plugins and to propose a fix which may be better in long term.

aik.bold@gmail.com (JIRA)

unread,
May 31, 2016, 3:12:01 PM5/31/16
to jenkinsc...@googlegroups.com

I've asked DevOps team to do the upgrade and what they did is downloaded RPM package by hand and installed it. Now I can confirm, that issue (with ssh-agent) is indeed solved for me as well.

Thanks.

amuniz@cloudbees.com (JIRA)

unread,
Jun 14, 2016, 6:09:07 AM6/14/16
to jenkinsc...@googlegroups.com

So, can this be closed?

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

aik.bold@gmail.com (JIRA)

unread,
Jun 14, 2016, 6:18:02 AM6/14/16
to jenkinsc...@googlegroups.com

Not happening for me. Not sure if it was fixed for other people participating in discussion.

jdrawneek@nationaltheatre.org.uk (JIRA)

unread,
Jun 14, 2016, 6:38:02 AM6/14/16
to jenkinsc...@googlegroups.com

I get this issue with Java 8 but not with Java 7 with 1.651.2 LTS

riccardo.russo79@gmail.com (JIRA)

unread,
Jun 15, 2016, 4:47:02 AM6/15/16
to jenkinsc...@googlegroups.com

Me too, still having this issue.

[Pipeline] sshagent
[ssh-agent] Using credentials prova
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Java/JNR ssh-agent
[ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
	at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)

org.jenkins-ci.main:jenkins-war:2.9
org.bouncycastle:bcpkix-jdk15on:1.54
org.bouncycastle:bcprov-jdk15on:1.54
SSH Agent Plugin 1.11
SSH Credentials Plugin 1.12

riccardo.russo79@gmail.com (JIRA)

unread,
Jun 15, 2016, 4:53:03 AM6/15/16
to jenkinsc...@googlegroups.com
Riccardo Russo edited a comment on Bug JENKINS-32120
Me too, still having this issue.


{code:java}

[Pipeline] sshagent
[ssh-agent] Using credentials prova
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Java/JNR ssh-agent
[ssh-agent] Unable to read key: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
org.bouncycastle.openssl.PEMException: Unable to create OpenSSL PBDKF: PBKDF-OpenSSL SecretKeyFactory not available
at org.bouncycastle.openssl.jcajce.PEMUtilities.getKey(Unknown Source)
{code}

org.jenkins-ci.main:jenkins-war:2.9
org. bouncycastle :bcpkix - jdk15on: api 1. 54 648
org.bouncycastle:bcprov-jdk15on:1.54
SSH Agent Plugin 1.11
SSH Credentials Plugin 1.12
Running on slave

alobato@cloudbees.com (JIRA)

unread,
Jun 15, 2016, 1:04:11 PM6/15/16
to jenkinsc...@googlegroups.com

alobato@cloudbees.com (JIRA)

unread,
Jun 15, 2016, 1:04:15 PM6/15/16
to jenkinsc...@googlegroups.com
Alvaro Lobato started work on Bug JENKINS-32120
 
Change By: Alvaro Lobato
Status: Open In Progress

alobato@cloudbees.com (JIRA)

unread,
Jun 15, 2016, 2:42:01 PM6/15/16
to jenkinsc...@googlegroups.com
Alvaro Lobato commented on Bug JENKINS-32120
 
Re: With ssh-agent 1.9, local builds fail with "Unable to create OpenSSL PBDKF"

The problems are related with the Bouncy Castle version conflicts, the solution is to use Bouncy Castle API plugin to do all the Bouncy Castle related stuff. See Bouncy Castle API Plugin.

Some of the problems are related with BC not being correctly registered on build agents, I've submitted a PR hopefully fixing those cases: PR-14

riccardo.russo79@gmail.com (JIRA)

unread,
Jun 16, 2016, 4:25:03 AM6/16/16
to jenkinsc...@googlegroups.com

Thanks!

Because I'm really a newbie, this means that from Update Center in Jenkins it will be possible to download a new release?

alobato@cloudbees.com (JIRA)

unread,
Jun 16, 2016, 4:31:02 AM6/16/16
to jenkinsc...@googlegroups.com

This means that the code for the fix has been submitted. It has to be reviewed and after that a released can be cut so you have it available on the Update Center.

alobato@cloudbees.com (JIRA)

unread,
Jun 23, 2016, 5:10:01 AM6/23/16
to jenkinsc...@googlegroups.com

alobato@cloudbees.com (JIRA)

unread,
Jun 23, 2016, 5:28:05 AM6/23/16
to jenkinsc...@googlegroups.com
Alvaro Lobato resolved as Fixed
 
Change By: Alvaro Lobato
Status: In Progress Resolved
Resolution: Fixed

glon.sebastien@free.fr (JIRA)

unread,
Nov 30, 2016, 3:43:02 AM11/30/16
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages