Jenkins 2.164 Prompting Terminal for Passphrase

[Randall Becker]

This is a new situation that I have encountered after a clean install of 2.164. Attempting to use an SSH keypair (with passphrase) to talk to BitBucket.org, in a Multibranch Pipeline job. The key is properly configured as an Private Key (entered manually) with a Passphrase.

When I hit Scan Multibranch Pipeline Now, Jenkins immediately prompts the JVM's stderr for the passphrase. As an example:

Enter passphrase for key '/tmp/ssh6766884284790436436.key':

I also get an error in the Multibranch scan log:

 > git ls-remote g...@bitbucket.org:group/repo.git # edited, of course
ERROR: [Wed Apr 24 16:58:41 UTC 2019] Could not update folder level actions from source 73a4292e-,,,,
hudson.plugins.git.GitException: Command "git ls-remote g...@bitbucket.org:group/repo.git" returned status code 128:
stdout: 
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

This is bizarre. I'm running Ubuntu 16.04 using the default JRE. This did not happen on earlier versions of Jenkins that were running in Docker images - I had to remove Docker for VPN reasons.

I'm basically off the air at this point. Did I just miss a plugin or do something silly?

TIA,
Randall

[Mark Waite]

Command line git uses ssh for communications with git servers through the secure shell protocol.  The ssh command is designed to prompt the user for the passphrase.  The ssh command prompts for a passphrase in cases that are sometimes surprising (at least for the git plugin).  You probably have one of those cases.

I would guess that your Jenkins server process (the Java process running the master) is attached to a controlling terminal or it has environment variables set which make the ssh command line process believe that it is attached to a controlling terminal.  That causes the ssh command line to prompt for the passphrase on the private key.

Refer to https://github.com/jenkinsci/git-client-plugin/commit/882ecdfb53d627eeeea130720685681ce2553193 for the commit message that describes the change that was made in the git client plugin to handle that case, and the property you can set to enable that change.

Alternatives you can choose:

1. Run Jenkins without a controlling terminal (for example, if running in Docker, be sure that you "detach")
   
2. Use private keys without passphrases
3. Set the Java property org.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true on the command line that starts the master

I suspect that the critical difference between your prior condition and the current condition is that the Jenkins process is now attached to a controlling terminal.

Mark Waite

 

TIA,
Randall

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cf306a8d-14c2-4ebc-9173-3ddeab0a1020%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Mark Waite]

On Wed, Apr 24, 2019 at 11:32 AM Mark Waite <mark.ea...@gmail.com> wrote:

On Wed, Apr 24, 2019 at 11:09 AM Randall Becker wrote:

This is a new situation that I have encountered after a clean install of 2.164. Attempting to use an SSH keypair (with passphrase) to talk to BitBucket.org, in a Multibranch Pipeline job. The key is properly configured as an Private Key (entered manually) with a Passphrase.

When I hit Scan Multibranch Pipeline Now, Jenkins immediately prompts the JVM's stderr for the passphrase. As an example:

Enter passphrase for key '/tmp/ssh6766884284790436436.key':

I also get an error in the Multibranch scan log:

 > git ls-remote g...@bitbucket.org:group/repo.git # edited, of course
ERROR: [Wed Apr 24 16:58:41 UTC 2019] Could not update folder level actions from source 73a4292e-,,,,
hudson.plugins.git.GitException: Command "git ls-remote g...@bitbucket.org:group/repo.git" returned status code 128:
stdout: 
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

This is bizarre. I'm running Ubuntu 16.04 using the default JRE. This did not happen on earlier versions of Jenkins that were running in Docker images - I had to remove Docker for VPN reasons.

I'm basically off the air at this point. Did I just miss a plugin or do something silly?

Command line git uses ssh for communications with git servers through the secure shell protocol.  The ssh command is designed to prompt the user for the passphrase.  The ssh command prompts for a passphrase in cases that are sometimes surprising (at least for the git plugin).  You probably have one of those cases.

I would guess that your Jenkins server process (the Java process running the master) is attached to a controlling terminal or it has environment variables set which make the ssh command line process believe that it is attached to a controlling terminal.  That causes the ssh command line to prompt for the passphrase on the private key.

Refer to https://github.com/jenkinsci/git-client-plugin/commit/882ecdfb53d627eeeea130720685681ce2553193 for the commit message that describes the change that was made in the git client plugin to handle that case, and the property you can set to enable that change.

Alternatives you can choose:

1. Run Jenkins without a controlling terminal (for example, if running in Docker, be sure that you "detach")
   
2. Use private keys without passphrases
3. Set the Java property org.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true on the command line that starts the master

Alternative 4 may also be viable - enable JGit as a git implementation and use JGit rather than command line git.  The command line git implementation is the reference implementation, but there are many use cases where the JGit implementation is good enough to do the job as well.

Mark Waite

I suspect that the critical difference between your prior condition and the current condition is that the Jenkins process is now attached to a controlling terminal.

Mark Waite

 

TIA,
Randall

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cf306a8d-14c2-4ebc-9173-3ddeab0a1020%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

--

Thanks!

Mark Waite

[Randall Becker]

Thanks, however, I'm not sure this is it. My startup script is:

unset TERM # Not really useful

nohup java \
        -Dorg.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true \
        -jar jenkins.war \
        < /dev/null 2>&1 \
        > jenkins.log &

which should not be attached to a terminal, but is. The define seems to have no effect. There are no SSH variables in the environment. And, most importantly, I cannot use a passphrase-less keypair to talk to bitbucket.org without being fired. Did I miss something?

[Randall Becker]

I'll bite. How can I use JGit instead of git in Jenkins? I thought it was hard-burnt into the Git Plugin.

On Wednesday, April 24, 2019 at 1:08:48 PM UTC-4, Randall Becker wrote:

[Mark Waite]

On Wed, Apr 24, 2019 at 12:05 PM Randall Becker <the.n...@gmail.com> wrote:

Thanks, however, I'm not sure this is it. My startup script is:

unset TERM # Not really useful

nohup java \
        -Dorg.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true \
        -jar jenkins.war \
        < /dev/null 2>&1 \
        > jenkins.log &

which should not be attached to a terminal, but is. The define seems to have no effect. There are no SSH variables in the environment. And, most importantly, I cannot use a passphrase-less keypair to talk to bitbucket.org without being fired. Did I miss something?

-Dorg.jenkinsci.plugins.gitclient.CliGitAPIImpl.useSETSID=true is what I use.  Are you running git client plugin 2.4.2 or later?

You could also try prefixing the java process with setsid instead of nohup, in case that behaves better with command line ssh.

 

On Wednesday, April 24, 2019 at 1:08:48 PM UTC-4, Randall Becker wrote:

This is a new situation that I have encountered after a clean install of 2.164. Attempting to use an SSH keypair (with passphrase) to talk to BitBucket.org, in a Multibranch Pipeline job. The key is properly configured as an Private Key (entered manually) with a Passphrase.

When I hit Scan Multibranch Pipeline Now, Jenkins immediately prompts the JVM's stderr for the passphrase. As an example:

Enter passphrase for key '/tmp/ssh6766884284790436436.key':

I also get an error in the Multibranch scan log:

 > git ls-remote g...@bitbucket.org:group/repo.git # edited, of course
ERROR: [Wed Apr 24 16:58:41 UTC 2019] Could not update folder level actions from source 73a4292e-,,,,
hudson.plugins.git.GitException: Command "git ls-remote g...@bitbucket.org:group/repo.git" returned status code 128:
stdout: 
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

This is bizarre. I'm running Ubuntu 16.04 using the default JRE. This did not happen on earlier versions of Jenkins that were running in Docker images - I had to remove Docker for VPN reasons.

I'm basically off the air at this point. Did I just miss a plugin or do something silly?

TIA,
Randall

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8c70123f-5388-48cc-a6b8-dcca6d36fb00%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Mark Waite]

On Wed, Apr 24, 2019 at 12:10 PM Randall Becker wrote:

I'll bite. How can I use JGit instead of git in Jenkins? I thought it was hard-burnt into the Git Plugin.

In the "Global Tools Configuration" page of "Manage Jenkins", use the "Git" button and the "Add" button under it to add "jgit" as an implementation.  Once you've done that, then jobs will have an additional option to select which git implementation they will use.

 

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/30f56e57-d3de-4e43-96dc-d07aa4ebdcda%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

The Multibranch Pipeline Plugin using JGit ends up incompatible.

 > /usr/local/bin/jgit --version # timeout=10
 > /usr/local/bin/jgit ls-remote --symref g...@Bitbucket.org:nonstop-git/nsgit.git # timeout=10
ERROR: [Wed Apr 24 18:23:01 UTC 2019] Could not update folder level actions from source 0819144a...
hudson.plugins.git.GitException: Command "/usr/local/bin/jgit ls-remote --symref g...@Bitbucket.org:group/repo.git" returned status code 1:
stdout: 
stderr: fatal: "--symref" is not a valid option

Looks like JGit (5.3.0 anyway) is no longer an option for polling. I'm on Git Plugin 3.9.3.

setsid made no difference.

On Wednesday, April 24, 2019 at 2:24:00 PM UTC-4, Mark Waite wrote:

On Wed, Apr 24, 2019 at 12:10 PM Randall Becker wrote:

I'll bite. How can I use JGit instead of git in Jenkins? I thought it was hard-burnt into the Git Plugin.

In the "Global Tools Configuration" page of "Manage Jenkins", use the "Git" button and the "Add" button under it to add "jgit" as an implementation.  Once you've done that, then jobs will have an additional option to select which git implementation they will use.

 

On Wednesday, April 24, 2019 at 1:08:48 PM UTC-4, Randall Becker wrote:

This is a new situation that I have encountered after a clean install of 2.164. Attempting to use an SSH keypair (with passphrase) to talk to BitBucket.org, in a Multibranch Pipeline job. The key is properly configured as an Private Key (entered manually) with a Passphrase.

When I hit Scan Multibranch Pipeline Now, Jenkins immediately prompts the JVM's stderr for the passphrase. As an example:

Enter passphrase for key '/tmp/ssh6766884284790436436.key':

I also get an error in the Multibranch scan log:

 > git ls-remote g...@bitbucket.org:group/repo.git # edited, of course
ERROR: [Wed Apr 24 16:58:41 UTC 2019] Could not update folder level actions from source 73a4292e-,,,,
hudson.plugins.git.GitException: Command "git ls-remote g...@bitbucket.org:group/repo.git" returned status code 128:
stdout: 
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

This is bizarre. I'm running Ubuntu 16.04 using the default JRE. This did not happen on earlier versions of Jenkins that were running in Docker images - I had to remove Docker for VPN reasons.

I'm basically off the air at this point. Did I just miss a plugin or do something silly?

TIA,
Randall

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/30f56e57-d3de-4e43-96dc-d07aa4ebdcda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

I should qualify... the passphrase prompt disappeared when setsid is used, but that still does not allow a passphrase-less keypair.

[Mark Waite]

On Wed, Apr 24, 2019 at 12:33 PM Randall Becker wrote:

The Multibranch Pipeline Plugin using JGit ends up incompatible.

 > /usr/local/bin/jgit --version # timeout=10
 > /usr/local/bin/jgit ls-remote --symref g...@Bitbucket.org:nonstop-git/nsgit.git # timeout=10
ERROR: [Wed Apr 24 18:23:01 UTC 2019] Could not update folder level actions from source 0819144a...
hudson.plugins.git.GitException: Command "/usr/local/bin/jgit ls-remote --symref g...@Bitbucket.org:group/repo.git" returned status code 1:
stdout: 
stderr: fatal: "--symref" is not a valid option

That indicates you created a new name for the command line git implementation and named it 'jgit'.  That's not what you want to do.  

You want to add 'jgit' as one of the available implementations.

The configuration screen should look like this:

 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f475e8c0-ab7a-421b-9bcd-95a85ecb8c67%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Mark Waite]

On Wed, Apr 24, 2019 at 12:35 PM Randall Becker <the.n...@gmail.com> wrote:

I should qualify... the passphrase prompt disappeared when setsid is used, but that still does not allow a passphrase-less keypair.

That may indicate that you have another problem.  If the passphrase prompt disappeared, then ssh should have either completed the ssh authentication or reported an error message.

 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/69755a15-a748-49e6-bb95-a9347afa75cc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

I went back to try to use 2.164 on Docker and am experiencing similar issues. I wonder whether this update is DOA.

On Wednesday, April 24, 2019 at 2:45:16 PM UTC-4, Mark Waite wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/69755a15-a748-49e6-bb95-a9347afa75cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Mark Waite]

I've been using 2.164.1 and 2.164.2 since their release with both Java 8 and Java 11, alternating between various configurations, I'm confident the update is not DOA.

I use bitbucket repositories that are secured with passphrase protected private keys.  I use GitHub repositories that are secured with passphrase protected private keys.  Check your configuration carefully.  Has the passphrase protected private key been disabled in bitbucket?  Is Jenkins reporting any issue with the private key format?

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d54ac82a-1a1d-4b40-968a-d2357addac6d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

I'm rebuilding both docker and standalone images and will compare. The docker image is fine with the key pair after I restarted it. I also had to pick off the host signature using ssh -i blah g...@bitbucket.org... THEN the docker image was able to authenticate. I'm going to try the same on the standalone.

On Wednesday, April 24, 2019 at 2:57:59 PM UTC-4, Mark Waite wrote:

I've been using 2.164.1 and 2.164.2 since their release with both Java 8 and Java 11, alternating between various configurations, I'm confident the update is not DOA.

I use bitbucket repositories that are secured with passphrase protected private keys.  I use GitHub repositories that are secured with passphrase protected private keys.  Check your configuration carefully.  Has the passphrase protected private key been disabled in bitbucket?  Is Jenkins reporting any issue with the private key format?

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d54ac82a-1a1d-4b40-968a-d2357addac6d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Randall Becker]

My docker image is doing fine, but the standalone Jenkins just won't authenticate with either JGit or git. It would be really nice to be able to do this without docker. Is there a standard launch recipe for my situation (in Ubuntu) or is SSH with passphrases just not available anymore?

java -jar jenkins.war

doesn't cut it.

:(

[Mark Waite]

On Wed, Apr 24, 2019 at 4:27 PM Randall Becker <the.n...@gmail.com> wrote:

My docker image is doing fine, but the standalone Jenkins just won't authenticate with either JGit or git. It would be really nice to be able to do this without docker. Is there a standard launch recipe for my situation (in Ubuntu) or is SSH with passphrases just not available anymore?

If ssh with passphrases is not available anymore, that is a catastrophic bug.  I'm reasonably confident that ssh with passphrases continues to be available.  However, I won't be able to configure the test setup until later this evening.  My Ubuntu 16 machine is busy right now running a Docker image that would conflict with the native package install.

 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c4fc1bf8-31ec-4b3c-a1e7-3ae00994a047%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks!

Mark Waite

[Mark Waite]

On Wed, Apr 24, 2019 at 4:54 PM Mark Waite wrote:

On Wed, Apr 24, 2019 at 4:27 PM Randall Becker  wrote:

My docker image is doing fine, but the standalone Jenkins just won't authenticate with either JGit or git. It would be really nice to be able to do this without docker. Is there a standard launch recipe for my situation (in Ubuntu) or is SSH with passphrases just not available anymore?

If ssh with passphrases is not available anymore, that is a catastrophic bug.  I'm reasonably confident that ssh with passphrases continues to be available.  However, I won't be able to configure the test setup until later this evening.  My Ubuntu 16 machine is busy right now running a Docker image that would conflict with the native package install.

I can't duplicate the problem you're seeing.  Steps I took while attempting to duplicate the problem included:

1. Create a passphrase protected RSA private key (no special characters in the passphrase, since the git client plugin is known to have issues with special characters in the passphrase)
   
2. Register the public key of that passphrase protected RSA key with my account on bitbucket.org
3. Update ~/.ssh/config so that ssh access to the bitbucket repository from my account 'mwaite' will use the newly created passphrase protected RSA private key
4. Confirm that git clone from bitbucket.org prompts for the passphrase for that private key and fails if I do not provide that passphrase
5. Confirm that git clone  from bitbucket.org prompts for the passphrase for that private key and succeeds when I provide the correct passphrase
6. Install Jenkins 2.164.2 on a fully patched Ubuntu 16.04 machine using the instructions from https://jenkins.io/doc/book/installing/#debianubuntu.  I installed the recommended plugins from the installation wizard and made no other configuration changes (this installs and runs as the user 'jenkins', not the user 'mwaite')
7. Define a Jenkins credential using the passphrase protected RSA private key
8. Define a Jenkins freestyle job that clones a private repository.  Confirm that without the credential, the job will not clone.  Confirm that with the credential, it clones as expected
9. Enable JGit 
10. Reconfigure the freestyle job to use JGit instead of command line git as the implementation.  Confirm that with the credential, it clones as expected using JGit

As far as I can tell, passphrase protected private keys are working as expected from Ubuntu 16.04 to bitbucket using either a Docker image (with useSETSID=true) or the native package installation (with default value for useSETSID).

Mark Waite

--

Thanks!

Mark Waite

[Mark Waite]

On Wed, Apr 24, 2019 at 6:30 PM Mark Waite <mark.ea...@gmail.com> wrote:

On Wed, Apr 24, 2019 at 4:54 PM Mark Waite wrote:

On Wed, Apr 24, 2019 at 4:27 PM Randall Becker  wrote:

My docker image is doing fine, but the standalone Jenkins just won't authenticate with either JGit or git. It would be really nice to be able to do this without docker. Is there a standard launch recipe for my situation (in Ubuntu) or is SSH with passphrases just not available anymore?

If ssh with passphrases is not available anymore, that is a catastrophic bug.  I'm reasonably confident that ssh with passphrases continues to be available.  However, I won't be able to configure the test setup until later this evening.  My Ubuntu 16 machine is busy right now running a Docker image that would conflict with the native package install.

I can't duplicate the problem you're seeing.  Steps I took while attempting to duplicate the problem included:

1. Create a passphrase protected RSA private key (no special characters in the passphrase, since the git client plugin is known to have issues with special characters in the passphrase)
   
2. Register the public key of that passphrase protected RSA key with my account on bitbucket.org
3. Update ~/.ssh/config so that ssh access to the bitbucket repository from my account 'mwaite' will use the newly created passphrase protected RSA private key
4. Confirm that git clone from bitbucket.org prompts for the passphrase for that private key and fails if I do not provide that passphrase
5. Confirm that git clone  from bitbucket.org prompts for the passphrase for that private key and succeeds when I provide the correct passphrase
6. Install Jenkins 2.164.2 on a fully patched Ubuntu 16.04 machine using the instructions from https://jenkins.io/doc/book/installing/#debianubuntu.  I installed the recommended plugins from the installation wizard and made no other configuration changes (this installs and runs as the user 'jenkins', not the user 'mwaite')
7. Define a Jenkins credential using the passphrase protected RSA private key

I failed to note one subtle value in the credential definition.  Since the clone url provided by bitbucket is 'g...@bitbucket.org', I defined the credential with the username 'git', not with my bitbucket username.  Since my bitbucket username includes the '@' character (mark.ea...@gmail.com), it is not usable as the username portion of the repository URL  If the previous version of git client plugin that you were running was before git client plugin 2.7.3, then the credential may have worked even with the wrong username in the credential.

There was a change made several versions of the git client ago to adapt to a change in OpenSSH.  OpenSSH versions prior to 7.7 would accept an incorrect value for the username and would then override that username with the username that was embedded in the repository URL.  OpenSSH versions 7.7 and later fixed that OpenSSH bug.

The git client plugin had a dependency on that OpenSSH bug.  I don't think that bug affects this case, since the OpenSSH version on Ubuntu 16.04 is 7.2, however, you can read about it at https://issues.jenkins-ci.org/browse/JENKINS-50573

--

Thanks!

Mark Waite