Server ssh errors on pipeline jobs

[Ion Roșca]

Hi All,

I managed to implement the build infrastructure using Jenkin multibranch and simple pipelines. Today I suddenly started getting errors for all the jobs involving pipelines - simple pipeline jobs or jobs created via multibranch plugin. The errors happen on Jenkins server before any of the pipeline code gets chance to be executed. I cleared up the workspace from Jenkins server, job logs - nothing helped.

If I clone the failing job using a different job name, or if I delete the failing job and create a new one from scratch with the same name and settings - the new job still fails.

If I create a brand new job with a new name and the same settings - the job starts working. This means that the access to git via ssh works, and it leads me to thinking about some bugs on the server side.

Do you have any ideas why is this error happening and how can I fix it?

The build log is below.

Thank you,

Ion.

Started by user anonymous
Connecting to https://git.corp.rosca.com/api/v3/ using *****/******
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from 2 remote Git repositories
 > git config remote.origin.url git@git.*****.com:Org/project.git # timeout=10
Fetching upstream changes from git@git.corp.rosca.com:Org/project.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git -c core.askpass=true fetch --tags --progress git@git.corp.rosca.com:Org/project.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from g...@git.corp.rosca.com:Org/project.git
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:792)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1048)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1079)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
        at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
        at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:85)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206)
        at hudson.model.ResourceController.execute(ResourceController.java:98)
        at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress g...@git.corp.rosca.com:Org/project.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: /apps/instances/jenkins_12000/temp/ssh2144286552649045768.sh: line 6: ssh: command not found
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1719)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1463)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63)
        at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314)
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:790)
        ... 8 more

GitHub has been notified of this commit’s build result

ERROR: null
Finished: FAILURE

[gary....@avegant.com]

I am seeing the exact same problem.  It just started happening out of the blue, no known changes to Jenkins or builder node (which is a Mac).

Ion, did you resolve it?

--gary

[Mark Waite]

Did your Jenkins server recently upgrade to git plugin 3.0.0?  There has been at least one report of authentication issues on a Mac running git plugin 3.0.0.

If so, then you might check if a revert of the upgrade to git plugin 3.0.0 resolves the issue.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c7a722ee-e138-4eee-a504-1172f16a4921%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gary Hooper]

It's happening with both 3.0.0 and 2.4.4.  (We just upgraded to 3.0.0 to see if that would help; alas, it did not.)

--gary

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/RPbo-C92-3c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtFQxmXnYk-9_4k4fs5W%2BiOS_8UDsZ0yKA2moS7%3DCr0Edw%40mail.gmail.com.

[Mark Waite]

Some of the possible investigations to perform:

• If the URL you're using is an ssh style or an scp style URL, then you need to check that your private key authentication from the Jenkins agent to the git server are functioning
• If using ssh, you might try the instructions at http://askubuntu.com/questions/336907/really-verbose-way-to-test-git-connection-over-ssh to diagnose git connections over ssh
• If the URL you're using is an http or https URL, then check that your user name and password are working from the Jenkins agent to the git server
• Read the log files on the git server to see if they provide hints of the cause of the failures

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/57E1D7A3.1030402%40avegant.com.

[gary....@avegant.com]

so my problem has been solved.  the issue ended up being a malformed PATH setting in the Mac node configuration.  however, due to a bug in Jenkins, that setting was never active, thus the builds were working fine, until a Jenkins restart for an unrelated issue caused this malformed PATH definition to take effect and break the build.

thanks for the ideas to solve.  using GET_SSH_COMMAND led to the real problem and solution.