Git repository connection times out sometimes
Jennifer Hofmeister
Hello,
I have several build jobs that clone from a remote Git repository. On some of them, the git fetch part of the command fails with the following output:
Building on master in workspace C:\Program Files (x86)\Jenkins\jobs\test_123\workspace
> git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git.exe config remote.origin.url ssh://g...@stash.mycompany.com:port/path/test-project1.git # timeout=10
Fetching upstream changes from ssh://g...@stash.mycompany.com:port/path/test-project1.git
> git.exe --version # timeout=10
> git.exe -c core.askpass=true fetch --tags --progress ssh://g...@stash.mycompany.com:port/path/test-project1.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Timeout after 10 minutes
ERROR: Error fetching remote repo 'origin'
ERROR: Error fetching remote repo 'origin'
I get no more information. The process doesn’t get rejected by Stash, it just times out waiting for admission. The odd thing is that at the same time, other builds that do the very same thing will execute. On some builds, the effect even seems to occur arbitrarily, or not. Only one job really never seems to work (the one above).
This just started happening on Monday. No changes or updates were made to either Stash or Jenkins (Windows 7 x64, v1.580.1). All connections are going out from this one machine, and the Stash connection has been working fine up until now. I tried upgrading the Git plugin, without effect, so now it is back at 2.3.4.
Any ideas are appreciated! (A while ago, I had a similar inexplicable issue with Subversion I never got to solve, which was terribly frustrating. So any help is very much appreciated, really.)
All the best,
Jennifer
John Mellor
That sounds more like network issues that git issues. Is someone playing with duplicate IPs on your network and triggering spanning-tree or something?
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CAA587D532C45541B62417404F1D22030E81D7A4%40you-exch1.younicos.local.
For more options, visit https://groups.google.com/d/optout.
Jennifer Hofmeister
Hi,
thanks for your idea! It sounded promising, however, there don’t seem to be any funny eddies in the network, and definitely no duplicate IPs. I now suspect a problem on the repository server’s side.
Thanks,
Jennifer
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a7da27796098470682a07ac46d61372a%40mbx01colo01p.esentire.local.
Dunnigan, Terrence J
Your job runs on the Jenkins master server.
What version of the git client do you have installed on your Jenkins master?
I have seen issues where earlier versions of git for Windows (pre-1.9.5) will hang during routine operations like git fetch. I’ve seen this happen both at my desk and in our Jenkins ecosystem.
The workaround pre-1.9.5 was to install Putty and set the GIT_SSH environment variable to use Putty’s plink.exe.
With git 1.9.5 they have seemingly improved performance such that using Putty is no longer necessary.
Terry
From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com]
On Behalf Of Jennifer Hofmeister
Sent: Thursday, April 16, 2015 4:38 AM
To: jenkins...@googlegroups.com
Subject: Git repository connection times out sometimes
Hello,
--
Mark Waite
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9AD8B655B5126D4FAE1397203CDDBF2D167B6A%40XCH-PHX-205.sw.nos.boeing.com.
Dunnigan, Terrence J
I noticed the improvement in 1.9.5, but looking at the release notes it appears that the upgrades have been happening for a while.
https://github.com/msysgit/msysgit/releases/tag/Git-1.9.5-preview20150319
https://github.com/msysgit/msysgit/releases/tag/Git-1.9.4-preview20140815
On git 1.8.3:
$ ssh -v
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
On git 1.9.5:
$ ssh -V
OpenSSH_6.6.1p1, OpenSSL 1.0.1i 6 Aug 2014
Yay to the git team!
Terry
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtE0BH9pUzBy3m0fUOJP0FsY4m9jLkEQEBU2yQ1o8bG6gQ%40mail.gmail.com.
Jennifer Hofmeister
Hi,
just to thank everyone for your ideas and to give an update:
It was no network issue, everything looked clean on that side. Also, it was no git client issue on the Jenkins master side, I updated it along with the plugin with no effect. (Although on the repository side, it was 1.7.11.8 … so, ancient … no idea if that caused any trouble. Didn’t want to meddle on that system yet, though.)
Somehow, the server side didn’t want the master’s Git SSH key anymore. So I tried things out on my own system, where it worked, and copied my keys to the Git/.ssh directory on Jenkins master. That fixed the connection, but the Jenkins job still would not work compared to the command line. Since the only difference between these two is that the Jenkins job, running as a service, starts Git as SYSTEM, and cmd starts it as USER, I reconfigured the Jenkins service to run as USER. Problem solved for me.
Thanks for everything, though, it helped me learn a lot!
Jennifer
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9AD8B655B5126D4FAE1397203CDDBF2D167C53%40XCH-PHX-205.sw.nos.boeing.com.
web-deploy
I have encountered same problem starting last week. My Jenkins master and slaves are running on linux tho.
...
ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
...
Dunnigan, Terrence J
Hi,
I’m also encountering that very error. It seems to happen at random on various nodes.
Building remotely on Goku in workspace C:\Hudson_remote\workspace\code.git
> C:\Program Files (x86)\Git\cmd\git.exe rev-parse --is-inside-work-tree # timeout=120
Fetching changes from the remote Git repository
> C:\Program Files (x86)\Git\cmd\git.exe config remote.origin.url git@server:code # timeout=120
ERROR: Error fetching remote repo 'origin'
ERROR: Error fetching remote repo 'origin'
I’m using Jenkins LTS 1.544.3 on a Windows 2008 server running Java 7.
Git plugin 2.2.9
Git client plugin 1.12.0
The nodes are all running Windows 7, Java 7, and either 1.8.3 or git 1.9.5.
Terry
From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com]
On Behalf Of web-deploy
Sent: Wednesday, May 06, 2015 11:33 AM
To: jenkins...@googlegroups.com
Cc: jennifer....@younicos.com
Subject: Re: Git repository connection times out sometimes
Hi,
I haven't encountered similar problem starting last week. Our Jenkins master and slaves both run on Linux tho.
...
ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
...
Anyone has any ideas?
On Wednesday, April 22, 2015 at 1:59:32 AM UTC-7, Jennifer Hofmeister wrote:
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d56089e4-331f-478e-a45a-c191dbdd0ebc%40googlegroups.com.