Jenkins Git suddenly stopped working

[mwpowellhtx]

I am experiencing the following error(s).

I've been working on build jobs today in Jenkins and polling our online repository hosting frequently. It's been working most of today until recently the past hour or two. Now suddenly I am receiving the following error(s), for one job I am currently working on, and verified, a prior job I got working earlier is now breaking for the same reason.

I've also contacted our host thinking that perhaps the issue is with them. The strange part is that I can still clone from the command line, but Jenkins is consistently failing where Jenkins was working earlier.

Any suggestions?

Thank you...

Started by user anonymous
[EnvInject] - Loading node environment variables.
Building in workspace <workspace/>
Checkout:workspace / <workspace/> - hudson.remoting.LocalChannel@18f57d2
Using strategy: Default
Last Built Revision: Revision <sha/> (origin/master)
Fetching changes from 1 remote Git repository
Fetching upstream changes from ssh://git@<repo-host/>/<path/>/<git-repo/>.git
ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway
hudson.plugins.git.GitException: Command "git.exe fetch -t ssh://git@<repo-host/>/<path/>/<git-repo/>.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: ssh: <repo-host/>: no address associated with name
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

	at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:897)
	at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:858)
	at hudson.plugins.git.GitAPI.fetch(GitAPI.java:200)
	at hudson.plugins.git.GitAPI.fetch(GitAPI.java:1105)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:813)
	at hudson.plugins.git.GitSCM.access$100(GitSCM.java:72)
	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1018)
	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:986)
	at hudson.FilePath.act(FilePath.java:852)
	at hudson.FilePath.act(FilePath.java:825)
	at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:986)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1142)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1324)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
	at hudson.model.Run.execute(Run.java:1518)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:236)
ERROR: Could not fetch from any repository
FATAL: Could not fetch from any repository
hudson.plugins.git.GitException: Could not fetch from any repository
	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1025)
	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:986)
	at hudson.FilePath.act(FilePath.java:852)
	at hudson.FilePath.act(FilePath.java:825)
	at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:986)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1142)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1324)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
	at hudson.model.Run.execute(Run.java:1518)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:236)

[Scott Evans]

Looks like you're missing an environment setting for repo-host

Scott

[mwpowellhtx]

I'm not sure what you mean. Can you be more specific? It's possible something got unset, but like I said, it was working all day until just now.

[mwpowellhtx]

To clarify, I've edited the actual names out. The names are there, it was working earlier. Now suddenly it has stopped working.

A colleague of mine and I are wondering whether it's a SSH thing, or perhaps there's a limit set by our host, or something.

[Scott Evans]

The error would indicate to me that it can't find the host where your repo is, whether a typo in that variable setting that's not getting resolved through your nameserver, or some sort of network issue, most likely.

[mwpowellhtx]

And I can assure you it's not. I copy and paste the exact same address on the command line, and I can clone from the command line. Only Jenkins is now consistently failing across the board for jobs that were previously succeeding.

We're speculating whether repositoryhosting.com (our host) has some limits set on SSH.

[Andrew Melo]

On Thu, Dec 13, 2012 at 11:24 PM, mwpowellhtx <mwpow...@gmail.com> wrote:
> And I can assure you it's not. I copy and paste the exact same address on
> the command line, and I can clone from the command line. Only Jenkins is now
> consistently failing across the board for jobs that were previously
> succeeding.
>
> We're speculating whether repositoryhosting.com (our host) has some limits
> set on SSH.

This seems to look like your DNS isnt resolving the host name right,
it's not (necessarily) a matter of you fumbling the hostname...

stderr: ssh: <repo-host/>: no address associated with name

fatal: Could not read from remote repository.

If you connect to your slaves, can you do a git clone manually?

>
>
> On Thursday, December 13, 2012 3:59:42 PM UTC-6, SA Evans wrote:
>>
>> The error would indicate to me that it can't find the host where your repo
>> is, whether a typo in that variable setting that's not getting resolved
>> through your nameserver, or some sort of network issue, most likely.
>>
>>
>>
>> On Thu, Dec 13, 2012 at 3:47 PM, mwpowellhtx <mwpow...@gmail.com> wrote:
>>>
>>> To clarify, I've edited the actual names out. The names are there, it was
>>> working earlier. Now suddenly it has stopped working.
>>>
>>> A colleague of mine and I are wondering whether it's a SSH thing, or
>>> perhaps there's a limit set by our host, or something.
>>>
>>> On Thursday, December 13, 2012 3:43:44 PM UTC-6, mwpowellhtx wrote:
>>>>
>>>> I'm not sure what you mean. Can you be more specific? It's possible
>>>> something got unset, but like I said, it was working all day until just now.
>>>>
>>>> On Thursday, December 13, 2012 3:38:27 PM UTC-6, SA Evans wrote:
>>>>>
>>>>> Looks like you're missing an environment setting for repo-host
>>>>>
>>>>> Scott
>>>>>
>>>>> On Thu, Dec 13, 2012 at 3:35 PM, mwpowellhtx <mwpow...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> stderr: ssh: <repo-host/>: no address associated with name
>>>>>
>>>>>
>>
>

--
--
Andrew Melo

[mwpowellhtx]

How do you mean, "connect to your slaves, clone manually"? From a command line or Git Bash? Yes, I can clone manually with the same (copy and paste) SSH URI.

[Andrew Melo]

I mean, presumably the machine you work on day to day isn't the machine your Jenkins slaves run on (since it seems like you have a larger install)

If you SSH to your Jenkins master/slaves, can you clone from them?

----

Andrew Melo

Sent from my secret fortress.

[mwpowellhtx]

No, Jenkins is running on the same machine as my day to day development. Not the best of ideas I know.

For the second part, yes, I can clone from a command line or Git Bash.

[mwpowellhtx]

Don't know what happened. Turns out it was a strange DNS issue (as folks pointed out). What makes it strange is why it would fail for Jenkins and still succeed for a command line Git Bash.

To workaround I added an entry in my hosts file and that has "fixed" at least my part of the issue. Still in question is whether whatever DNS services are up and running properly.

[Scott Evans]

Not sure what OS you're running on, but could it be that the user that the Jenkins agent is running as might not have permissions to access network resources to do a DNS lookup?  Seems odd, but that's the only thing that comes to mind for this case.

Scott

[mwpowellhtx]

I checked this, and Jenkins is running as a Windows Service logging in as System. It doesn't get any more permissive than that. It is running as "anonymous" during the job itself. Not sure how to set that other than null or anonymous?

[mwpowellhtx]

The mystery still has me puzzled: Git Bash worked cloning, fetching, pulling. Yet Jenkins persistently failed even after repeated attempts restarting Jenkins Windows Service. It sounds plausible that Windows DNS Cache Service needed restarting, but then why did Git Bash work and Jenkins service did not? Are there any other services upon which Jenkins depends that might cause something like this? For now I've got an entry in my hosts file, but I still don't think I should have to do this, provided of course DNS is working right.

[Scott Evans]

In your jenkins build, can you try just running a command "nslookup" (or ping would probably work too) on your agent machine to see what response you get from actually within the build session?  That might indicate whether you're even getting a valid lookup back from DNS from within your job environment. 

Might need to set up a new temp job which doesn't do any Git calls, etc. to actually get to your build steps.

Scott

[Jeff]

I wasn't going to say anything but I think I can explain what the issue possibly was since you mentioned Windows is in use.

I don't believe the Java stack works with the Windows networking layer at the same level.  I have problems all the time with DNS lookups in our Java apps because our company configures the standard DNS search paths as part of the system configuration so that the full domain name is not required when hitting a service.

For example our intranet can be reached by doing http://intranet/  because the domain is in the search path.  This works fine for windows binaries that use the full windows networking stack but I must use the fully qualified name in Java.

The same holds true for our database host names.  In Microsoft's SQL Server Management Studio, I can use the host name of the database such as devdb1, but in Java connecting to the same database using JDBC, I must use devdb1.mycompany.com or it won't find it.

I believe the same holds true for the DNS cache.  DNS lookups in windows will utilize the cache if available even if the DNS service was having issues.  The Java stacks don't use the Windows DNS cache.

This explains why Jenkins wouldn't work but command-line did, even with the DNS service down.

Just my $.02.

--

Jeff Vincent
preda...@gmail.com
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !!