[JIRA] (JENKINS-60297) Jenkins could not read from remote repository

27 views
Skip to first unread message

b.loeffler@envea.global (JIRA)

unread,
Nov 27, 2019, 3:37:03 AM11/27/19
to jenkinsc...@googlegroups.com
Benjamin Loeffler created an issue
 
Jenkins / Bug JENKINS-60297
Jenkins could not read from remote repository
Issue Type: Bug Bug
Assignee: Mark Waite
Components: git-plugin
Created: 2019-11-27 08:36
Environment: ace-editor 1.1
analysis-core 1.96
analysis-model-api 7.0.2
ant 1.10
antexec 1.11
antisamy-markup-formatter 1.6
apache-httpcomponents-client-4-api 4.5.10-2.0
bouncycastle-api 2.17
branch-api 2.5.4
cloudbees-folder 6.9
cobertura 1.15
code-coverage-api 1.1.1
command-launcher 1.3
config-file-provider 3.6.2
credentials 2.3.0
display-url-api 2.3.2
external-monitor-job 1.7
forensics-api 0.6.1
git 4.0.0
git-client 3.0.0
greenballs 1.15
jackson2-api 2.10.0
javadoc 1.5
jdk-tool 1.3
jquery 1.12.4-1
jquery-detached 1.2.1
jsch 0.1.55.1
junit 1.28
junit-attachments 1.6
ldap 1.21
mailer 1.29
managed-scripts 1.4
matrix-auth 2.5
matrix-project 1.14
maven-plugin 3.4
pam-auth 1.6
scm-api 2.6.3
script-security 1.67
ssh-credentials 1.18
ssh-slaves 1.31.0
structs 1.20
token-macro 2.10
trilead-api 1.0.5
warnings 5.0.1
warnings-ng 7.2.0
windows-slaves 1.5
workflow-api 2.37
workflow-cps 2.76
workflow-job 2.36
workflow-scm-step 2.9
workflow-step-api 2.20
workflow-support 3.3
Labels: jenkins git git-plugin
Priority: Critical Critical
Reporter: Benjamin Loeffler

Jenkins fails sometimes with the error: Could not read from remote repository.

Jenkins runs on a local Win10 machine with access rights to our local server. On the server there is the GIT repository stored. The WIN10 user account jenkins runs the jenkins-service with has access rights to the server. Communication to the server is stable.

This happend after a update from jenkins 2.176.3 to 2.190.2 (and update of all plugins)

 

The following console output is generated:

Build wurde zeitgesteuert ausgelöst.
Running as SYSTEM
Baue auf Master in Arbeitsbereich C:\jenkins\workspace\xxxxx
No credentials specified
> git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git.exe config remote.origin.url //xxxx/xxxxx/xxxx/xxxx/xxxx/xxxx.git # timeout=10
Cleaning workspace
> git.exe rev-parse --verify HEAD # timeout=10
Resetting working tree
> git.exe reset --hard # timeout=10
> git.exe clean -fdx # timeout=10
> git.exe submodule foreach --recursive git reset --hard # timeout=10
> git.exe submodule foreach --recursive git clean -fdx # timeout=10
Fetching upstream changes from //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git
> git.exe --version # timeout=10
> git.exe fetch --tags --force --progress – //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git +refs/heads/:refs/remotes/origin/ # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:899)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1114)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1145)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --force --progress – //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git +refs/heads/:refs/remotes/origin/" returned status code 128:
stdout:
stderr: fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2372)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1985)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:80)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:563)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:897)
... 11 more
ERROR: Error fetching remote repo 'origin'
[WARNINGS] Skipping publisher since build result is FAILURE
Archiviere Artefakte
Sending e-mails to: xx...@xxxx.com xx...@xxxx.com
Finished: FAILURE

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

b.loeffler@envea.global (JIRA)

unread,
Nov 27, 2019, 3:58:03 AM11/27/19
to jenkinsc...@googlegroups.com
Benjamin Loeffler updated an issue
Change By: Benjamin Loeffler
Jenkins fails sometimes with the error: Could not read from remote repository.

Jenkins runs on a local Win10 machine with access rights to our local server. On the server there is the GIT repository stored. The WIN10 user account jenkins runs the jenkins-service with has access rights to the server. Communication to the server is stable.

This happend after a update from jenkins 2.176.3 to 2.190.2 (and update of all plugins) .

The error occurs very often, if more than one build-processor is used (parallel builds). If only one build-processor is used, the error occurs less, but it still occurs.
 

The following console output is generated:

Build wurde zeitgesteuert ausgelöst.
Running as SYSTEM
Baue auf Master in Arbeitsbereich C:\jenkins\workspace\xxxxx
No credentials specified
> git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git.exe config remote.origin.url //xxxx/xxxxx/xxxx/xxxx/xxxx/xxxx.git # timeout=10
Cleaning workspace
> git.exe rev-parse --verify HEAD # timeout=10
Resetting working tree
> git.exe reset --hard # timeout=10
> git.exe clean -fdx # timeout=10
> git.exe submodule foreach --recursive git reset --hard # timeout=10
> git.exe submodule foreach --recursive git clean -fdx # timeout=10
Fetching upstream changes from //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git
> git.exe --version # timeout=10
> git.exe fetch --tags --force --progress -- – //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git +refs/heads/*:refs/remotes/origin/* # timeout=10

ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:899)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1114)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1145)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --force --progress -- – //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:

stdout:
stderr: fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2372)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1985)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:80)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:563)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:897)
... 11 more
ERROR: Error fetching remote repo 'origin'
[WARNINGS] Skipping publisher since build result is FAILURE
Archiviere Artefakte
Sending e-mails to: xx...@xxxx.com xx...@xxxx.com
Finished: FAILURE

 

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 27, 2019, 2:59:04 PM11/27/19
to jenkinsc...@googlegroups.com
Change By: Mark Waite
Status: Open Fixed but Unreleased
Resolution: Not A Defect

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 27, 2019, 2:59:04 PM11/27/19
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 27, 2019, 2:59:05 PM11/27/19
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-60297
 
Re: Jenkins could not read from remote repository

The message from command line git is:

{{ fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository}}

If you're accessing the remote git repository through https or ssh, that might indicate that the remote git repository is temporarily inaccessible on that remote server has some issue with its file system. In my case, I can generate that message when I use a a valid ssh private key credential to clone a repository which does not exist on a remote server which I host myself. My example looks like this:

mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone mwa...@git.markwaite.net:git/bare/jenkins/not-really-a-repository.git
Cloning into 'not-really-a-repository'...
fatal: 'git/bare/jenkins/not-really-a-repository.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

However, when I attempt to use a valid ssh private key to clone a non-existent repository from github.com, I don't see that exact message. I see:

mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone g...@github.com:MarkEWaite/not-really-a-repository.git
Cloning into 'not-really-a-repository'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Based on that, I would guess that you're cloning a repository from a server that you're hosting. That internal server is temporarily unable to detect the repository, especially during times of high load (concurrent clones). You could test that theory by running multiple git clone commands on different machines at the same time.

If you're accessing the remote git repository through file system access (for example through an NFS mount or a CIFS mount), that message might indicate that command line git was unable to read the directory it needed. It might also mean that it was unable to find a specific file or to acquire a lock on a specific file. If you're accessing a git repository through a remote file access protocol, you should reconsider that choice and use the remote protocols that are included with command line git (http/https and ssh).

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 27, 2019, 3:00:03 PM11/27/19
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Bug JENKINS-60297
The message from command line git is:

{ { noformat}
fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository

{noformat } }

If you're accessing the remote git repository through https or ssh, that might indicate that the remote git repository is temporarily inaccessible on that remote server has some issue with its file system.  In my case, I can generate that message when I use a a valid ssh private key credential to clone a repository which does not exist on a remote server which I host myself.  My example looks like this:

{noformat}

mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone mwa...@git.markwaite.net:git/bare/jenkins/not-really-a-repository.git
Cloning into 'not-really-a-repository'...
fatal: 'git/bare/jenkins/not-really-a-repository.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
{noformat}


However, when I attempt to use a valid ssh private key to clone a non-existent repository from github.com, I don't see that exact message.  I see:

{noformat}

mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone g...@github.com:MarkEWaite/not-really-a-repository.git
Cloning into 'not-really-a-repository'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
{noformat}


Based on that, I would guess that you're cloning a repository from a server that you're hosting.  That internal server is temporarily unable to detect the repository, especially during times of high load (concurrent clones).  You could test that theory by running multiple {{git clone}} commands on different machines at the same time.

If you're accessing the remote git repository through file system access (for example through an NFS mount or a CIFS mount), that message might indicate that command line git was unable to read the directory it needed.  It might also mean that it was unable to find a specific file or to acquire a lock on a specific file.  If you're accessing a git repository through a remote file access protocol, you should reconsider that choice and use the remote protocols that are included with command line git (http/https and ssh).

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 27, 2019, 3:00:03 PM11/27/19
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Bug JENKINS-60297
The message from command line git is:

{noformat}
fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository
{noformat}

If you're accessing the remote git repository through https or ssh, that might indicate that the remote git repository is temporarily inaccessible on or that remote server has some issue with its file system.  In my case, I can generate that message when I use a a valid ssh private key credential to clone a repository which does not exist on a remote server which I host myself.  My example looks like this:


{noformat}
mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone mwa...@git.markwaite.net:git/bare/jenkins/not-really-a-repository.git
Cloning into 'not-really-a-repository'...
fatal: 'git/bare/jenkins/not-really-a-repository.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
{noformat}

However, when I attempt to use a valid ssh private key to clone a non-existent repository from github.com, I don't see that exact message.  I see:

{noformat}
mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone g...@github.com:MarkEWaite/not-really-a-repository.git
Cloning into 'not-really-a-repository'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
{noformat}

Based on that, I would guess that you're cloning a repository from a server that you're hosting.  That internal server is temporarily unable to detect the repository, especially during times of high load (concurrent clones).  You could test that theory by running multiple {{git clone}} commands on different machines at the same time.

If you're accessing the remote git repository through file system access (for example through an NFS mount or a CIFS mount), that message might indicate that command line git was unable to read the directory it needed.  It might also mean that it was unable to find a specific file or to acquire a lock on a specific file.  If you're accessing a git repository through a remote file access protocol, you should reconsider that choice and use the remote protocols that are included with command line git (http/https and ssh).

b.loeffler@envea.global (JIRA)

unread,
Nov 28, 2019, 3:01:04 AM11/28/19
to jenkinsc...@googlegroups.com

Thanks Mark Waite for this very fast response!

But I cant resolve the issue.

Yes, I clone a repository from a server we host. The error occures, when just a single clone is done and when nothing happens on the server (tried at night and on the early morning when noone is at work).

I clone the GIT repo in the jenkins-job "Source-Code-Management" in "Repository URL" via //SERVERNAME/xxxxx/xxxxx/xxxxx/xxxxx/xxxx.git

No credentials are used. (The error occures too if credentials are used: username and password of the user jenkins is running and which has access rights to the server)

This should work without ssh or https right? Use of ssh or https is not needed, its a internal server and access is only via the LAN.

So whats the poblem that it works sometimes and sometimes not?

b.loeffler@envea.global (JIRA)

unread,
Nov 28, 2019, 3:01:04 AM11/28/19
to jenkinsc...@googlegroups.com
Change By: Benjamin Loeffler
Resolution: Not A Defect
Status: Fixed but Unreleased Reopened

b.loeffler@envea.global (JIRA)

unread,
Nov 28, 2019, 3:02:02 AM11/28/19
to jenkinsc...@googlegroups.com
Benjamin Loeffler assigned an issue to Mark Waite
Change By: Benjamin Loeffler
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 28, 2019, 10:20:03 AM11/28/19
to jenkinsc...@googlegroups.com
Change By: Mark Waite
Status: Reopened Open

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 28, 2019, 10:33:03 AM11/28/19
to jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-60297
 
Re: Jenkins could not read from remote repository

As far as I can tell, the issue is reported by command line git and can only be handled by command line git. The git plugin might be extended to retry on command line git errors, but retry on error is a complication that has not been added to the Jenkins git plugin yet.

I assume that /SERVERNAME/path/to/repository is accessing the git repository through a Windows network file system protocol (likely CIFS). If so, then you'll need to request assistance from the git for Windows development team or the git development team. I have a personal suspicion that the git team does not spend significant effort assuring that the various implementations of CIFS (Windows 7, Windows 10, Windows Server 2008, Windows Server 2012, Windows Server 2016, Windows Server 2019, Samba in various versions on FreeNAS, FreeBSD, Linux, and macOS) provide the necessary file locking and data transfer semantics that command line git assumes.

Refer to this stackoverflow article for information from one user that struggled with a git repository over CIFS.

If you must use CIFS, then you may consider enabling core.ignoreStat in hopes of reducing issues.

I don't plan to spend any further effort on this issue.

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 28, 2019, 10:34:02 AM11/28/19
to jenkinsc...@googlegroups.com
Mark Waite assigned an issue to Unassigned
Change By: Mark Waite
Assignee: Mark Waite

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 28, 2019, 10:34:03 AM11/28/19
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Bug JENKINS-60297
As far as I can tell, the issue is reported by command line git and can only be handled by command line git.  The git plugin might be extended to retry on command line git errors, but retry on error is a complication that has not been added to the Jenkins git plugin yet.

I assume that / / SERVERNAME/path/to/repository is accessing the git repository through a Windows network file system protocol (likely CIFS).  If so, then you'll need to request assistance from the git for Windows development team or the git development team.  I have a personal suspicion that the git team does not spend significant effort assuring that the various implementations of CIFS (Windows 7, Windows 10, Windows Server 2008, Windows Server 2012, Windows Server 2016, Windows Server 2019, Samba in various versions on FreeNAS, FreeBSD, Linux, and macOS) provide the necessary file locking and data transfer semantics that command line git assumes.

Refer to this [stackoverflow article|https://stackoverflow.com/questions/37051071/git-wont-reset-file-permissions-over-cifs-mount] for information from one user that struggled with a git repository over CIFS.

If you must use CIFS, then you may consider enabling [core.ignoreStat|https://git-scm.com/docs/git-config/2.16.0#git-config-coreignoreStat] in hopes of reducing issues.


I don't plan to spend any further effort on this issue.

mark.earl.waite@gmail.com (JIRA)

unread,
Dec 5, 2019, 6:11:02 PM12/5/19
to jenkinsc...@googlegroups.com
Mark Waite closed an issue as Not A Defect
 

Not a defect in the git plugin. Command line git or the remote git server are the most likely places to look
Change By: Mark Waite
Status: Open Closed
Resolution: Not A Defect
Reply all
Reply to author
Forward
0 new messages