[JIRA] [git-plugin] (JENKINS-32479) multiple git repositories sometimes fail to checkout some of them into subdirectory

[jacob.keller@gmail.com (JIRA)]

Jacob Keller created an issue

Jenkins / JENKINS-32479 multiple git repositories sometimes fail to checkout some of them into subdirectory Issue Type: Bug Assignee: Mark Waite Components: git-plugin, multiple-scms-plugin Created: 15/Jan/16 10:30 PM Environment: Jenkins 1.643 Linux Multiple SCMs Plugin 0.5 Git Plugin 2.5.0-beta3 Priority: Major Reporter: Jacob Keller I have yet to have a reliable production, and I am not sure if the bug is in Multiple SCM or within the Git plugin. I have 3 repositories, one checked out at the tip of workspace, and two checked out into subdirectories src/A and src/B. Sometimes the src/A or src/B fail to be checked out, despite the log looking perfectly normal, and saying it cloned and checked them out. Sometimes re-running the job simply fixes the issue. A sample of the log that Isee when this issue occurs: Started by upstream project "ixgbe/gerrit_compat" build number 17 originally caused by: Manually triggered by user jekeller for Gerrit: https://git-amr-1.devtools.intel.com/gerrit/74271 Building remotely on rhel5sp11-60c709d8 (rhel5sp11 swarm linux rhel5) in workspace /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5 Cloning the remote Git repository Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat > git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src/COMPAT # timeout=10 Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat > git --version # timeout=10 using GIT_SSH to set credentials ND Linux CI Server > git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat +refs/heads/:refs/remotes/origin/ > git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat # timeout=10 Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat using GIT_SSH to set credentials ND Linux CI Server > git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat refs/changes/71/74271/1 Checking out Revision 73f1f63fb1f4c5da8030d46b3cdf762cefc28e10 (master) > git config core.sparsecheckout # timeout=10 > git checkout -f 73f1f63fb1f4c5da8030d46b3cdf762cefc28e10 First time build. Skipping changelog. Cloning the remote Git repository Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe > git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src/SHARED # timeout=10 Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe > git --version # timeout=10 using GIT_SSH to set credentials ND Linux CI Server > git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe +refs/heads/:refs/remotes/origin/ > git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe # timeout=10 Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe using GIT_SSH to set credentials ND Linux CI Server > git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe +refs/heads/:refs/remotes/origin/ Checking out Revision c0443ba24e527146ce7e8a331047aff366aaa578 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f c0443ba24e527146ce7e8a331047aff366aaa578 First time build. Skipping changelog. Cloning the remote Git repository Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe > git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5 # timeout=10 Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe > git --version # timeout=10 using GIT_SSH to set credentials ND Linux CI Server > git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe +refs/heads/:refs/remotes/origin/ > git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe # timeout=10 Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe using GIT_SSH to set credentials ND Linux CI Server > git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe +refs/heads/:refs/remotes/origin/ Checking out Revision 08bb35e104c5b440e20933cec05f1ed91abcd6b0 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 08bb35e104c5b440e20933cec05f1ed91abcd6b0 First time build. Skipping changelog. [rhel5] $ /bin/bash -el /tmp/hudson1549523813705894193.sh + set -o pipefail ++ echo ixgbe/gerrit_compat/build=PROJECT,label=rhel5 ++ sed s/label=// ++ sed s/BUILD_TYPE=// ++ tr /, - + project=ixgbe-gerrit_compat-build=PROJECT-rhel5 ++ echo ixgbe-gerrit_compat-build=PROJECT-rhel5-17 ++ tr '[:upper:]' '[:lower:]' + name=ixgbe-gerrit_compat-build=project-rhel5-17 + logfile=ixgbe-gerrit_compat-build=project-rhel5-17.log + [[ PROJECT = ESX* ]] + which sparse which: no sparse in (/usr/kerberos/bin:/sbin:/usr/sbin:/bin:/usr/bin:/home/jenkins/bin) + C=0 + CFLAGS='-Wno-missing-field-initializers -Wno-aggregate-return' + CHECKIDS_FLAGS= + W=1 + make -C src -f build.mk TEST_BUILD=YES USE_ISYSTEM=YES CHECKIDS_FLAGS= BUILD=PROJECT + tee ixgbe-gerrit_compat-build=project-rhel5-17.log make: Entering directory `/home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src' build.mk:61: *** Cannot find COMPAT/compat.inc for kcompat flags. Stop. make: Leaving directory `/home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src' Build step 'Execute shell' marked build as failure [WARNINGS] Skipping publisher since build result is FAILURE Archiving artifacts Finished: FAILURE As you can see, 3 repositories are "checked out" according to the docs, but the workspace for this job appears to only include the nd_linux-i40e project, and not the nd_shared-i40e or nd_linux-compat projects. This results in a compilation failure due to inability to locate the COMPAT files. What are the next steps I should take to help debug this issue? Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory This issue can occur if somehow the contents of the submodule directory get deleted such as due to a retry, and it will not be recovered by a submodule update unless the submodule changes locations and is configured to perform a checkout. One solution to this would be to perform a hard reset after the checkout completes. I don't believe this is currently done for submodules even if the clean option is used. It seems that the primary cause of this is due to somehow deleting the contents of the submodule when we perform an update. I am not sure how best to handle this yet. Somehow we end up deleting all the files, and then a future update does not correctly reset the working tree.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

The variant of this caused by a submodule failure is fixed by https://github.com/jenkinsci/git-client-plugin/pull/200 I am still investigating exactly what causes failures in the case with just multiple work trees checked out to subdirectories.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

I believe but have not been able to prove that this is related to git-clean issues.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite assigned an issue to Unassigned

Jenkins / JENKINS-32479

multiple git repositories sometimes fail to checkout some of them into subdirectory

Change By: Mark Waite Assignee: Mark Waite

Add Comment

[dzuczek@dlc-solutions.com (JIRA)]

Devin Zuczek commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory We have this issue as well, projects that were configured with "check out to a subdirectory" sometimes don't. We're also using Multi-SCM.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

Any chance you happen to have a smaller recreation we could use for testing? I'd like to try and sort out exactly what's happening here, as it recently happened again. I believe it may only occur when a fresh clone is initiated. What version of Git command line are you using, and what version of Git and Multi SCM are you using?

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

I found the root cause of the issue is due to using MultiSCM to checkout submodules that exist within the repository. This causes the git commands to fail if it happens to handle the parent project first. I believe this is an issue that should be fixed from within git. An alternative work-around would be to extend the submodule support in Git plugin to enable "checkout submodule to specific sha1sum". I am asking on the git mailing list if this is expected behavior and how best to fix it.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller edited a comment on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

I  found  believe  the root cause  of the issue  is due to  using MultiSCM to checkout submodules that exist within  how  the  repository. This causes the git commands to fail if it happens to handle the parent project first. I believe this is an issue that should be fixed from within git. An alternative work-around would be to extend the submodule support in  Git plugin  to enable  handles clones, via  " git init" + "git fetch" + "git checkout  submodule to specific sha1sum "  which causes it to accidentally remove the other repositories . I am  asking on the git mailing list if  investigating  this  is expected behavior and how best to fix it  now .

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

I found the root cause of the issue is due to lines 453 in the Git-client plugin, it runs a recursive "delete workspace" command, and so entirely depending on the ordering of the repositories the Multiple SCM plugin will fail. I don't even think the options are necessary to use, since a checkout -f will overwrite any files to begin with, and the only other option is implementing our own "clean" setup which would be more expensive than just relying on the clean extension.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller assigned an issue to Jacob Keller

Jenkins / JENKINS-32479

multiple git repositories sometimes fail to checkout some of them into subdirectory

Change By: Jacob Keller Assignee: Jacob Keller

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory So the recursive delete call was originally added since "git clone" does not let you clone into a non-empty directory. The root of this problem is due to how MultiSCM orders its checkouts, and there is no (obvious) way to tell which repositories belong in which directories from within the MultiSCM plugin, as this is part of the Git plugin settings. In addition, we can't really depend on a specific ordering from within the MultiSCM plugin, and we can't reliably remove the restriction that the work space be empty. I believe the problem is that MultiSCM should be made aware of the "checkout to separate directory" option, and indeed should actually implement it itself, since it is a very common or obvious problem when using multiple repositories. That would avoid the issues here without having to remove the constraint (which probably affects other SCMs as well!)

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

An alternative would be to implement getModuleRoots for the Git Plugin to return the actual root when Git is using the extension, and then order checkouts based on this. I am thinking this is simpler and will try to implement this.

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller assigned an issue to Unassigned

Jenkins / JENKINS-32479

multiple git repositories sometimes fail to checkout some of them into subdirectory

Change By: Jacob Keller Assignee: Jacob Keller

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller assigned an issue to Craig Rodrigues

Jenkins / JENKINS-32479 multiple git repositories sometimes fail to checkout some of them into subdirectory

Change By: Jacob Keller Assignee: Craig Rodrigues

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory Can we get some input from developers of Git plugin and Multiple-SCM plugin? The issue stems from ordering of repositories since the Git Plugin's "clone" operation calls a full delete of the current workspace. I have a possible solution revolving around making the Git Plugin report its actual sub directory using the getModuleRoot() interface, and then having Multiple-SCM "sort"/"order" the repositories in such a way as to avoid checking out into a sub directory first. Another option would be for Multiple SCM to implement for itself "checkout to a sub directory" instead of relying on the plugins to do so themselves. I believe this would be preferred since it would be much more universal, and allow easier correct access since my fix requires both a change to the git plugin and the multiple SCM plugin. Finally it may be possible to not need to remove the working directory when implementing clone() in the Git plugin. However, this requires continued use of the init+fetch setup, which currently breaks other uses such as Git-LFS and should be returned back to clone. There does not appear to be a good way to make git-clone behave correctly and allow cloning into a non-empty directory.

Add Comment

[tewing@gmail.com (JIRA)]

Terry Ewing commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory

I think this issue is the same as https://issues.jenkins-ci.org/browse/JENKINS-27668.

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[rodrigc@FreeBSD.org (JIRA)]

Craig Rodrigues resolved as Won't Fix

Suggest that you migrate to Pipeline plugin, which offers a supported way of checking out from multiple scms https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin

Jenkins / JENKINS-32479

multiple git repositories sometimes fail to checkout some of them into subdirectory

Change By: Craig Rodrigues Status: Open Resolved Resolution: Won't Fix

Add Comment

[jacob.keller@gmail.com (JIRA)]

Jacob Keller commented on JENKINS-32479

Re: multiple git repositories sometimes fail to checkout some of them into subdirectory Unfortunately this would still result in the same problem, because the git plugin wipes out the workspace when cloning, so it is still possible to have the issue occur. It is easier to avoid in the pipeline as long as the pipeline steps are ordered correctly. The user just has to know and understand that the git plugin will wipe out the workspace path so you must ensure that the repositories are ordered correctly. In general, for pipeline this is obvious, but it is not obvious for the multi-scm plugin.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite closed an issue as Won't Fix

Jenkins / JENKINS-32479

multiple git repositories sometimes fail to checkout some of them into subdirectory

Change By: Mark Waite Status: Resolved Closed

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)