[JIRA] [git-client-plugin] (JENKINS-21248) Add shallow update init for submodule

18 views
Skip to first unread message

colin@gibibit.com (JIRA)

unread,
Aug 11, 2015, 1:04:02 PM8/11/15
to jenkinsc...@googlegroups.com
Colin Bennett commented on Improvement JENKINS-21248
 
Re: Add shallow update init for submodule

+1 I would like to see this implemented. That pull request may not be perfect in terms of design purity but it is better than not having the feature.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

zjfroot@gmail.com (JIRA)

unread,
Oct 12, 2015, 10:33:04 AM10/12/15
to jenkinsc...@googlegroups.com

+1 This would help us as well.

man@caleotech.com (JIRA)

unread,
Mar 23, 2016, 6:04:03 AM3/23/16
to jenkinsc...@googlegroups.com

+1 for adding this. We have very large sub-module repositories and would really benefit from this feature.

alexk@Motu.com (JIRA)

unread,
Nov 11, 2016, 2:41:01 PM11/11/16
to jenkinsc...@googlegroups.com

+1 from me too. A clone takes 6min instead of 9sec because I can't do a shallow update. This would really help.

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

fujii.hironori@gmail.com (JIRA)

unread,
Mar 9, 2018, 12:50:02 AM3/9/18
to jenkinsc...@googlegroups.com
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

sa_git@strukton.com (JIRA)

unread,
Mar 21, 2018, 3:54:02 AM3/21/18
to jenkinsc...@googlegroups.com

I really hope the PR will pass the tests and is available soon!

mark.earl.waite@gmail.com (JIRA)

unread,
Aug 13, 2018, 8:57:03 PM8/13/18
to jenkinsc...@googlegroups.com

I've added an acceptance test to my regression test suite that runs inside the docker-lfs configuration. It confirms shallow clone support for submodules works as expected.
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

rene.scheibe@gmail.com (JIRA)

unread,
Aug 14, 2018, 11:46:03 AM8/14/18
to jenkinsc...@googlegroups.com
René Scheibe started work on Improvement JENKINS-21248
 
Change By: René Scheibe
Status: Open In Progress

rene.scheibe@gmail.com (JIRA)

unread,
Aug 14, 2018, 11:47:04 AM8/14/18
to jenkinsc...@googlegroups.com
Change By: René Scheibe
Status: In Progress Fixed but Unreleased
Resolution: Fixed

rene.scheibe@gmail.com (JIRA)

unread,
Aug 14, 2018, 11:50:02 AM8/14/18
to jenkinsc...@googlegroups.com

stefan.verhoeff@here.com (JIRA)

unread,
May 17, 2019, 8:20:07 AM5/17/19
to jenkinsc...@googlegroups.com

Tried this feature in beta git-4.0.0-beta9. But the plugin is refusing the allow the option:

> git submodule init # timeout=10
[WARNING] Git client older than 1.8.4 doesn't support shallow submodule updates. This flag is ignored.

However my Git version should support this, see output of sh 'git --version' in my pipeline script:

+ git --version
git version 2.11.0

Bug in the version detection?
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

mark.earl.waite@gmail.com (JIRA)

unread,
May 17, 2019, 8:49:04 AM5/17/19
to jenkinsc...@googlegroups.com

There could be a bug in the version detection logic, but you should probably first check that the version of git being used by your job is the same as the version reported in the pipeline script sh step.

I've not seen any other bug reports of broken version detection. There are specific automated tests in the plugin to confirm that git version numbers in variouos formats are detected correctly. Those tests don't use 2.11.0 specifically, but they use 2.10.0 and other interesting values near it.

If there is a bug in the detection logic, I'll need to update my acceptance test to exercise that bug.

mark.earl.waite@gmail.com (JIRA)

unread,
May 17, 2019, 8:50:02 AM5/17/19
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Improvement JENKINS-21248
There could be a bug in the [version detection logic|https://github.com/jenkinsci/git-client-plugin/blob/63dd47319914cbe72948e18d83f62c62365ba6c7/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L1269], but you should probably first check that the version of git being used by your job is the same as the version reported in the pipeline script {{sh}} step.

I've not seen any other bug reports of broken version detection. There are [specific automated tests|https://github.com/jenkinsci/git-client-plugin/blob/88def8591098610255185f7d895f4474589bc9e2/src/test/java/org/jenkinsci/plugins/gitclient/CliGitAPIImplTest.java#L99] in the plugin to confirm that git version numbers in variouos various formats are detected correctly.  Those tests don't use 2.11.0 specifically, but they use 2.10.0 and other interesting values near it.

If there is a bug in the detection logic, I'll need to update my [acceptance test|https://github.com/MarkEWaite/jenkins-bugs/blob/52c665176dac63a20b81238c38f25c849e877151/Jenkinsfile#L22] to exercise that bug.

mark.earl.waite@gmail.com (JIRA)

unread,
May 17, 2019, 9:21:04 AM5/17/19
to jenkinsc...@googlegroups.com
Mark Waite edited a comment on Improvement JENKINS-21248
There could be a bug in the [version detection logic|https://github.com/jenkinsci/git-client-plugin/blob/63dd47319914cbe72948e18d83f62c62365ba6c7/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L1269], but you should probably first check that the version of git being used by your job is the same as the version reported in the pipeline script {{sh}} step.

I've not seen any other bug reports of broken version detection. There are [specific automated tests|https://github.com/jenkinsci/git-client-plugin/blob/88def8591098610255185f7d895f4474589bc9e2/src/test/java/org/jenkinsci/plugins/gitclient/CliGitAPIImplTest.java#L99] in the plugin to confirm that git version numbers in various formats are detected correctly.  Those tests don't use 2.11.0 specifically, but they use 2.10.0 and other interesting values near it.


If there is a bug in the detection logic, I'll need to update my [acceptance test|https://github.com/MarkEWaite/jenkins-bugs/blob/52c665176dac63a20b81238c38f25c849e877151/Jenkinsfile#L22] to exercise that bug.


There seem to be some interesting cases in that [acceptance test|https://github.com/MarkEWaite/jenkins-bugs/blob/52c665176dac63a20b81238c38f25c849e877151/Jenkinsfile#L22] when running on command line git version 2.11.0, but the cases that I'm seeing are not related to version checking logic.  In my case, the {{git submodule update}} fails with a warning that there are {{no common commits}}.  The same message does not appear on at least some other command line git versions.  I'll need to extend the acceptance test to execute with each of the versions of command line git in my test environment.  That will then tell me if there are problems related to command line git version.  Note that the issue I'm investigating shows that in my environment I'm able to perform the expected command line git operations.  Thus, at least in my environment, the version check is working, since my investigation could not have started without passing that version check.

mark.earl.waite@gmail.com (JIRA)

unread,
May 17, 2019, 10:12:07 AM5/17/19
to jenkinsc...@googlegroups.com

I've further investigated the JENKINS-21248 acceptance test failures in my environment.

The initial submodule checkout succeeds on the following versions of command line git:

  • 2.21
  • 2.20
  • 2.17
  • 2.16

The initial submodule checkout fails on the following versions of command line git:

  • 2.11
  • 2.10
  • 2.7
  • 2.1

The stack trace in the checkout failure is:

hudson.plugins.git.GitException: Command "git submodule update --reference /var/lib/git/mwaite/bugs/jenkins-bugs.git --depth=1 modules/JENKINS-46504.url" returned status code 1:
stdout: 
stderr: Cloning into 'modules/JENKINS-46504.url'...
warning: no common commits
fatal: reference is not a tree: 0736ba35a0d8c05236e3b71584bc4e149aa5f10a
Unable to checkout '0736ba35a0d8c05236e3b71584bc4e149aa5f10a' in submodule path 'modules/JENKINS-46504.url'

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2298)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1910)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1334)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:75)
	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:64)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to debian8-mwaite
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:957)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
		at jdk.internal.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
		at com.sun.proxy.$Proxy119.execute(Unknown Source)
		at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:148)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1231)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
		at java.base/java.lang.Thread.run(Thread.java:834)
Caused: hudson.plugins.git.GitException
	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:87)
	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:68)
	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1337)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused: java.io.IOException: Could not perform submodule update
	at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:153)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1231)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Prior Failures Everywhere

Failures were happening in that acceptance test prior to my creating a branch on the submodule repository which pointed to the SHA-1 referenced by the parent repository. Once that branch was created, then the newer command line git versions would consistently pass. That earlier failure message was:

hudson.plugins.git.GitException: Command "git submodule update --reference /var/lib/git/mwaite/bugs/jenkins-bugs.git --depth=1 modules/JENKINS-46504.url" returned status code 1:
stdout: 
stderr: Cloning into '/mnt/xvdba/home/jagent/mark-pc2.markwaite.net-agent/workspace/ine-no-localbranch_JENKINS-21248/modules/JENKINS-46504.url'...
warning: no common commits
error: Server does not allow request for unadvertised object 0736ba35a0d8c05236e3b71584bc4e149aa5f10a
Fetched in submodule path 'modules/JENKINS-46504.url', but it did not contain 0736ba35a0d8c05236e3b71584bc4e149aa5f10a. Direct fetching of that commit failed.

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2298)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1910)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1334)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:75)
	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:64)

stefan.verhoeff@here.com (JIRA)

unread,
May 17, 2019, 12:37:04 PM5/17/19
to jenkinsc...@googlegroups.com

Thanks for the details Mark Waite. I'm happy to help with testing this useful feature.

How can I check what version of Git is used by the checkout() command? I assume it would be the same I get when I run git --version just above it. Is there a different Git binary used? Is the checkout actually happening on the Master and the sh() command on the agent? Is JGit involved? I'm confused here.

As per your acceptance test, does this point at an issue with the version parsing or not? I've checked the code around where the warning Git client older than 1.8.4 doesn't support shallow submodule updates. This flag is ignored. is raised, the shallow clone arguments are never actually added to the Git command because of the failed version check, so I'm not getting errors from Git on this.

mark.earl.waite@gmail.com (JIRA)

unread,
May 17, 2019, 1:22:03 PM5/17/19
to jenkinsc...@googlegroups.com

My acceptance test is strong evidence that there is not an issue with version parsing .

Check the jenkins system configuration of the git tool and the agent configuration of the git tool

gegles@gmail.com (JIRA)

unread,
Oct 27, 2019, 2:42:03 AM10/27/19
to jenkinsc...@googlegroups.com

When will this be released?
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

mark.earl.waite@gmail.com (JIRA)

unread,
Oct 27, 2019, 6:44:03 AM10/27/19
to jenkinsc...@googlegroups.com

It is available in git plugiin 4.0.0-beta12 and git client plugin 3.0.0 beta12 now. Please test it in your use case. I intend to release git plugin 4.0.0 and git client plugin 3.0.0 in about 1 week and would sincerely appreciate additional beta testers confirming that the new plugin features are working as expected.

Release Notes

Download

mark.earl.waite@gmail.com (JIRA)

unread,
Nov 2, 2019, 8:07:12 AM11/2/19
to jenkinsc...@googlegroups.com
Mark Waite updated Improvement JENKINS-21248
 

Released with git client plugin 3.0.0 and git plugin 4.0.0 on Nov 2, 2019.
Change By: Mark Waite
Status: Fixed but Unreleased Closed
Reply all
Reply to author
Forward
0 new messages