[JIRA] (JENKINS-26660) Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

瀏覽次數:23 次
跳到第一則未讀訊息

marcus.a.philip@gmail.com (JIRA)

未讀,
2016年9月20日 上午10:33:062016/9/20
收件者:jenkinsc...@googlegroups.com
Marcus Philip updated an issue
 
Jenkins / Bug JENKINS-26660
Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline
Change By: Marcus Philip
Environment: Jenkins and multi SCM plugin configured with git repository using submodules
Alternatively: A pipeline that checks out a second repository in a sub-dir
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

marcus.a.philip@gmail.com (JIRA)

未讀,
2016年9月20日 上午10:33:072016/9/20
收件者:jenkinsc...@googlegroups.com
Marcus Philip updated an issue
Change By: Marcus Philip
Summary:
Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

marcus.a.philip@gmail.com (JIRA)

未讀,
2016年9月20日 上午10:39:032016/9/20
收件者:jenkinsc...@googlegroups.com
Marcus Philip updated an issue
h4. Submodules clean up
Jenkins SCM plugin can handle git repositories with submodules except for clean up.

When "Clean before checkout" is selected, it runs 'git reset --hard && git clean -dxf' on the repository and submodules if "Recursively update submodules" is selected. This works for most stuff but it does not work if git submodule is removed from the tree which leaves the submodule directory in workspace even after 'git clean -dxf'.

If another '-f' is added, also the removed submodule will be deleted which is the behavior I'd expect when git submodules are in use.

So Jenkins git plugin should call 'git clean -dxf -f' in that case.


h4. Pipelines workspace reuse
An alternative use case, which I believe is very common, is that a pipeline checks out a repository in a sub-directory. Since the workspaces are reused between builds, there is a big risk that code or generated files in that sub-directory from previous builds are still there. This can seriously impact reliability of build or even fail them. One case is that old cucumber test reports are found.

Completely wiping the workspace has a performance cost that is unreasonable.

manschwetus@cs-software-gmbh.com (JIRA)

未讀,
2016年9月21日 凌晨1:29:022016/9/21
收件者:jenkinsc...@googlegroups.com
Florian Manschwetus commented on Bug JENKINS-26660
 
Re: Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

Keep in mind a pipeline with multiple scms can and SHOULD set clean for each repo used in its own scm step, the behavior you sketched will wipe the others completely.

marcus.a.philip@gmail.com (JIRA)

未讀,
2016年9月21日 凌晨3:55:022016/9/21
收件者:jenkinsc...@googlegroups.com

Florian Manschwetus I don't understand your comment. If my pipeline for repo A checks out repo B in a later stage, how does adding a 'clean before/after checkout' to B checkout help when running next instance of pipeline and we see the build dir of B before we have gotten to the step where we checkout B?!

Are you suggesting that A pipeline in a first 'init' stage should keep track of what it has checked out in later steps or earlier runs and clean this dirs? I think that is ludicrous and does not follow a good decoupled design.

I still claim that I want to clean the workspace from sub-dirs with repos when I use the 'clean before/after checkout' option. I understand that this means that the first checkout on a node gets special treatment, and I claim that this is exactly what you want.

marcus.a.philip@gmail.com (JIRA)

未讀,
2016年9月21日 凌晨3:58:012016/9/21
收件者:jenkinsc...@googlegroups.com
Marcus Philip edited a comment on Bug JENKINS-26660
In essence the fix is really simple and belongs to the epic category of bugs/features where the actual change is only one character :) Well, if there hadn't been 2 implementations...

h5. CLI
Git cli clean help: "Git will refuse to delete directories with .git sub directory or file unless a second -f is given.

An '{{f}}' needs to be added here:
[CliGitAPIImpl.java Line: 671|https://github.com/jenkinsci/git-client-plugin/blob/8454a9b41e1f88de86302224b4e72df341820552/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L671]

h5. JGit
{{org.eclipse.jgit.api.CleanCommand}} javadoc: 'If force is set, directories that are git repositories will also be deleted.'

And '{{.setForce(true)}}' needs to be added here:
[JGitAPIImpl.java Line: 1300| https://github.com/jenkinsci/git-client-plugin/blob/8454a9b41e1f88de86302224b4e72df341820552/src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java#L1300 ]

I claim this is a bug and not a feature request, because I think in the context of CI I think when you say 'Clean!' you really mean 'Clean everything!'. Thus we can change the behavior without providing any options. Can you imagine any sane use case relying on he current behavior?

marcus.a.philip@gmail.com (JIRA)

未讀,
2016年9月21日 清晨7:33:042016/9/21
收件者:jenkinsc...@googlegroups.com

I started working on a PR in https://github.com/Diabol/git-client-plugin.

However, for the JGit impl, I noticed that the setForce() method was added only very recently in JGit and that change has not been released yet (for some reason they have not done a release since July 15 whereas previously they have done it at least once per month). We need: https://github.com/eclipse/jgit/commit/e1ffab1cac55179011777b9a60bac447b0a62ccf.

Furthermore, the JGit version that's used in git-client is pretty old (3.7.1.201504261725), so upgrading that may be difficult. I have no idea.

marcus.a.philip@gmail.com (JIRA)

未讀,
2016年9月21日 清晨7:36:042016/9/21
收件者:jenkinsc...@googlegroups.com
Marcus Philip updated an issue
Change By: Marcus Philip
h4. Submodules clean up
Jenkins SCM plugin can handle git repositories with submodules except for clean up.

When "Clean before checkout" is selected, it runs 'git reset --hard && git clean -dxf' on the repository and submodules if "Recursively update submodules" is selected. This works for most stuff but it does not work if git submodule is removed from the tree which leaves the submodule directory in workspace even after 'git clean -dxf'.

If another '-f' is added, also the removed submodule will be deleted which is the behavior I'd expect when git submodules are in use.

So Jenkins git plugin should call 'git clean -dxf -f' in that case.

h4. Pipelines workspace reuse
An alternative use case, which I believe is very common, is that a pipeline checks out a repository in a sub-directory. Since the workspaces are reused between builds, there is a big risk that code or generated files in that sub-directory from previous builds are still there in the next run . This can seriously impact reliability of build or even fail them. One case is that old cucumber test reports are found.


Completely wiping the workspace has a performance cost that is unreasonable.

scottwd@gmail.com (JIRA)

未讀,
2016年9月26日 下午1:19:032016/9/26
收件者:jenkinsc...@googlegroups.com
Scott Davis commented on Bug JENKINS-26660
 
Re: Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

I've also noticed that the plugin is using "git config --get-regexp ^submodule" to get the list is submodules to update. This is giving errors when trying to build after a submodule deletion because the local config still has an entry for it. To fix I had to force a fresh clone for one build. Will the fix described above also solve this?

tanyapylatgithub@gmail.com (JIRA)

未讀,
2016年9月27日 凌晨3:57:032016/9/27
收件者:jenkinsc...@googlegroups.com

Scott Davis, as a workaround we've updated git configuration on the machine where Jenkins is running. git clean -f calls now git clean -ff. Another work around is create an alias for git clean -f => git clean -ff .

scottwd@gmail.com (JIRA)

未讀,
2016年9月27日 上午10:54:032016/9/27
收件者:jenkinsc...@googlegroups.com

Thanks Tetyana. I no longer have any broken repos to check this against but will try it if the problem occurs again. I'm betting that -ff won't fix my problem, though, because the clean appears to occur before new changes are checked out. Hence git has no way of knowing that it needs to clean up a newly deleted submodule. Also, the docs say that -ff will clean up paths in .git/modules but don't say anything about .git/config that I can see, which is where the problem starts. I think the real issue is that git config should not be used to enumerate submodules. Running git submodule status or parsing .gitmodules would be more appropriate.

melezhik@gmail.com (JIRA)

未讀,
2016年10月28日 上午8:05:022016/10/28
收件者:jenkinsc...@googlegroups.com

HI! I am just curios , what ETA on this issue? Thanks

(PS. it's really blocker to use git plugin for us )

mark.earl.waite@gmail.com (JIRA)

未讀,
2016年11月3日 晚上8:41:022016/11/3
收件者:jenkinsc...@googlegroups.com

Alexey Melezhik A pull request was submitted requesting to change git clean from "git clean -xfd" to "git clean -xffd". I rejected the pull request because it would silently and significantly change the behavior for all users of the git plugin. There are now over 90 000 installations of the git plugin and I cannot silently and significantly change the behavior of "git clean". Refer to the comments on PR163 for the detailed rationale.

If the "Clean before checkout" and "Clean after checkout" were extended to allow an additional behavior to "clean submodules", or if the submodules additional behavior were extended to optionally allow the user to specify that they want submodules cleaned, I would be willing to consider it. The change would need to be accompanied by automated tests which show the condition without the extra clean, and the condition with the extra clean.

roguishmountain@gmail.com (JIRA)

未讀,
2016年12月6日 下午3:19:042016/12/6
收件者:jenkinsc...@googlegroups.com
Sam Schwarz edited a comment on Bug JENKINS-26660
Hi all,
I created two PRs for the git-client-plugin and the git-plugin that I believe address the issues outlined in this bug report and [PR163|https://github.com/jenkinsci/git-client-plugin/pull/163]. They are https://github.com/jenkinsci/git-plugin/pull/449 and https://github.com/jenkinsci/git-client-plugin/pull/222, Feel free to review and/or test them and comment on the PRs. The git-plugin PR is dependent on the git-client
- PR so make sure to update the pom file in the git-plugin.

roguishmountain@gmail.com (JIRA)

未讀,
2016年12月6日 下午3:19:042016/12/6
收件者:jenkinsc...@googlegroups.com

Hi all,
I created two PRs for the git-client-plugin and the git-plugin that I believe address the issues outlined in this bug report and PR163. They are https://github.com/jenkinsci/git-plugin/pull/449 and https://github.com/jenkinsci/git-client-plugin/pull/222, Feel free to review and/or test them and comment on the PRs. The git-plugin PR is dependent on the git-client-PR so make sure to update the pom file in the git-plugin.

jan@vstone.eu (JIRA)

未讀,
2017年1月4日 凌晨2:31:032017/1/4
收件者:jenkinsc...@googlegroups.com

Adding the extra -f did not resolve the issue of submodules not being there yet in different branches:

git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -ffdx # timeout=10
 > git submodule foreach --recursive git reset --hard # timeout=10
 > git submodule foreach git clean -fdx # timeout=10
Checking out Revision ab59656484b83bd33e8bdcca07116db660a27b61 (origin/feature/mcollective_modules)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ab59656484b83bd33e8bdcca07116db660a27b61
 > git rev-list ab59656484b83bd33e8bdcca07116db660a27b61 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -ffdx # timeout=10
 > git submodule foreach --recursive git reset --hard # timeout=10
 > git submodule foreach git clean -fdx # timeout=10
 > git remote # timeout=10
 > git submodule init # timeout=10
 > git submodule sync # timeout=10
 > git config --get remote.origin.url # timeout=10
 > git config --get-regexp ^submodule # timeout=10
 > git config --get submodule.modules/upstream/accounts.url # timeout=10
...
> git submodule update modules/upstream/locales
FATAL: Command "git submodule update modules/upstream/locales" returned status code 1:
stdout: 
stderr: error: pathspec 'modules/upstream/locales' did not match any file(s) known to git.
Did you forget to 'git add'?

hudson.plugins.git.GitException: Command "git submodule update modules/upstream/locales" returned status code 1:
stdout: 
stderr: error: pathspec 'modules/upstream/locales' did not match any file(s) known to git.
Did you forget to 'git add'?

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1765)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1509)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:67)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1052)
	at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:102)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1149)
	at hudson.scm.SCM.checkout(SCM.java:485)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1738)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)

In this case, the modules/upstream/locales was added in a previous build from a different branch.

my solution was to remove submodule processing and execute git submodule init / sync / update myself.

melezhik@gmail.com (JIRA)

未讀,
2017年1月4日 清晨6:00:012017/1/4
收件者:jenkinsc...@googlegroups.com

HI Mark Waite!

If the "Clean before checkout" and "Clean after checkout" were extended to allow an additional behavior to "clean submodules" ...

Not diving into implementation of PR mentioned too much , but, say I have about ~ 200 sub modules in my superproject and I checked "Clean ..." options , what time does it take me to get sub modules update? My concerns if this has side effect in uploading things from the scratch ? Which in case of many git sub modules usually takes awhile ...

scottwd@gmail.com (JIRA)

未讀,
2017年1月4日 上午11:17:052017/1/4
收件者:jenkinsc...@googlegroups.com

Yes, this issue (or at least the issue Jan Vansteenkiste and I are experiencing) is still not addressed. Unfortunately the title is misleading, as adding extra flags to git commands won't accomplish anything as long as the plugin continues to use git config as a source of submodule information. git config retains submodule information from all previous checked out branches which is not removed by any form of clean command, so when a submodule exists in one branch and not another, it all blows up.

RTFM:
> git config --get-regexp ^submodule # this is broken and will not work properly except on fresh clones. Use 'git submodule status' instead.

scottwd@gmail.com (JIRA)

未讀,
2017年1月4日 上午11:17:052017/1/4
收件者:jenkinsc...@googlegroups.com
Scott Davis edited a comment on Bug JENKINS-26660
Yes, this issue (or at least the issue [~vstone] and I are experiencing) is still not addressed. Unfortunately the title is misleading, as adding extra flags to git commands won't accomplish anything as long as the plugin continues to use git config as a source of submodule information. git config retains submodule information from all previous checked out branches which is not removed by any form of clean command, so when a submodule exists in one branch and not another, it all blows up.

RTFM TLDR Version :

> git config --get-regexp ^submodule # this is broken and will not work properly except on fresh clones. Use 'git submodule status' instead.

melezhik@gmail.com (JIRA)

未讀,
2017年1月4日 上午11:21:102017/1/4
收件者:jenkinsc...@googlegroups.com

Hi Scott!

won't accomplish anything as long as the plugin continues to use git config as a source of submodule information

Exactly. Now I have to ssh to jenkins server and edit git config file manually at workspace to allow plugin to work after I removed git sub module from super repo.

scottwd@gmail.com (JIRA)

未讀,
2017年1月4日 上午11:22:042017/1/4
收件者:jenkinsc...@googlegroups.com
Scott Davis edited a comment on Bug JENKINS-26660
Yes, this issue (or at least the issue [~vstone] and I are experiencing) is still not addressed. Unfortunately the title is misleading, as adding extra flags to git commands won't accomplish anything as long as the plugin continues to use git config as a source of submodule information. git config retains submodule information from all previous checked out previously encountered submodules on all previously encountered branches which that is not removed by any form of clean command , so when . So if a submodule is deleted or exists in one branch and not another, it all blows up and can only be fixed by forcing a fresh clone each build or manually managing submodules inside the job .


TLDR Version:
> git config --get-regexp ^submodule # this is broken and will not work properly except on fresh clones. Use 'git submodule status' instead.

mark.earl.waite@gmail.com (JIRA)

未讀,
2017年1月4日 上午11:24:012017/1/4
收件者:jenkinsc...@googlegroups.com

Scott Davis have you checked your issue is not the same as JENKINS-37495, JENKINS-37419, or JENKINS-38860?

The git client plugin 2.2.0 release from 3 Jan 2017 is believed to resolve those three issues. It resolved the failures in my test environment, and the changes have a number of new automated tests which showed the problem (before the fix) and which now run successfully.

scottwd@gmail.com (JIRA)

未讀,
2017年1月4日 下午2:44:012017/1/4
收件者:jenkinsc...@googlegroups.com

Hey Mark Waite, thanks for the info. I wasn't aware of the latest plugin version and didn't find JENKINS-37419 or JENKINS-38860 when I searched initially. It does sound like the problem has probably been fixed. Thanks for your efforts!

scm_issue_link@java.net (JIRA)

未讀,
2018年2月10日 晚上9:47:032018/2/10
收件者:jenkinsc...@googlegroups.com

Code changed in jenkins
User: Samantha Schwarz
Path:
src/main/java/hudson/plugins/git/GitAPI.java
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
http://jenkins-ci.org/commit/git-client-plugin/58a322534f70180f5cba17e8f09c0e40df7fd962
Log:
Clean takes a flag that will add a --force flag to clean.

See bug report https://issues.jenkins-ci.org/browse/JENKINS-26660 for more details

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

rene.scheibe@gmail.com (JIRA)

未讀,
2019年11月2日 下午2:27:112019/11/2
收件者:jenkinsc...@googlegroups.com
René Scheibe updated an issue
 
Change By: René Scheibe
Component/s: git-plugin
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

rene.scheibe@gmail.com (JIRA)

未讀,
2019年11月2日 下午2:38:032019/11/2
收件者:jenkinsc...@googlegroups.com
René Scheibe commented on Bug JENKINS-26660
 
Re: Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

Current State

rene.scheibe@gmail.com (JIRA)

未讀,
2019年11月2日 下午2:42:032019/11/2
收件者:jenkinsc...@googlegroups.com
René Scheibe edited a comment on Bug JENKINS-26660
Current State
- Step 1 is done. With
[ git-client-plugin 3.0.0 | https://github.com/jenkinsci/git-client-plugin/ releases/tag/git-client-3.0.0] [https://github.com/jenkinsci/git-client-plugin/ pull/222 ] is released now.
- Step 2 is still to be done. Using the new feature from git-client-plugin in git-plugin (
[ https://github.com/jenkinsci/git-plugin/pull/449 ] ).

rene.scheibe@gmail.com (JIRA)

未讀,
2019年12月20日 下午4:53:082019/12/20
收件者:jenkinsc...@googlegroups.com
René Scheibe edited a comment on Bug JENKINS-26660
Current State
- Step 1 is done. With [git-client-plugin 3.0.0|https://github.com/jenkinsci/git-client-plugin/releases/tag/git-client-3.0.0] [https://github.com/jenkinsci/git-client-plugin/pull/222] is released now.
- Step 2 is
still to be done. Using the new feature from git-client-plugin in git-plugin ([https://github.com/jenkinsci/git-plugin/pull/ 449 792 ]).

rene.scheibe@gmail.com (JIRA)

未讀,
2019年12月20日 下午4:54:092019/12/20
收件者:jenkinsc...@googlegroups.com

rene.scheibe@gmail.com (JIRA)

未讀,
2019年12月20日 下午4:54:102019/12/20
收件者:jenkinsc...@googlegroups.com

rene.scheibe@gmail.com (JIRA)

未讀,
2019年12月20日 下午4:54:102019/12/20
收件者:jenkinsc...@googlegroups.com

rene.scheibe@gmail.com (JIRA)

未讀,
2019年12月20日 下午4:55:042019/12/20
收件者:jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

未讀,
2019年12月20日 下午4:56:062019/12/20
收件者:jenkinsc...@googlegroups.com

mark.earl.waite@gmail.com (JIRA)

未讀,
2019年12月20日 下午4:59:042019/12/20
收件者:jenkinsc...@googlegroups.com
Mark Waite commented on Bug JENKINS-26660
 
Re: Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

Built hpi of a pre-release of the plugin can be pulled from the master branch of the ci.jenkins.io job.

Plan is to release git plugin 4.1 before Dec 31, 2019 with this fix included.

mark.earl.waite@gmail.com (JIRA)

未讀,
2020年1月18日 下午1:47:152020/1/18
收件者:jenkinsc...@googlegroups.com
Mark Waite updated Bug JENKINS-26660
 

Fixed in git plugin 4.1.0, released Jan 18, 2020.

Change By: Mark Waite
Status: Fixed but Unreleased Closed
Released As: git-client-plugin 3.0.0, git-plugin TODO 4.1.0

rene.scheibe@gmail.com (JIRA)

未讀,
2020年1月28日 下午2:59:042020/1/28
收件者:jenkinsc...@googlegroups.com
René Scheibe commented on Bug JENKINS-26660
 
Re: Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

NOTE

This change breaks backward compatibility regarding Job DSL for the methods cleanBeforeCheckoutTrait() and cleanAfterCheckoutTrait().

The changes in the UI are backward compatible.

old syntax

multibranchPipelineJob('example-old') {
  branchSources {
    branchSource {
      source {
        git {
          id('example-old')
          remote('https://github.com/jenkinsci/job-dsl-plugin.git')
          traits {
            cleanBeforeCheckoutTrait() // <--- fails with git-plugin v4.1.0
            cleanAfterCheckoutTrait()  // <--- fails with git-plugin v4.1.0
          }
        }
      }
    }
  }
}

new syntax

multibranchPipelineJob('example-new') {
  branchSources { 
    branchSource { 
      source { 
        git { 
          id('example-new')
          remote('https://github.com/jenkinsci/job-dsl-plugin.git')
          traits { 
            cleanBeforeCheckoutTrait {
              extension {
                deleteUntrackedNestedRepositories(false)
              }
            }
            cleanAfterCheckoutTrait {
              extension {
                deleteUntrackedNestedRepositories(false)
              }
            }
          }
        }
      }
    }
  }
}
回覆所有人
回覆作者
轉寄
0 則新訊息