*Description*
When git checkout crash (for example, due to a timeout) before the Jenkinsfile execution, the internal multibranch workspace is broken. Indeed, git add .lock files in its repository when it runs. If git crash, these files are not removed. When you try to restart the build, it will fails because git finds a .lock file and think another git operation is running.
*Workaround*
You have to manually remove the workspace of your branch in the Jenkins Home (or at least the .lock files)
*Logs*
Here is the git crash caused by a timeout (probably a problem with my internet connection).
{noformat}
Started by user anonymous
Setting origin to https://my.server.com/my-repo.git
Fetching origin...
> git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git.exe config remote.origin.url https://my.server.com/my-repo.git # timeout=10
Cleaning workspace
> git.exe rev-parse --verify HEAD # timeout=10
Resetting working tree
> git.exe reset --hard # timeout=10
> git.exe clean -fdx # timeout=10
Fetching upstream changes from https://my.server.com/my-repo.git
> git.exe --version # timeout=10
using GIT_ASKPASS to set credentials vsts
> git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=120
ERROR: Timeout after 120 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://my.server.com/my-repo.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:799)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1055)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:85)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1" returned status code -1:
stdout:
stderr: Fatal: InvalidOperationException encountered.
remote: Microsoft (R) Visual Studio (R) Team Services
remote:
remote: Found 459 objects to send.
remote: Found 5449 objects to send. (14062 ms)
Receiving objects: 0% (1/5449)
Receiving objects: 1% (55/5449)
Receiving objects: 2% (109/5449)
Receiving objects: 3% (164/5449)
Receiving objects: 4% (218/5449)
Receiving objects: 5% (273/5449)
Receiving objects: 6% (327/5449)
Receiving objects: 7% (382/5449)
Receiving objects: 8% (436/5449)
Receiving objects: 9% (491/5449)
Receiving objects: 10% (545/5449)
Receiving objects: 11% (600/5449)
Receiving objects: 12% (654/5449)
Receiving objects: 13% (709/5449)
Receiving objects: 14% (763/5449)
Receiving objects: 15% (818/5449)
Receiving objects: 16% (872/5449)
Receiving objects: 17% (927/5449)
Receiving objects: 18% (981/5449)
Receiving objects: 19% (1036/5449)
Receiving objects: 20% (1090/5449)
Receiving objects: 21% (1145/5449)
Receiving objects: 22% (1199/5449)
Receiving objects: 23% (1254/5449)
Receiving objects: 24% (1308/5449)
Receiving objects: 25% (1363/5449)
Receiving objects: 26% (1417/5449)
Receiving objects: 27% (1472/5449)
Receiving objects: 28% (1526/5449)
Receiving objects: 29% (1581/5449)
Receiving objects: 30% (1635/5449)
Receiving objects: 30% (1687/5449), 1.65 MiB | 1.93 MiB/s
Receiving objects: 31% (1690/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 32% (1744/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 33% (1799/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 34% (1853/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 35% (1908/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 36% (1962/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 37% (2017/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 38% (2071/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 39% (2126/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 40% (2180/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 41% (2235/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 42% (2289/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 43% (2344/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 44% (2398/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 45% (2453/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 46% (2507/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 47% (2562/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 48% (2616/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 49% (2671/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 50% (2725/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 51% (2779/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 52% (2834/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 53% (2888/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 54% (2943/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 55% (2997/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 56% (3052/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 57% (3106/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 58% (3161/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 59% (3215/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 60% (3270/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 61% (3324/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 62% (3379/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 63% (3433/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 64% (3488/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 65% (3542/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 66% (3597/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 67% (3651/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 68% (3706/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 69% (3760/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 70% (3815/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 71% (3869/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 72% (3924/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 72% (3962/5449), 2.66 MiB | 1.81 MiB/s
Receiving objects: 73% (3978/5449), 4.53 MiB | 1.99 MiB/s
Receiving objects: 74% (4033/5449), 6.83 MiB | 2.46 MiB/s
Receiving objects: 74% (4051/5449), 6.83 MiB | 2.46 MiB/s
Receiving objects: 74% (4074/5449), 8.88 MiB | 2.05 MiB/s
Receiving objects: 74% (4074/5449), 10.19 MiB | 1.90 MiB/s
Receiving objects: 74% (4074/5449), 11.63 MiB | 1.81 MiB/s
Receiving objects: 74% (4074/5449), 13.25 MiB | 1.37 MiB/s
Receiving objects: 74% (4075/5449), 14.44 MiB | 1.40 MiB/s
Receiving objects: 74% (4082/5449), 18.06 MiB | 1.57 MiB/s
Receiving objects: 74% (4082/5449), 19.06 MiB | 1.51 MiB/s
Receiving objects: 74% (4083/5449), 19.81 MiB | 1.50 MiB/s
Receiving objects: 75% (4087/5449), 19.96 MiB | 1.31 MiB/s
Receiving objects: 75% (4112/5449), 19.96 MiB | 1.31 MiB/s
Receiving objects: 76% (4142/5449), 21.62 MiB | 1.46 MiB/s
Receiving objects: 76% (4159/5449), 22.50 MiB | 1.46 MiB/s
Receiving objects: 77% (4196/5449), 22.95 MiB | 1.32 MiB/s
Receiving objects: 78% (4251/5449), 22.95 MiB | 1.32 MiB/s
Receiving objects: 79% (4305/5449), 22.95 MiB | 1.32 MiB/s
Receiving objects: 80% (4360/5449), 22.95 MiB | 1.32 MiB/s
Receiving objects: 81% (4414/5449), 22.95 MiB | 1.32 MiB/s
Receiving objects: 82% (4469/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 83% (4523/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 84% (4578/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 85% (4632/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 86% (4687/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 87% (4741/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 88% (4796/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 88% (4807/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 89% (4850/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 90% (4905/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 91% (4959/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 92% (5014/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 93% (5068/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 94% (5123/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 95% (5177/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 96% (5232/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 97% (5286/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 98% (5341/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 99% (5395/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 100% (5449/5449), 24.34 MiB | 1.69 MiB/s
Receiving objects: 100% (5449/5449), 24.68 MiB | 1.32 MiB/s, done.
Resolving deltas: 0% (0/4170)
Resolving deltas: 1% (44/4170)
Resolving deltas: 2% (84/4170)
Resolving deltas: 3% (141/4170)
Resolving deltas: 4% (167/4170)
Resolving deltas: 5% (210/4170)
Resolving deltas: 6% (254/4170)
Resolving deltas: 7% (309/4170)
Resolving deltas: 9% (397/4170)
Resolving deltas: 10% (427/4170)
Resolving deltas: 11% (463/4170)
Resolving deltas: 12% (511/4170)
Resolving deltas: 13% (543/4170)
Resolving deltas: 14% (590/4170)
Resolving deltas: 15% (631/4170)
Resolving deltas: 16% (669/4170)
Resolving deltas: 17% (709/4170)
Resolving deltas: 18% (752/4170)
Resolving deltas: 19% (797/4170)
Resolving deltas: 20% (835/4170)
Resolving deltas: 21% (883/4170)
Resolving deltas: 22% (927/4170)
Resolving deltas: 23% (962/4170)
Resolving deltas: 24% (1001/4170)
Resolving deltas: 25% (1043/4170)
Resolving deltas: 26% (1095/4170)
Resolving deltas: 27% (1153/4170)
Resolving deltas: 28% (1168/4170)
Resolving deltas: 29% (1225/4170)
Resolving deltas: 30% (1251/4170)
Resolving deltas: 31% (1295/4170)
Resolving deltas: 32% (1346/4170)
Resolving deltas: 33% (1397/4170)
Resolving deltas: 34% (1422/4170)
Resolving deltas: 35% (1479/4170)
Resolving deltas: 36% (1502/4170)
Resolving deltas: 37% (1544/4170)
Resolving deltas: 38% (1587/4170)
Resolving deltas: 39% (1631/4170)
Resolving deltas: 40% (1681/4170)
Resolving deltas: 41% (1710/4170)
Resolving deltas: 41% (1712/4170)
Resolving deltas: 41% (1714/4170)
Resolving deltas: 42% (1752/4170)
Resolving deltas: 43% (1795/4170)
Resolving deltas: 44% (1835/4170)
Resolving deltas: 45% (1885/4170)
Resolving deltas: 46% (1921/4170)
Resolving deltas: 47% (1967/4170)
Resolving deltas: 48% (2003/4170)
Resolving deltas: 49% (2053/4170)
Resolving deltas: 50% (2087/4170)
Resolving deltas: 51% (2137/4170)
Resolving deltas: 52% (2169/4170)
Resolving deltas: 53% (2211/4170)
Resolving deltas: 54% (2252/4170)
Resolving deltas: 55% (2298/4170)
Resolving deltas: 56% (2345/4170)
Resolving deltas: 57% (2396/4170)
Resolving deltas: 58% (2430/4170)
Resolving deltas: 59% (2462/4170)
Resolving deltas: 60% (2502/4170)
Resolving deltas: 61% (2548/4170)
Resolving deltas: 62% (2594/4170)
Resolving deltas: 63% (2635/4170)
Resolving deltas: 64% (2677/4170)
Resolving deltas: 65% (2711/4170)
Resolving deltas: 66% (2753/4170)
Resolving deltas: 67% (2802/4170)
Resolving deltas: 68% (2836/4170)
Resolving deltas: 69% (2878/4170)
Resolving deltas: 69% (2911/4170)
Resolving deltas: 70% (2919/4170)
Resolving deltas: 71% (2963/4170)
Resolving deltas: 72% (3003/4170)
Resolving deltas: 73% (3049/4170)
Resolving deltas: 74% (3086/4170)
Resolving deltas: 75% (3130/4170)
Resolving deltas: 76% (3170/4170)
Resolving deltas: 77% (3214/4170)
Resolving deltas: 78% (3260/4170)
Resolving deltas: 79% (3296/4170)
Resolving deltas: 80% (3337/4170)
Resolving deltas: 81% (3382/4170)
Resolving deltas: 82% (3422/4170)
Resolving deltas: 83% (3464/4170)
Resolving deltas: 84% (3504/4170)
Resolving deltas: 85% (3545/4170)
Resolving deltas: 86% (3588/4170)
Resolving deltas: 87% (3629/4170)
Resolving deltas: 88% (3670/4170)
Resolving deltas: 89% (3712/4170)
Resolving deltas: 90% (3762/4170)
Resolving deltas: 91% (3795/4170)
Resolving deltas: 92% (3849/4170)
Resolving deltas: 93% (3881/4170)
Resolving deltas: 94% (3928/4170)
Resolving deltas: 95% (3962/4170)
Resolving deltas: 96% (4005/4170)
Resolving deltas: 97% (4045/4170)
Resolving deltas: 98% (4087/4170)
Resolving deltas: 99% (4129/4170)
Resolving deltas: 100% (4170/4170)
Resolving deltas: 100% (4170/4170), completed with 951 local objects.
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1739)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1482)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797)
... 8 more
ERROR: null
Finished: FAILURE
{noformat}
And after that crash, I've the following error each time I want to start a new build:
{noformat}
Started by user anonymous
Setting origin to https://my.server.com/my-repo.git
Fetching origin...
> git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git.exe config remote.origin.url https://my.server.com/my-repo.git # timeout=10
Cleaning workspace
> git.exe rev-parse --verify HEAD # timeout=10
Resetting working tree
> git.exe reset --hard # timeout=10
> git.exe clean -fdx # timeout=10
Fetching upstream changes from https://my.server.com/my-repo.git
> git.exe --version # timeout=10
using GIT_ASKPASS to set credentials vsts qdu pat
> git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=120
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://my.server.com/my-repo.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:799)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1055)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:85)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1" returned status code 128:
stdout:
stderr: Fatal: InvalidOperationException encountered.
fatal: Unable to create 'M:/workspace/b/f2a6f488@script/.git/shallow.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1739)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1482)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797)
... 8 more
ERROR: null
Finished: FAILURE
{noformat}
*How to reproduce*
# Create a git repository with a huge file and a Jenkinsfile
# Create a multibranch pipeline linked to the previous repository
# Set the git checkout and clone timeout to the smallest value you can to provoke a git timeout
# Try to restart the build, you'll have the error with the .lock file
*Ideas to fix it*
I don't know if it's hard to fix. Some idea would be to scan the git repository to search for .lock files dans remove it ? Or doing the equivalent of this Jenkinsfile groovy script:
{code:java}
try { checkout scm } catch (caughtError) { deleteDir(); checkout scm }
{code}
_Note 1: Even if it's a problem linked to git, I thought that you'll want to handle this case in the workflow multibranch pipeline. I may be wrong._ |
|
|
