[JIRA] (JENKINS-37121) WorkspaceListLeasePickle should help diagnose locked workspaces

29 views
Skip to first unread message

jglick@cloudbees.com (JIRA)

unread,
Aug 2, 2016, 11:21:01 AM8/2/16
to jenkinsc...@googlegroups.com
Jesse Glick created an issue
 
Jenkins / Bug JENKINS-37121
WorkspaceListLeasePickle should help diagnose locked workspaces
Issue Type: Bug Bug
Assignee: Jesse Glick
Components: workflow-plugin
Created: 2016/Aug/02 3:20 PM
Environment: 1.625.x / 1.14.2
Labels: diagnostics api robustness
Priority: Minor Minor
Reporter: Jesse Glick 
"Waiting to acquire /.../workspace/... : jenkins.util.Timer [#...]" id=... (0x...) state=WAITING cpu=75%
    - waiting on <0x...> (a hudson.slaves.WorkspaceList)
    - locked <0x...> (a hudson.slaves.WorkspaceList)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:255)
    at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:234)
    at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:223)
    at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:67)
    at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:52)
    at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:62)
    at ...

Looks like a workspace did not get relocked fast enough to avoid getting grabbed by some other job?

At a minimum, WorkspaceListLeasePickle should printWaitingMessage when acquire blocks, so it is clearer from the build log why the build is still stuck.

Possibly it should fail if it cannot acquire the workspace immediately, since in this case the workspace can be assumed to have already been clobbered by something else. Currently there is no such core API.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Aug 2, 2016, 6:49:01 PM8/2/16
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-37121
 
Re: WorkspaceListLeasePickle should help diagnose locked workspaces

JENKINS-22767 is one conceivable cause of the situation reported here: one copy of a build locked the workspace upon resumption, and soon after another copy was somehow produced and tried to run, but got stuck attempting to relock the same workspace.

jglick@cloudbees.com (JIRA)

unread,
Aug 2, 2016, 6:51:02 PM8/2/16
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-37121
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Aug 3, 2016, 10:56:02 AM8/3/16
to jenkinsc...@googlegroups.com
Jesse Glick resolved as Fixed
 
Change By: Jesse Glick
Status: In Progress Resolved
Resolution: Fixed

scm_issue_link@java.net (JIRA)

unread,
Aug 3, 2016, 12:40:11 PM8/3/16
to jenkinsc...@googlegroups.com
SCM/JIRA link daemon commented on Bug JENKINS-37121
 
Re: WorkspaceListLeasePickle should help diagnose locked workspaces

Code changed in jenkins
User: Jesse Glick
Path:
support/src/main/java/org/jenkinsci/plugins/workflow/support/pickles/WorkspaceListLeasePickle.java
http://jenkins-ci.org/commit/pipeline-plugin/bed2b2e58b181f7b67df14229410199f307eedce
Log:
JENKINS-37121 Use WorkspaceList.record, not .acquire, when dehydrating a pickle
Backports https://github.com/jenkinsci/workflow-durable-task-step-plugin/pull/14.

me@basilcrow.com (JIRA)

unread,
Mar 15, 2019, 1:56:03 PM3/15/19
to jenkinsc...@googlegroups.com

Got this error a few times today after a Jenkins restart:

Resuming build at Thu Mar 14 20:01:13 PDT 2019 after Jenkins restart
Waiting to resume part of devops-gate » master » git-blackbox
[Pipeline] cleanWs (hide)
20:01:14 [WS-CLEANUP] Deleting project workspace...
20:01:14 [WS-CLEANUP] Deferred wipeout is used...
20:01:14 [WS-CLEANUP] done
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline
[Pipeline] }
java.lang.IllegalStateException: JENKINS-37121: something already locked /var/tmp/jenkins_slaves/jenkins-selfservice/workspace/devops-gate/master/git-blackbox
	at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:75)
	at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:51)
	at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92)
	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
Caused: java.io.IOException: Failed to load build state
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:854)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:852)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:906)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

swapnil_patne@bmc.com (JIRA)

unread,
Jun 28, 2019, 5:29:02 AM6/28/19
to jenkinsc...@googlegroups.com

Getting this error today and strangely it's not even listing this build in left pane in history

 
                                                            

                                                        
 
java.lang.IllegalStateException: JENKINS-37121: something already locked /var/lib/jenkins/workspace/AutomationPipeline@12

 at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:75)
 at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:51)
 at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92)
 at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 Caused: java.io.IOException: Failed to load build state
 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:842)
 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:840)
 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:894)

 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
 Finished: FAILURE{code}
 		 
                                                
 
                                            
                                        
 		 
                                
swapnil_patne@bmc.com (JIRA)
unread,
Jun 28, 2019, 5:30:03 AM6/28/19
to jenkinsc...@googlegroups.com


     
         
         
         
         
     
    
 
        
 
             
            
                
 
                     
                        
 
                            
                                
 
                                       
                                     Swapnil Patne edited a comment on Bug JENKINS-37121  
                                
 
                            
                        
 		 
                
 
                
 
                     
                        
 
                            
 Getting this error today and strangely it's not even listing this build in left pane in history
 
 
{code :java } 
swapnil_patne@bmc.com (JIRA)
unread,
Jun 28, 2019, 5:33:03 AM6/28/19
to jenkinsc...@googlegroups.com

                
 
                     
                        
 
                            
                                
 
                                       
                                     Swapnil Patne edited a comment on Bug JENKINS-37121  
                                
 
                            
                        
 		 
                
 
                
 
                                     
                                        
 
                                            
                                                
 
                                                     Getting this error today and strangely it's not even listing this build in left pane in history . 
 
Jenkins ver. 2.107.2
Reply all
Reply to author
Forward
0 new messages