[JIRA] (JENKINS-58382) Build is stuck due to blocked thread

[arielman22@gmail.com (JIRA)]

Ariel M created an issue

Jenkins / JENKINS-58382 Build is stuck due to blocked thread Issue Type: Bug Assignee: Gabe Montero Components: openshift-sync-plugin Created: 2019-07-08 14:46 Priority: Major Reporter: Ariel M There is a build which is stuck on intial stage. I saw in monitoring screen the below message: Threads on 50aa85...@172.17.0.4: Number = 110, Maximum = 526, Total started = 464,563 Warning, the following threads are deadlocked : Handling POST /job/oc-cd-drop22/job/oc-cd-drop22-oc-cd-ocp/214/kill from 10.75.36.119 : qtp544724190-466789, jenkins.util.Timer 9, OkHttp https://ilocpde07-master.corp.amdocs.com:8443/..., OkHttp https://ilocpde07-master.corp.amdocs.com:8443/...   I tried to kill that thread but it's not getting deleted. Then I went to the threadDump log and saw: jenkins.util.Timer 9 "jenkins.util.Timer 9" Id=71 Group=main BLOCKED on java.lang.String@39c5d884 owned by "OkHttp https://ilocpde07-master.corp.amdocs.com:8443/..." Id=139290 at io.fabric8.jenkins.openshiftsync.JenkinsUtils.triggerJob(JenkinsUtils.java:422) - blocked on java.lang.String@39c5d884 at io.fabric8.jenkins.openshiftsync.BuildWatcher.addEventToJenkinsJobRun(BuildWatcher.java:345) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.handleBuildList(JenkinsUtils.java:815) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.maybeScheduleNext(JenkinsUtils.java:717) at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.onDeleted(BuildSyncRunListener.java:184) at hudson.model.listeners.RunListener.fireDeleted(RunListener.java:269) at hudson.model.Run.delete(Run.java:1573) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.deleteRun(JenkinsUtils.java:567) - locked java.lang.Class@b802391 at io.fabric8.jenkins.openshiftsync.BuildWatcher.reconcileRunsAndBuilds(BuildWatcher.java:494) at io.fabric8.jenkins.openshiftsync.BuildWatcher.access$100(BuildWatcher.java:65) at io.fabric8.jenkins.openshiftsync.BuildWatcher$1.doRun(BuildWatcher.java:148) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@28c61284     so I guess it's something with the openshift sync plugin. can you suggest what is the next step I should try to do? Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[gmontero@redhat.com (JIRA)]

Gabe Montero commented on JENKINS-58382

Re: Build is stuck due to blocked thread hey Ariel M - without seeing the full thread dump, it is hard to say.  Possibly cancelling all openshift pipeline strategy builds running on jenkins might help.  Otherwise, restarting Jenkins is the only option.   I think I have enough from the single thread stack trace to devise a fix, but again, if by some chance you have the full thread dump still, please attach as a text file (posting them as Jira comments are hard to read).   thanks

Add Comment

[arielman22@gmail.com (JIRA)]

Ariel M updated an issue

Jenkins / JENKINS-58382

Build is stuck due to blocked thread

Change By: Ariel M Attachment: Thread Dump.docx

Add Comment

[arielman22@gmail.com (JIRA)]

Ariel M commented on JENKINS-58382

Re: Build is stuck due to blocked thread attached

Add Comment

[gmontero@redhat.com (JIRA)]

Gabe Montero started work on JENKINS-58382

Change By: Gabe Montero Status: Open In Progress

Add Comment

[gmontero@redhat.com (JIRA)]

Gabe Montero commented on JENKINS-58382

Re: Build is stuck due to blocked thread

PR https://github.com/openshift/jenkins-sync-plugin/pull/317 is up for addressing the deadlock   After it merges, I'll cherry pick it into https://github.com/jenkinsci/openshift-sync-plugin and cut a new version of the plugin

Add Comment

[arielman22@gmail.com (JIRA)]

Ariel M commented on JENKINS-58382

Re: Build is stuck due to blocked thread

Will it solve also the below issue I'm facing: Hitting build with parameters not triggering the build, only rebuild option works for me

Add Comment

[gmontero@redhat.com (JIRA)]

Gabe Montero commented on JENKINS-58382

Re: Build is stuck due to blocked thread

I think so yes

Add Comment

[gmontero@redhat.com (JIRA)]

Gabe Montero commented on JENKINS-58382

Re: Build is stuck due to blocked thread

so v1.0.39 of the sync plugin has been released and it has the fix   thanks

Add Comment

[gmontero@redhat.com (JIRA)]

Gabe Montero resolved as Fixed

Jenkins / JENKINS-58382

Build is stuck due to blocked thread

Change By: Gabe Montero Status: In Progress Resolved Resolution: Fixed Released As: v1.0.39

Add Comment

[arielman22@gmail.com (JIRA)]

Ariel M commented on JENKINS-58382

Re: Build is stuck due to blocked thread Gabe, why do you think it will fix the build with parameters issue? what is the connection between two things?

Add Comment

[gmontero@redhat.com (JIRA)]

Gabe Montero commented on JENKINS-58382

Re: Build is stuck due to blocked thread

because the parameter propagation goes through some of the same callback channels that your original deadlock went through Ariel   That said, without debug data, I'm making an educated guess.  But if you want to pursue debugging, open a separate issue and provide another thread dump and the jenkins logs / job logs when the failure occurs.

Add Comment