[JIRA] [core] (JENKINS-25704) Scheduled triggering stops working until restart

[jwstric2@gmail.com (JIRA)]

Jonathan Strickland commented on JENKINS-25704

Re: Scheduled triggering stops working until restart Finding this issue with 1.621 ... https://issues.jenkins-ci.org/browse/JENKINS-30558. When viewing some of our git polling jobs we see that they are no longer showing as polling anymore. We poll every 5 minutes for these jobs and found they were 8+ hours from last poll time. Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[jwstric2@gmail.com (JIRA)]

Jonathan Strickland commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

Before I restart it; I have tried to turn on fine level for the hudson.trigger.Trigger and hudson.trigger.Trigger$Cron logs; nothing coming in. I plan to leave this logger on after reload.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

Check the /threadDump URL for a "Jenkins cron thread" – what is it doing when the triggers stop working?

Add Comment

[rasenlatscher@gmail.com (JIRA)]

Tom Lachner commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

I have the same issue and according to attached thread-dump1.txt it looks like the following: "Jenkins cron thread" prio=5 TIMED_WAITING java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) For me it looked like this when the issue appears: "Jenkins cron thread" Id=19 Group=main WAITING on java.util.TaskQueue@73054d75 at java.lang.Object.wait(Native Method) - waiting on java.util.TaskQueue@73054d75 at java.lang.Object.wait(Object.java:503) at java.util.TimerThread.mainLoop(Timer.java:526) at java.util.TimerThread.run(Timer.java:505) Could be the same issue as in JENKINS-30558 because we have also the stash plugin installed and I can not remember same kind of issues before stash plugin.

Add Comment

[abunetta@gmail.com (JIRA)]

David Resnick commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

I didn't encounter this problem when I had the stash plugin installed. I now have the stash plugin and haven't encountered this problem.

Add Comment

[jwstric2@gmail.com (JIRA)]

Jonathan Strickland edited a comment on JENKINS-25704

Re: Scheduled triggering stops working until restart

[~timguy] See in your thread dump if you see that of a thread dump at JENKINS-30588 java.net.SocketInputStream.socketRead0 .... followed in stack is the stashpullrequest timer Its misleading but yes the cron thread  is  may be waiting on completion of the timer task of the stash pull request, which will never complete due to a hung socket.  [~david_resnick], we actually didn't see the issue until multiple teams in our group starting having multiple repos each with pull requests (thus multiple jobs with more load on our production stash instance)

Add Comment

[jwstric2@gmail.com (JIRA)]

Jonathan Strickland commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

Tom Lachner See in your thread dump if you see that of a thread dump at JENKINS-30588

java.net.SocketInputStream.socketRead0 .... followed in stack is the stashpullrequest timer

Its misleading but yes the cron thread is may be waiting on completion of the timer task of the stash pull request, which will never complete due to a hung socket. David Resnick, we actually didn't see the issue until multiple teams in our group starting having multiple repos each with pull requests (thus multiple jobs with more load on our production stash instance)

Add Comment

[rasenlatscher@gmail.com (JIRA)]

Tom Lachner updated an issue

Jenkins / JENKINS-25704

Scheduled triggering stops working until restart

Change By: Tom Lachner Attachment: jenkins_log_threaddump-tom.txt

Add Comment

[rasenlatscher@gmail.com (JIRA)]

Tom Lachner commented on JENKINS-25704

Re: Scheduled triggering stops working until restart Jonathan Strickland Well, I see the socketRead0 and something with stash in the thread dump. I attached my thread dump jenkins_log_threaddump-tom.txt and it looks quite similar like in ticket JENKINS-30558. Thanks

Add Comment

[dostal.it@gmail.com (JIRA)]

Tomas D updated an issue

Jenkins / JENKINS-25704

Scheduled triggering stops working until restart

Change By: Tomas D Attachment: threadDump.txt

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[dostal.it@gmail.com (JIRA)]

Tomas D commented on JENKINS-25704

Re: Scheduled triggering stops working until restart We just found the same problem. We have pretty much nothing running on our new Jenkins server, created new pipeline. Manually it runs fine, but it failed to run last night. Schedule is set to H 2 * * * I am attaching threadDump threadDump.txt

Add Comment

[roshan786.chandekar@gmail.com (JIRA)]

roshan chandekar commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

We too are facing the same issue after upgrading the Jenkins to 2.73 version. Suddenly poll scm and timer builds stop working. Jenkins restart fixed the issue but based  on other jenkins forum looks this issue will come again. Please let us know if permanent solution for this issue.

Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[harishkumartr22@gmail.com (JIRA)]

HARISHKUMAR TR commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

David Resnick Can it be possible to assign this to any of known developers??

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[abunetta@gmail.com (JIRA)]

David Resnick commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

HARISHKUMAR TR I think someone has to be convinced to take it.

Add Comment

[chandan.gupta@gmail.com (JIRA)]

Chandan Kumar commented on JENKINS-25704

Re: Scheduled triggering stops working until restart

This is happening to my Jenkins jobs as well. I wanted to run a job at exactly 8:25 am every day with this cron expression: "25 8 * * *". This was working fine for months until now. Changing the exact minute to 'H' seems to help sometimes. e.g. "H 8 * * *" triggers the job, but not at the exact time I need it to trigger. That's a workaround that might work for me, but it's disappointing to not be able to debug this. Cron log is not much help: cron checking hudson.model.FreeStyleProject@505fc44a[Cleanup_Builds/my-job] with spec ‘25 8 * * *’ Dec 17, 2019 8:25:13 AM FINER hudson.triggers.Trigger did not trigger hudson.model.FreeStyleProject@505fc44a[Cleanup_Builds/my-job] Dec 17, 2019 8:25:13 AM FINE hudson.triggers.Trigger

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)