[JIRA] (JENKINS-59215) gerrit trigger stuck and can not trigger jobs

gerrit trigger stuck and can not trigger jobs

Issue Type:	Task
Assignee:	rsandell
Attachments:	20190904_103746.png
Components:	gerrit-trigger-plugin
Created:	2019-09-04 02:38
Environment:	jenkins: 2.176.3 gerrit trigger: 2.30
Labels:	gerrit-trigger
Priority:	Major
Reporter:	BO WANG

After upgrade gerrit trigger to 2.30, sometimes gerrit trigger can not trigger jenkins jobs, but the jenkins account can receive the gerrit stream events at the same time. We find a lot of WARNING logs in jenkins log,

The Gerrit incoming events queue contains 6389 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thead-pool size: 10.

By the way, we set gerrit-trigger only listening to change-merged and comment-added event types.

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

wangbo.hfut@outlook.com (JIRA)

unread,

Sep 3, 2019, 10:39:03 PM9/3/19

to jenkinsc...@googlegroups.com

BO WANG updated an issue

Jenkins /

gerrit trigger stuck and can not trigger jobs

Change By:	BO WANG
Issue Type:	Task Bug

wangbo.hfut@outlook.com (JIRA)

unread,

Sep 3, 2019, 10:41:02 PM9/3/19

to jenkinsc...@googlegroups.com

BO WANG updated an issue

Jenkins /

gerrit trigger stuck and can not trigger jobs

Change By:	BO WANG

After upgrade gerrit trigger to 2.30, sometimes gerrit trigger can not trigger jenkins jobs, but the jenkins account can receive the gerrit stream events at the same time. We find a lot of WARNING logs in jenkins log,

{color:#de350b}_The Gerrit incoming events queue contains 6389 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thead-pool size: 10._{color}

{color:#172b4d}By the way, we set gerrit-trigger only listening to change-merged and comment-added event types. Even we restart jenkins server, gerrit-trigger seems still stuck. {color}

elliliu1984@gmail.com (JIRA)

unread,

Sep 4, 2019, 4:15:04 AM9/4/19

to jenkinsc...@googlegroups.com

Li Liu commented on

Exactly the same problem has found on Jenkins 2.176.2 and gerrit trigger plugin 2.30. Downgraded to 2.27.7 helped.

jenkins@thorzten.de (JIRA)

unread,

Sep 12, 2019, 4:04:03 AM9/12/19

to jenkinsc...@googlegroups.com

Thorsten Möllers commented on

I found the same problem and found a way to reproduce. For me it appears when I have a

disabled job at jenkins start time
configured with dynamic trigger

If the job is enabled and jenkins start time, I don't see the problem, even if I disable it later. If the job is disabled at start time, the problem occurs and still stays even if I enable the job later. If I remove the dynamic trigger configuration, the problem disappears.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

jenkins@thorzten.de (JIRA)

unread,

Sep 12, 2019, 4:05:03 AM9/12/19

to jenkinsc...@googlegroups.com

Thorsten Möllers edited a comment on

I found the same problem and found a way to reproduce. For me it appears when I have a

* disabled job at jenkins start time
* configured with dynamic trigger

If the job is enabled and jenkins start time, I don't see the problem, even if I disable it later. If the job is disabled at start time, the problem occurs and still stays even if I enable the job later. If I remove the dynamic trigger configuration, the problem disappears.

I can reproduce the problem with 2.29.0 and 2.30.0, didn't see it with 2.27.X

sascht@sas.com (JIRA)

unread,

Sep 17, 2019, 9:54:02 AM9/17/19

to jenkinsc...@googlegroups.com

Charles Tudor commented on

Ran into the same problem on 2.29 and 2.30. Went back to 2.28 and things are working as expected.

georg.bremer@here.com (JIRA)

unread,

Oct 22, 2019, 6:47:03 AM10/22/19

to jenkinsc...@googlegroups.com

Georg Bremer commented on

At least the issue described by having a dynamic trigger job disabled on startup is caused by https://github.com/jenkinsci/gerrit-trigger-plugin/pull/350

The executors are waiting for the project to be ready, i.e. the dynamic trigger to be loaded. However, the the GerritTriggerTimerTask does a couple of checks (including job.isBuildable()) before updateTriggerConfigURL. This means the latch introduced in that pull request is never released and the executors are blocked forever.

georg.bremer@here.com (JIRA)

unread,

Oct 22, 2019, 7:49:05 AM10/22/19

to jenkinsc...@googlegroups.com

Georg Bremer edited a comment on