[JIRA] (JENKINS-55038) `Parameterized Remote Trigger Plugin` sometimes fails with poll interval value more than 5 minutes

8 views
Skip to first unread message

lifemanship.k@gmail.com (JIRA)

unread,
Dec 5, 2018, 11:39:02 AM12/5/18
to jenkinsc...@googlegroups.com
Nick Korsakov created an issue
 
Jenkins / Bug JENKINS-55038
`Parameterized Remote Trigger Plugin` sometimes fails with poll interval value more than 5 minutes
Issue Type: Bug Bug
Assignee: KaiHsiang Chang
Components: parameterized-remote-trigger-plugin
Created: 2018-12-05 16:38
Environment: Jenkins version - 2.89.4 (Ubuntu 16.04.5 LTS x64, openjdk version "1.8.0_151")
Parameterized Remote Trigger Plugin version - 3.0.5
Priority: Major Major
Reporter: Nick Korsakov

 

There is a problem when we try to use poll interval parameter's value more than 300 seconds(5 minutes). We have a Jenkins pipeline which may take from 30 to 60 minutes.

In some cases Jenkins `queued` item may move to the `pending` state as it described here:
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L139

In such case the plugin use user specified poll interval time out value to check queued item state:

https://github.com/jenkinsci/parameterized-remote-trigger-plugin/blob/master/src/main/java/org/jenkinsci/plugins/ParameterizedRemoteTrigger/RemoteBuildConfiguration.java#L696

But all `queued` items in the Jenkins have time to live only 5 minutes as you can see it here:
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L218

As result we have triggered `build` on the remote Jenkins server but failed build(`Max number of connection retries have been exeeded` error from the plugin) on the main Jenkins server: 

Triggering parameterized remote job 'http://192.168.200.8:6680/job/RunTests'
 Using job-level defined 'Credentials Authentication' as user '***' (Credentials ID '***')
Triggering remote job now.
CSRF protection is enabled on the remote server.
The remote job is pending. Waiting for next available executor on master.
 Remote job queue number: 124200
Waiting for remote build to be executed...
Waiting for 300 seconds until next poll.
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters: 
Retry attempt #1 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters: 
Retry attempt #2 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters: 
Retry attempt #3 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters: 
Retry attempt #4 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters: 
Retry attempt #5 out of 5
Max number of connection retries have been exeeded.

 

From my point of view there is a simple fix: use default value of the poll interval(10 seconds) to check `queued` item state. Because in the `pending` state the `queued` item is staying only for a few seconds: 

 

int pollIntervalForQueuedItem = this.pollInterval;
if (pollIntervalForQueuedItem > DEFAULT_POLLINTERVALL) {
 pollIntervalForQueuedItem = DEFAULT_POLLINTERVALL;
}
while (buildInfo.isQueued()) {
 context.logger.println("Waiting for " + pollIntervalForQueuedItem + " seconds until next poll.");
 Thread.sleep(pollIntervalForQueuedItem * 1000);
 buildInfo = updateBuildInfo(buildInfo, context);
 handle.setBuildInfo(buildInfo);
}

 

 

 

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

lifemanship.k@gmail.com (JIRA)

unread,
Dec 6, 2018, 2:40:07 AM12/6/18
to jenkinsc...@googlegroups.com
Nick Korsakov updated an issue
Change By: Nick Korsakov
 

There is a problem when we try to use poll interval parameter's value more than 300 seconds(5 minutes). We have a Jenkins pipeline which may take from 30 to 60 minutes.

In some cases Jenkins `queued` item may move to the `pending` state as it described here:
[https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L139]

In such case the plugin use user specified poll interval time out value to check queued item state:

[https://github.com/jenkinsci/parameterized-remote-trigger-plugin/blob/master/src/main/java/org/jenkinsci/plugins/ParameterizedRemoteTrigger/RemoteBuildConfiguration.java#L696]

But all `queued` items in the Jenkins have time to live only 5 minutes as you can see it here:
[https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L218]

As result we have triggered `build` on the remote Jenkins server but failed build(`Max number of connection retries have been exeeded` error from the plugin) on the main Jenkins server: 
{code:java}

Triggering parameterized remote job 'http://192.168.200.8:6680/job/RunTests'
Using job-level defined 'Credentials Authentication' as user '***' (Credentials ID '***')
Triggering remote job now.
CSRF protection is enabled on the remote server.
The remote job is pending. Waiting for next available executor on master.
Remote job queue number: 124200
Waiting for remote build to be executed...
Waiting for 300 seconds until next poll.
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters:
Retry attempt #1 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters:
Retry attempt #2 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters:
Retry attempt #3 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters:
Retry attempt #4 out of 5
Connection to remote server failed [404], waiting for to retry - 300 seconds until next attempt. URL: http://192.168.200.8:6680//queue/item/124200/api/json/, parameters:
Retry attempt #5 out of 5
Max number of connection retries have been exeeded.{code}

 

From my point of view there is a simple fix: use default value of the poll interval(10 seconds) to check `queued` item state. Because in the `pending` state the `queued` item is staying only for a few seconds: 

 
{code:java}

int pollIntervalForQueuedItem = this.pollInterval;
if (pollIntervalForQueuedItem > DEFAULT_POLLINTERVALL) {
pollIntervalForQueuedItem = DEFAULT_POLLINTERVALL;
}
while (buildInfo.isQueued()) {
context.logger.println("Waiting for " + pollIntervalForQueuedItem + " seconds until next poll.");
Thread.sleep(pollIntervalForQueuedItem * 1000);
buildInfo = updateBuildInfo(buildInfo, context);
handle.setBuildInfo(buildInfo);

cash924n@gmail.com (JIRA)

unread,
Dec 6, 2018, 3:04:05 AM12/6/18
to jenkinsc...@googlegroups.com

cash924n@gmail.com (JIRA)

unread,
Dec 8, 2018, 11:15:02 AM12/8/18
to jenkinsc...@googlegroups.com

lifemanship.k@gmail.com (JIRA)

unread,
Dec 8, 2018, 4:23:02 PM12/8/18
to jenkinsc...@googlegroups.com

cash924n@gmail.com (JIRA)

unread,
Dec 9, 2018, 12:33:02 AM12/9/18
to jenkinsc...@googlegroups.com

lou.fiorino@csaa.com (JIRA)

unread,
Jun 14, 2019, 1:01:05 PM6/14/19
to jenkinsc...@googlegroups.com
Lou Fiorino commented on Bug JENKINS-55038
 
Re: `Parameterized Remote Trigger Plugin` sometimes fails with poll interval value more than 5 minutes

There's an additional (and apparently SLIPPERY) problem with checking the status of the submitted job... I believe in acquiring the remote context.
We've got multipledynamic instances sitting behind a load balancer (HAPROXY). 

For the sake of this discussion let's use the following configuration:
haproxy.mydomain.com with listeners on ports 9001 ->myjenkins1.mycompany.com:8443, 9002 ->myjenkins2.mycompany.com:8443, 9003 ->myjenkins3.mycompany.com:8443

 

The SSL certificates installed on the 3 back end servers are the same cert for the proxy (haproxy)... as the back end servers are rebuilt on demand when there's a revision to the LTS release or significant plugin updates.

The instance on myjenkins1 attempts to trigger a job on myjenkins2 (i.e. https://haproxy.mydomain.com:*9002*/remote_job_to_be_triggered) via the parameterized remote trigger plugin in a pipleine job.  The job SUCCESSFULLY triggers, however when the pipeline attempts to check the status of the SUCCESSFULLY triggered job, instead of querying https://haproxy.mydomain.com:*9002*/job/remote_job_to_be_triggered/\{build#}/api/json/?seed={seed#}, when it constructs the URL to query, it seems to be pulling the port number from the remote context(evidenced by the presence of "GOT CONTEXT for Buildand Deploy" in the logs on myjenkisn1)  and instead fails to query https://haproxy.mydomain.com:*8443*/job/remote_job_to_be_triggered/\{build#}/api/json/?seed={seed#}.

The error will either be an HTTP 404 (not found)... or if there DOES happen to be a listener available on port 8443 on haproxy for a DIFFERENT jenkins instance but the requesting user does not have login access, an HTTP 401 (unauthorized).

This may be an odd configuration, but I have seen other users complaining of similar problems (i.e. using a VIEW based URL, web server front end, proxy, etc.)... and the heart of the problem here is utilization of an inconsistent base URL between the triggering request and polling for the job status.  While the complexity of having the remote context reported back consistently with "odd" configurations, a simple solution would be to simply pull the base URL(protocol/host/port) from the trigger request instead of the remote context.

I don't have the bandwidth at the moment to do a deep dive into the code, but I cannot imagine the would be a difficult fix... in the meantime we have resorted to performing remote triggers via groovy scripting over the SSH listener and polling for job status inline. (silly workaround for what should be a quick fix).  Hope this provides a little better insight as to the root cause

cash924n@gmail.com (JIRA)

unread,
Jun 14, 2019, 11:51:03 PM6/14/19
to jenkinsc...@googlegroups.com

Thanks for you report, but It would be nice if you can provide me a docker-compose environment cuz I'm not able to reproduce the issue without all you configurations. 

With the testable environment, maybe I can handle it when I'm free.

lifemanship.k@gmail.com (JIRA)

unread,
Mar 18, 2020, 5:18:05 AM3/18/20
to jenkinsc...@googlegroups.com
Nick Korsakov reopened an issue
 

Described problem still reproduced, because plugin does not take into account state of the remote build, i.e. all `queued` items should be processed in time less than 5 minutes.

KaiHsiang Chang, please look at my pull request:

https://github.com/jenkinsci/parameterized-remote-trigger-plugin/pull/65

Change By: Nick Korsakov
Resolution: Fixed
Status: Resolved Reopened
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

lifemanship.k@gmail.com (JIRA)

unread,
Mar 18, 2020, 10:11:02 AM3/18/20
to jenkinsc...@googlegroups.com
Nick Korsakov edited a comment on Bug JENKINS-55038
 
Re: `Parameterized Remote Trigger Plugin` sometimes fails with poll interval value more than 5 minutes
Described problem is still reproduced reproducing , because plugin does not take into account state of the remote build, i.e. all `queued` items should be processed in time less than 5 minutes.

[~cashlalala], please look at my pull request:

[ https://github.com/jenkinsci/parameterized-remote-trigger-plugin/pull/65 ]

lifemanship.k@gmail.com (JIRA)

unread,
Mar 30, 2020, 12:19:04 PM3/30/20
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages