Job unexpectedly being stopped with success exit code after exactly one hour

40 views
Skip to first unread message

mi...@ammd.com.au

unread,
Dec 5, 2016, 10:46:59 PM12/5/16
to Jenkins Users
I have a long running Jenkins job that can run for more than one hour. 

Via shell scripts, it kicks off an activity in AWS (runs a docker container as a task), then monitors the state of the task every 30 seconds and ends when the task finishes.

As part of the process, it finishes by logging the exit code of the container to the Jenkins console log.

For runs that take less than 1 hour, the output is as expected.

For runs that take more than one hour, the task in AWS continues to run. However the Jenkins log, at the 60 minute mark, abruptly stops and displays a SUCCESS status.

I am thus trying to track down what might be the cause of this. 

I am running Jenkins 2.33 via tomcat7 on a 64 bit AWS Linux server based on the 2016.09 AMI.

I suspect it may be the build_timeout plugin which is installed but not enabled for this job. I have version 1.18 installed.

I don't see anything of relevance in either the tomcat log or the Jenkins system Log at the time.

Any pointers as to what to look for next and where would be appreciated.


Arvind Jayaprakash

unread,
Dec 12, 2016, 11:27:41 PM12/12/16
to Jenkins Users
This might be unrelated, but I ran into the magic 60 minute barrier on AWS when using an ELB. Connections over ELB that are idle for 60 minutes get terminated. I'd check for similar settings on say your tomcat config, the connection to your slave etc. etc.

mi...@ammd.com.au

unread,
Dec 14, 2016, 3:31:54 PM12/14/16
to Jenkins Users
Thanks for your feedback.

The actual AWS activity is running as a task in ECS and continues after the Jenkins job ends. It is not configured with any connectivity via an ELB.

 
Reply all
Reply to author
Forward
0 new messages