Delay agent execution for auto-updates at boot

[James Green]

I finally used Packer to build an AMI based on Amazon Linux 2 to act as as agent. Yesterday, the builds began failing with a Java stack trace.

Turns out the EC2 instance starts, and in parallel the Jenkins agent begins and an auto-upgrade of openjdk kicked off. Further instructions failed. Rebuild the AMI and the new agent worked a treat.

Is there a way of avoiding this problem by preventing Jenkins from launching the agent until the boot steps are complete? I cannot be the first to encounter this issue :-)

Thanks,

James

[James Green]

So our builds failed again yesterday due to this automatic update kicking in. I checked, there was indeed a JDK security update so no complaints, but I need a better system of updating Jenkins.

Given Packer can search for and filter down AMIs, I thought the EC2 plugin should be able to, too. Solves our issue. Ticket if anyone is interested: https://issues.jenkins-ci.org/browse/JENKINS-59180

[Toni Van de Voorde]

Hey James,

I had the same issue without understanding what happened. Meanwhile I also understand that it's due to the automatic updates kicking in. I solved it by removing the JDK from that AMI and let the master install it. 

You could also update your AMI and not allow auto upgrades, or you could add in the "init-script" of the cloud configuration in jenkins master to perform an upgrade. From what I see in the code (https://github.com/jenkinsci/ec2-plugin/blob/master/src/main/java/hudson/plugins/ec2/ssh/EC2UnixLauncher.java#L184) it should wait for the upgrade to be finished.

Hope this helps

Cheers

On Tuesday, September 3, 2019 at 10:26:38 AM UTC+2, James Green wrote:

So our builds failed again yesterday due to this automatic update kicking in. I checked, there was indeed a JDK security update so no complaints, but I need a better system of updating Jenkins.

Given Packer can search for and filter down AMIs, I thought the EC2 plugin should be able to, too. Solves our issue. Ticket if anyone is interested: https://issues.jenkins-ci.org/browse/JENKINS-59180

[James Green]

Hi Toni,

Looks like we are not alone: https://issues.jenkins-ci.org/browse/JENKINS-43771 (see linked issue which is indeed the same).

I ended up adding a vanilla Amazon Linux 2 AMI as an agent yesterday to try things. It failed because git wasn't installed. I added a user data script to install this but then java wasn't installed. So I added this too and the agent log stalled entirely. Not sure what I did wrong and have not had a chance to investigate further. Your comments well well be relevant - the documentation inline did not make clear the difference between user data and init script when used on an Amazon Linux 2 AMI (there is cloud-init).

If I get further I will update this thread.

James

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/edc2e575-ffa3-4865-9e78-9a0466497e67%40googlegroups.com.