Latest Jenkins EC2 plugin version is unable to launch agents few hours after installing

[Abhijeet Ghosh]

Hi there,

I am observing an issue related to Jenkins ec2 plugin's latest version, specifically in all version post 1881.vc14132ea_f042. Please see the details below:

Problem:

In the latest Jenkins EC2 plugin versions that utilizes AWS SDK v2, agent provisioning fails after approximately 12 hours of continuous Jenkins master uptime when running inside a Kubernetes pod. Initially, agent launches function correctly post-deployment, but after around 12 hours, no new agents can be launched until the Jenkins pod is restarted.

This behaviour is not observed in earlier EC2 plugin versions that use AWS SDK v1, indicating a possible cause may be related to the migration to AWS SDK v2.

Impact:

• Agent provisioning stops working after about 12 hours without visible errors in the Jenkins UI.
• Restarting the Jenkins pod temporarily resolves the issue by resetting connections.

Request:

• Please check if this is something that I am only observing and suggest a possible resolution.
• In case if others are also facing this issue, is there a resolution that is planned?

Environment Details:

• Jenkins EC2 plugin version: 1935.v9b_77210472b_1
• Jenkins version: 2.504.1 running in a k8s pod

Thanks & Regards

Abhijeet Ghosh

[Abhijeet Ghosh]

It finally fails with an error like below after ~12 hours:

Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 16 failure: Request has expired. (Service: Ec2, Status Code: 400, Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxx)

[Abhijeet Ghosh]

For anyone facing this issue. Had proposed a fix and now this is merged. See the link below:
https://plugins.jenkins.io/ec2/releases/#version_1985.v20a_803fd248b_