[JIRA] (JENKINS-60407) Launched instances cannot reach public internet (regression)

0 views
Skip to first unread message

james.mk.green@gmail.com (JIRA)

unread,
Dec 9, 2019, 12:04:03 PM12/9/19
to jenkinsc...@googlegroups.com
James Green created an issue
 
Jenkins / Bug JENKINS-60407
Launched instances cannot reach public internet (regression)
Issue Type: Bug Bug
Assignee: FABRIZIO MANFREDI
Components: ec2-plugin
Created: 2019-12-09 17:03
Environment: Jenkins ver. 2.190.3
Priority: Major Major
Reporter: James Green

With plugin 1.45 all is working.

Upgraded to 1.46.1 and although the agents report themselves as available, they have no network access to the outside world. Reverting to 1.45 got them back again.

The agents are each configured with the same single private subnet to launch into. Investigations showed things like `ping -c 3 google.com` times out.

Checking out from bitbucket.org was the original fault reported. We have apparently made no changes to the Jenkins master, except to upgrade it to the current LTS from a recent release at the same time as updating the plugins this lunchtime.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

domi@fortysix.ch (JIRA)

unread,
Jan 10, 2020, 9:57:02 AM1/10/20
to jenkinsc...@googlegroups.com
Dominik Bartholdi commented on Bug JENKINS-60407
 
Re: Launched instances cannot reach public internet (regression)

we have the exact same issue, I think this is caused by JENKINS-58578 

I did some manual testing and setting a public IP on the agent solves the issue - this IP is set automatically on an agent launched with ec2-plugin:1.45

-> we had to go back to 1.45 too

domi@fortysix.ch (JIRA)

unread,
Jan 10, 2020, 11:31:02 AM1/10/20
to jenkinsc...@googlegroups.com

I was able to fix the issue by adding this to the JasC Config of the ec2 plugin:
 associatePublicIp: true
{{}}

domi@fortysix.ch (JIRA)

unread,
Jan 10, 2020, 11:32:03 AM1/10/20
to jenkinsc...@googlegroups.com
Dominik Bartholdi edited a comment on Bug JENKINS-60407
I was able to fix the issue by adding this to the JasC Config of the ec2 plugin:
  {code:java}
 associatePublicIp: true
{ { code } }
 

james.mk.green@gmail.com (JIRA)

unread,
Jan 16, 2020, 4:32:08 AM1/16/20
to jenkinsc...@googlegroups.com

Dominik Bartholdi where is the JasC Config file? I'm able to see plugins/ec2 but not this file.

A new version of this plugin has been released to fix a security vulnerability. Need to check if this issue remains.

domi@fortysix.ch (JIRA)

unread,
Jan 17, 2020, 3:32:08 AM1/17/20
to jenkinsc...@googlegroups.com
Dominik Bartholdi updated an issue
 
Change By: Dominik Bartholdi
Attachment: image-2020-01-17-09-31-03-387.png

domi@fortysix.ch (JIRA)

unread,
Jan 17, 2020, 3:32:10 AM1/17/20
to jenkinsc...@googlegroups.com
Dominik Bartholdi commented on Bug JENKINS-60407
 
Re: Launched instances cannot reach public internet (regression)

JCasC is an additional plugin to treat jenkins configuration as code: https://plugins.jenkins.io/configuration-as-code

But you will find the option in the UI too:

http://myjenkins/configure > cloud > Amazon EC2 > AMIs > Advanced > 'Associate Public IP'

justin.pihony@gmail.com (JIRA)

unread,
Jan 18, 2020, 1:45:03 PM1/18/20
to jenkinsc...@googlegroups.com

It would be helpful to document Dominik's solution - it was hard to find otherwise. Adding to the docs would have saved me hours.

raihaan.shouhell@autodesk.com (JIRA)

unread,
Jan 19, 2020, 10:17:02 PM1/19/20
to jenkinsc...@googlegroups.com

I don't see why it should not be able to reach public internet do you have the appropriate routes set up?

james.mk.green@gmail.com (JIRA)

unread,
Jan 20, 2020, 12:46:02 PM1/20/20
to jenkinsc...@googlegroups.com

Raihaan Shouhell simply upgrading a plugin should not break a behaviour. Downgrading of it proves the culprit. If the upgrade brings changes they should be documented.

raihaan.shouhell@autodesk.com (JIRA)

unread,
Jan 20, 2020, 10:06:05 PM1/20/20
to jenkinsc...@googlegroups.com

James Green I'd like to understand why it broke. Sure it broke your workflow but not being able to reach public internet and having a public IP are 2 distinct things. I would like to understand why you lose public internet access without a public ip. I can't understand why this change caused this issue.

domi@fortysix.ch (JIRA)

unread,
Jan 21, 2020, 3:31:20 AM1/21/20
to jenkinsc...@googlegroups.com

Raihaan Shouhell I had the exact same issue - lucky me, I was able to compare two EC2 instances (one launched with the old  EC2-plugin version and one with the new one). The only difference I found was the assignement of a public IP. As soon as I assigned a public IP to the instance launched by the new version, it all worked again.

raihaan.shouhell@autodesk.com (JIRA)

unread,
Jan 21, 2020, 3:38:06 AM1/21/20
to jenkinsc...@googlegroups.com

Dominik Bartholdi Could I have some details on your setup?

Do you launch in a vpc? If so is your subnet public or private? What is in that subnets route table?

domi@fortysix.ch (JIRA)

unread,
Jan 22, 2020, 3:15:03 AM1/22/20
to jenkinsc...@googlegroups.com

Raihaan Shouhell sure, I do my best to get you the required details: All I do is done with cloudformation, so it should be reproducible.

Everytime we install a new version, i create it from scratch: I remove everything and build it up from ground with cloudformation only (no manual steps and no cloudformation updates).

You can find a stripped down version of the cloudformation templates here: https://gist.github.com/imod/fb702d545dbe77292e8f4796c7804059 

The templates should contain all the details you need. The 'vpc-cloudformation-template,json' creates the full VPC with route tables, subnet and gateway and can be executed as is, but I had to remove quite a bit from the 'jenkins-cloudformation-template,json' - this one would install Jenkins on a EC2 instance and configure the security groups. 

I hope this is useful, if you don't get a long with cloudformation, please let me know.

 

james.mk.green@gmail.com (JIRA)

unread,
Jan 22, 2020, 5:03:03 AM1/22/20
to jenkinsc...@googlegroups.com

FWIW our working Jenkins installation (with ec2-plugin:1.45) has instances configured to launch within the same VPC as Jenkins itself, and the "Associate Public IP address" checkbox is not checked. Yet ec2 instances do have public IPs - we just never noticed.

I am guessing that updated plugin versions now require this checkbox to be checked.

domi@fortysix.ch (JIRA)

unread,
Jan 22, 2020, 6:09:03 AM1/22/20
to jenkinsc...@googlegroups.com

James Green that sounds like the exact same case then

james.mk.green@gmail.com (JIRA)

unread,
Jan 22, 2020, 8:19:02 AM1/22/20
to jenkinsc...@googlegroups.com

I have checked the option "Associate Public IP" for each agent and relaunched with ec2-plugin:1.49 - brand new agents are working.
Reply all
Reply to author
Forward
0 new messages