[JIRA] (JENKINS-55556) EC2 unable to retrieve private IP (+ other buggy behaviour)

4 views
Skip to first unread message

gunter@grodotzki.co.za (JIRA)

unread,
Jan 13, 2019, 8:28:02 AM1/13/19
to jenkinsc...@googlegroups.com
Gunter Grodotzki created an issue
 
Jenkins / New Feature JENKINS-55556
EC2 unable to retrieve private IP (+ other buggy behaviour)
Issue Type: New Feature New Feature
Assignee: FABRIZIO MANFREDI
Components: ec2-plugin
Created: 2019-01-13 13:27
Environment: docker jenkins/jenkins:lts-slim
ec2-plugin 1.42
Priority: Blocker Blocker
Reporter: Gunter Grodotzki

I must say any version > 1.39 seems ultra buggy for our use case (ec2 spot instances @ c4.xlarge):

 

  • when launching manually the plugin can not pick up the private IP and tries to connect to "null:22" endlessly
  • when launched "on-demand" the plugin will launch 10+ spot instances, non of them will work though

 

Downgrading to 1.39 makes the plugin work on the exact same setup.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

gunter@grodotzki.co.za (JIRA)

unread,
Jan 13, 2019, 8:29:02 AM1/13/19
to jenkinsc...@googlegroups.com
Gunter Grodotzki updated an issue
 
Jenkins / Bug JENKINS-55556
Change By: Gunter Grodotzki
Issue Type: New Feature Bug

gunter@grodotzki.co.za (JIRA)

unread,
Jan 13, 2019, 8:29:04 AM1/13/19
to jenkinsc...@googlegroups.com
Gunter Grodotzki updated an issue
I must say any version > 1.39 seems ultra buggy for our use case (ec2 spot instances @ c4.xlarge):

 
* when launching manually the plugin can not pick up the private IP and tries to connect to "null:22" endlessly
* when launched "on-demand" the plugin will launch 10+ spot instances, non of them will work though

 

Downgrading to 1.39 makes the plugin work on the exact same setup. Curious why AWS is not stepping in to give this plugin some love...

fabrizio.manfredi@gmail.com (JIRA)

unread,
Jan 13, 2019, 5:00:02 PM1/13/19
to jenkinsc...@googlegroups.com
FABRIZIO MANFREDI commented on Bug JENKINS-55556
 
Re: EC2 unable to retrieve private IP (+ other buggy behaviour)

Hi,

For the first problem, Can share more details on your configuration ?, how is your AWS network configuration , node configuration 

In the 1.42 should not have changes on the connection ( in the next version there are some improvements)

For the second problem, can you share any error message ? Did you update the IAM roles ?

gunter@grodotzki.co.za (JIRA)

unread,
Jan 13, 2019, 5:08:02 PM1/13/19
to jenkinsc...@googlegroups.com

When downgrading from 1.42 to 1.39 everything works with the exact same setup (no changes, just downgrading by manually uploading the hpi).

The second problem is a result of the first problem, but for some reason it would spin up 10+ instances almost instantly (could be that new setting "launch new instances right away"?) - the error was the same, that is unable to connect to "null:22"

Setup:

  • VPC with private/public subnets - private subnets outgoing via natgw (https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/1.51.0)
  • jenkins nodes are launched of a custom ami (debian + openjdk8/encrypted boot) with their own custom instance-profile
  • jenkins nodes are launched in private subnet with no public ip - a security group that explicitly only gives the jenkins-master access via port 22

 

Hope this info helps, else let me know  

fabrizio.manfredi@gmail.com (JIRA)

unread,
Jan 14, 2019, 6:08:02 AM1/14/19
to jenkinsc...@googlegroups.com

A couple of questions :

  • did you update the IAM policy connected to the user/role used for raise new node ? (
    ...
    Effect": "Allow",
    "Action": [
    "iam:ListInstanceProfilesForRole",
    "iam:PassRole"
    ],)
  • Is Jenkins master with Public ip ? 
  • Jenkins master and slaves are in the same VPC  and subnet? 
  • What is the configuration of the ec-plugin (use public dns, ...), the option enabled?

 

For the number of the nodes started, now the plugin is much more "reactive" to the status of the queue

 

josh@freelancer.com (JIRA)

unread,
Jan 14, 2019, 7:20:03 AM1/14/19
to jenkinsc...@googlegroups.com

ng this issue as well. I was able to work around it by not using spot instances.

josh@freelancer.com (JIRA)

unread,
Jan 14, 2019, 7:20:03 AM1/14/19
to jenkinsc...@googlegroups.com
Joshua Spence edited a comment on Bug JENKINS-55556
ng Hitting this issue as well. I was able to work around it by not using spot instances.

gunter@grodotzki.co.za (JIRA)

unread,
Jan 14, 2019, 7:54:02 AM1/14/19
to jenkinsc...@googlegroups.com

FABRIZIO MANFREDI

  • yes, those IAM actions are in place
  • jenkins master is with private IP but reachable over a public IP behind an ALB
  • jenkins master and nodes are in the same VPC, private subnets, but might be in different subnet-ids depending on launched AZ
  • public-dns is not enabled

 

Joshua Spence might be correct, we are launching spot ec2 c4.xlarge - so might be an issue with spot instances

shaun.lawrie@ptml.com (JIRA)

unread,
Jan 16, 2019, 5:38:02 PM1/16/19
to jenkinsc...@googlegroups.com

I have the same symptoms with spot instance private IP's not being identified by the ec2-plugin but ours also have public IP's assigned in the configuration.

I downgraded to 1.39 so it behaves in the meantime.

 

 

 

shaun.lawrie@ptml.com (JIRA)

unread,
Jan 16, 2019, 5:38:04 PM1/16/19
to jenkinsc...@googlegroups.com
Shaun Lawrie edited a comment on Bug JENKINS-55556

herophuong93@gmail.com (JIRA)

unread,
Jan 21, 2019, 1:45:02 AM1/21/19
to jenkinsc...@googlegroups.com

We are using spot instances too. Configuring master to connect to slaves using neither public IP or private IP works. Always

 

Jan 21, 2019 6:41:58 AM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: There was a problem while connecting to null:22

 

The plugin is indeed unusable for spot instance use case since 1.40, there was always different error in each version.

 

vdc@zymergen.com (JIRA)

unread,
Feb 4, 2019, 11:29:02 PM2/4/19
to jenkinsc...@googlegroups.com

On Jenkins 2.150.2 EC2 plugin 1.42 we get this behavior intermittently. We oscillate between this error and the error in JENKINS-55639.

With the null:22 error we additionally get the horrible side effect that the script continues to connect to the master node and runs the init script. Just by chance the init script I had in place wasn't destructive to the master. Now I have a little addition to my init script at the top:

master=jenkins-master
host=$(hostname -s)
if [ $host == $master ]
then
 echo "This is the master node! Exiting!!!"
 exit 1
else
 echo "Server is a spot node, apparently. Better yet, it's NOT the master node. Continuing..."
fi

fabrizio.manfredi@gmail.com (JIRA)

unread,
Feb 11, 2019, 11:26:01 AM2/11/19
to jenkinsc...@googlegroups.com

In the 1.43  will be added an option to specify how to connect to slave (private or public)

eric@ericknecht.com (JIRA)

unread,
Mar 19, 2019, 10:29:09 AM3/19/19
to jenkinsc...@googlegroups.com

Is there an estimate of when 1.43 will be released?

alistair.gilbert@basware.com (JIRA)

unread,
Mar 30, 2019, 8:04:02 AM3/30/19
to jenkinsc...@googlegroups.com

we made the changes to the iamrole and were able to spin the instace up fine. but now we are impacted by the second part of the issue where it spins up 10 instances in the backend. any suggestion for how to workaround that problem?

alistair.gilbert@basware.com (JIRA)

unread,
Mar 30, 2019, 8:07:02 PM3/30/19
to jenkinsc...@googlegroups.com
Alistair Gilbert updated an issue
 
Change By: Alistair Gilbert
Comment:
we made the changes to the iamrole and were able to spin the instace up fine. but now we are impacted by the second part of the issue where it spins up 10 instances in the backend. any suggestion for how to workaround that problem?

fabrizio.manfredi@gmail.com (JIRA)

unread,
May 17, 2019, 5:25:04 AM5/17/19
to jenkinsc...@googlegroups.com

fabrizio.manfredi@gmail.com (JIRA)

unread,
Jun 26, 2019, 11:31:05 AM6/26/19
to jenkinsc...@googlegroups.com
FABRIZIO MANFREDI closed an issue as Fixed
 
Change By: FABRIZIO MANFREDI
Status: Open Closed
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages