[JIRA] (JENKINS-59216) LinkageError in ec2 spot fleet while performing jenkins UserRequest

14 views
Skip to first unread message

hsutzuchin.register@gmail.com (JIRA)

unread,
Sep 4, 2019, 12:43:03 AM9/4/19
to jenkinsc...@googlegroups.com
TzuChin Hsu created an issue
 
Jenkins / Bug JENKINS-59216
LinkageError in ec2 spot fleet while performing jenkins UserRequest
Issue Type: Bug Bug
Assignee: Chad Schmutzer
Attachments: ec2_1_success.png, ec2_2_fail.png, ec2_3_fail.png, ec2_fail_build.png, ErrorInScaledOutInstance.pdf, Jenkinsfile, Jenkinsfile.txt, remoting_log.txt
Components: ec2-fleet-plugin, git-client-plugin, junit-plugin
Created: 2019-09-04 04:42
Environment: jenkins 2.176.3 with all plugins in latest version
java version (both jenkins master and spot fleet) 1.8.0
-- plugins --
ec2-fleet-plugin 1.10.2
git-client 2.8.2
junit 1.28
--------------

Jenkins master on ec2 r5.large, Amazon linux
Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
Labels: plugin jenkins pipeline slave newbie-friendly
Priority: Major Major
Reporter: TzuChin Hsu

Hi everyone, apologize first for any information leak or mis-understanding, this is my first time to report issue in community. And I have toke merely all way I can to deal with it.

[Scenario]

I use jenkins with spot-fleet plugin for CI

CI all pass in spot-fleet with initial ec2 mechine

While ec2 scale out, CI fail in new instance with "remote call on  i-xxxxxx failed"

Below is the log from instance of spot-fleet (retrieved from ${jenkinsHome/remoting/logs})

remoting_log.txt 

 

[Reproduce]

3 CI job with identical codebase run at the same time
a ec2-spot-fleet with init 2 instance

In my case, 1 CI job take 2 ec2 instance, so one job will take the initial two instance, and the other jobs have to run in scaled out instances.

  • Result : CI pass in one job and fail in other two

[More information]

1. Below is the jenkinsfile with sensitive information replaced

Jenkinsfile

2. I face the problem in two scenario, because docker build and testing in different spot fleet

scenario A:  

building spot-fleet rotate ec2 every day, new ec2 instance fail to build 

scenario B:

is the reproduce experiment, happen in spot-fleet used in testing stage
new ec2 (scaled out) fail in CI testing while initial instances success
you can see pipeline with blueOcean in ec2_N_success/fail.png attachment

 

3. Slave log viewed in jenkins
ErrorInScaledOutInstance.pdf

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

hsutzuchin.register@gmail.com (JIRA)

unread,
Sep 4, 2019, 12:44:01 AM9/4/19
to jenkinsc...@googlegroups.com
TzuChin Hsu updated an issue
Change By: TzuChin Hsu
Hi everyone, apologize first for any information leak or mis-understanding, this is my first time to report issue in community. And I have toke merely all way I can to deal with it.

 

[ Environment]


jenkins 2.176.3 with all plugins in latest version
java version (both jenkins master and spot fleet) 1.8.0
-- plugins --
ec2-fleet-plugin 1.10.2
git-client 2.8.2
junit 1.28
--------------

Jenkins master on ec2 r5.large, Amazon linux
Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux

 

[ Scenario]


I use jenkins with spot-fleet plugin for CI

CI all pass in spot-fleet with initial ec2 mechine

While ec2 scale out, CI fail in new instance with "*remote call on  i-xxxxxx failed*"


Below is the log from instance of spot-fleet (retrieved from ${jenkinsHome/remoting/logs})

[^remoting_log.txt] 


 

[Reproduce]

3 CI job with identical codebase run at the same time
a ec2-spot-fleet with init 2 instance

In my case, 1 CI job take 2 ec2 instance, so one job will take the initial two instance, and the other jobs have to run in scaled out instances.
* Result : CI pass in one job and fail in other two

!ec2_1_success.png!


[More information]

1. Below is the jenkinsfile with sensitive information replaced

[^Jenkinsfile]


2. I face the problem in two scenario, because docker build and testing in different spot fleet

scenario A:  

building spot-fleet rotate ec2 every day, new ec2 instance fail to build 

!ec2_fail_build.png!


scenario B:

is the reproduce experiment, happen in spot-fleet used in testing stage
new ec2 (scaled out) fail in CI testing while initial instances success
you can see pipeline with blueOcean in ec2_N_success/fail.png attachment

 

3. Slave log viewed in jenkins
[^ErrorInScaledOutInstance.pdf]

hsutzuchin.register@gmail.com (JIRA)

unread,
Sep 4, 2019, 12:49:02 AM9/4/19
to jenkinsc...@googlegroups.com
TzuChin Hsu updated an issue
Change By: TzuChin Hsu
Environment: jenkins 2.176.3 with all plugins in latest version \n

java version (both jenkins master and spot fleet) 1.8.0

-- plugins --
ec2-fleet-plugin 1.10.2
git-client 2.8.2
junit 1.28
--------------

Jenkins master on ec2 r5.large, Amazon linux
Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux

hsutzuchin.register@gmail.com (JIRA)

unread,
Sep 4, 2019, 12:49:02 AM9/4/19
to jenkinsc...@googlegroups.com
TzuChin Hsu updated an issue
Change By: TzuChin Hsu
Environment:
jenkins 2.176.3 with all plugins in latest version

java version (both jenkins master and spot fleet) 1.8.0
-- plugins --
ec2-fleet-plugin 1.10.2
git-client 2.8.2
junit 1.28
--------------

Jenkins master on ec2 r5.large, Amazon linux
Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux

hsutzuchin.register@gmail.com (JIRA)

unread,
Sep 4, 2019, 12:49:02 AM9/4/19
to jenkinsc...@googlegroups.com
TzuChin Hsu updated an issue
Change By: TzuChin Hsu
Environment: jenkins 2.176.3 with all plugins in latest version \n

java version (both jenkins master and spot fleet) 1.8.0
-- plugins --
ec2-fleet-plugin 1.10.2
git-client 2.8.2
junit 1.28
--------------

Jenkins master on ec2 r5.large, Amazon linux
Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux

mark.earl.waite@gmail.com (JIRA)

unread,
Sep 7, 2019, 11:22:03 PM9/7/19
to jenkinsc...@googlegroups.com

hsutzuchin.register@gmail.com (JIRA)

unread,
Sep 15, 2019, 11:01:02 PM9/15/19
to jenkinsc...@googlegroups.com
TzuChin Hsu commented on Bug JENKINS-59216
 
Re: LinkageError in ec2 spot fleet while performing jenkins UserRequest

Hi, this problem have been "SOLVED"
That's because "yum update" when jenkins slave have already run in machine,
   and some java lib might be affected, so cause "LinakageError", after update AMI this problem has not be seen anymore.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

artem.stasuk@gmail.com (JIRA)

unread,
Jan 24, 2020, 11:31:03 AM1/24/20
to jenkinsc...@googlegroups.com
Artem Stasiuk resolved as Not A Defect
 

thx for comment, closing

Change By: Artem Stasiuk
Status: Open Resolved
Assignee: Chad Schmutzer Artem Stasiuk
Resolution: Not A Defect

artem.stasuk@gmail.com (JIRA)

unread,
Jan 24, 2020, 11:31:03 AM1/24/20
to jenkinsc...@googlegroups.com
Artem Stasiuk closed an issue as Not A Defect
Change By: Artem Stasiuk
Status: Resolved Closed
Reply all
Reply to author
Forward
0 new messages