[JIRA] (JENKINS-35982) EC2 Plugin: High CPU in hudson.plugins.ec2.win.winrm.WindowsProcess

10 views
Skip to first unread message

jtsweet@cloudbees.com (JIRA)

unread,
Jun 15, 2016, 4:03:01 PM6/15/16
to jenkinsc...@googlegroups.com
James Sweet created an issue
 
Jenkins / Bug JENKINS-35982
EC2 Plugin: High CPU in hudson.plugins.ec2.win.winrm.WindowsProcess
Issue Type: Bug Bug
Assignee: Francis Upton
Components: ec2-plugin
Created: 2016/Jun/15 8:02 PM
Environment: EC2 Plugin 1.31
Jenkins 1.625.16.1
Labels: plugin
Priority: Major Major
Reporter: James Sweet

A Jenkins master running the EC2 Plugin will exhibit high CPU usage. High CPU Analysis will show threads with a stack trace similar to:

"input copy: java -jar C:\Windows\Temp\slave.jar" #53906 daemon prio=5 os_prio=0 tid=0x00007fab61963800 nid=0x1860 runnable [0x00007fab1288f000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)

  • locked <0x00000006b9e757f8> (a java.io.IOException)
    at java.lang.Throwable.<init>(Throwable.java:265)
    at java.lang.Exception.<init>(Exception.java:66)
    at java.io.IOException.<init>(IOException.java:58)
    at java.io.PipedInputStream.read(PipedInputStream.java:310)
  • locked <0x00000005583223c0> (a java.io.PipedInputStream)
    at java.io.PipedInputStream.read(PipedInputStream.java:377)
  • locked <0x00000005583223c0> (a java.io.PipedInputStream)
    at java.io.InputStream.read(InputStream.java:101)
    at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:124)

Which appears to be related to this code--https://github.com/jenkinsci/ec2-plugin/blob/0dc221934cbd087b2819b5660e9b778208c9f2dc/src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java#L125-L133 in the EC2 plugin.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Jun 15, 2016, 4:13:01 PM6/15/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
Change By: Jesse Glick
Labels: plugin performance windows

jglick@cloudbees.com (JIRA)

unread,
Jun 15, 2016, 4:14:01 PM6/15/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
A Jenkins master running the EC2 Plugin will exhibit high CPU usage.  High CPU Analysis will show threads with a stack trace similar to:
{code:none}
"input copy: java -jar C:\Windows\Temp\slave.jar" #53906 daemon prio=5 os_prio=0 tid=0x00007fab61963800 nid=0x1860 runnable [0x00007fab1288f000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00000006b9e757f8> (a java.io.IOException)

at java.lang.Throwable.<init>(Throwable.java:265)
at java.lang.Exception.<init>(Exception.java:66)
at java.io.IOException.<init>(IOException.java:58)
at java.io.PipedInputStream.read(PipedInputStream.java:310)
- locked <0x00000005583223c0> (a java.io.PipedInputStream)
at java.io.PipedInputStream.read(PipedInputStream.java:377)
- locked <0x00000005583223c0> (a java.io.PipedInputStream)

at java.io.InputStream.read(InputStream.java:101)
at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:124)

mika.jo.karjalainen@gmail.com (JIRA)

unread,
Jan 16, 2017, 7:53:03 AM1/16/17
to jenkinsc...@googlegroups.com
Mika Karjalainen commented on Bug JENKINS-35982
 
Re: EC2 Plugin: High CPU in hudson.plugins.ec2.win.winrm.WindowsProcess

Running jvmtop with --profile option gave me this analysis:

Profiling PID 1524: /usr/share/jenkins/jenkins.war --webroot

  99.22% (    76.60s) hudson.plugins.ec2.win.winrm.WindowsProcess$2.run()
   0.39% (     0.30s) ....sonyericsson.jenkins.plugins.bfa.model.FailureReader()
   0.16% (     0.12s) org.kohsuke.stapler.export.NotExportableException.<init>()
   0.04% (     0.03s) hudson.remoting.ObjectInputStreamEx.resolveClass()
   0.03% (     0.02s) ....amazonaws.http.protocol.SdkHttpRequestExecutor.doRec()
   0.02% (     0.02s) hudson.remoting.ChunkedOutputStream.sendFrame()
   0.02% (     0.02s) hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest()
   0.01% (     0.01s) hudson.util.IOUtils.copy()
   0.01% (     0.01s) hudson.remoting.Command.readFrom()
   0.01% (     0.01s) winstone.BoundedExecutorService$1.run()
   0.01% (     0.01s) hudson.remoting.FlightRecorderInputStream.read()
   0.01% (     0.01s) ....google.common.util.concurrent.ForwardingExecutorServ()
   0.00% (     0.00s) ....thoughtworks.xstream.io.xml.XmlFriendlyNameCoder.dec()
   0.00% (     0.00s) okio.Okio$2.read()
   0.00% (     0.00s) hudson.Util.tryOnceDeleteFile()
   0.00% (     0.00s) ....kohsuke.stapler.Function$InstanceFunction.getParamet()
   0.00% (     0.00s) com.jcraft.jzlib.InfCodes.inflate_fast()
   0.00% (     0.00s) hudson.PluginWrapper.getVersionOf()
   0.00% (     0.00s) com.thoughtworks.xstream.io.path.PathTracker.pushElement()
   0.00% (     0.00s) jenkins.util.VirtualFile$FileVF.isIllegalSymlink()

During the analysis I did not have any ec2 windows agents running so this seems a bit weird.

mika.jo.karjalainen@gmail.com (JIRA)

unread,
Jan 16, 2017, 7:55:05 AM1/16/17
to jenkinsc...@googlegroups.com
Mika Karjalainen edited a comment on Bug JENKINS-35982
Running jvmtop with --profile option gave me this analysis:

{code:none}

Profiling PID 1524: /usr/share/jenkins/jenkins.war --webroot

  99.22% (    76.60s) hudson.plugins.ec2.win.winrm.WindowsProcess$2.run()
   0.39% (     0.30s) ....sonyericsson.jenkins.plugins.bfa.model.FailureReader()
   0.16% (     0.12s) org.kohsuke.stapler.export.NotExportableException.<init>()
   0.04% (     0.03s) hudson.remoting.ObjectInputStreamEx.resolveClass()
   0.03% (     0.02s) ....amazonaws.http.protocol.SdkHttpRequestExecutor.doRec()
   0.02% (     0.02s) hudson.remoting.ChunkedOutputStream.sendFrame()
   0.02% (     0.02s) hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest()
   0.01% (     0.01s) hudson.util.IOUtils.copy()
   0.01% (     0.01s) hudson.remoting.Command.readFrom()
   0.01% (     0.01s) winstone.BoundedExecutorService$1.run()
   0.01% (     0.01s) hudson.remoting.FlightRecorderInputStream.read()
   0.01% (     0.01s) ....google.common.util.concurrent.ForwardingExecutorServ()
   0.00% (     0.00s) ....thoughtworks.xstream.io.xml.XmlFriendlyNameCoder.dec()
   0.00% (     0.00s) okio.Okio$2.read()
   0.00% (     0.00s) hudson.Util.tryOnceDeleteFile()
   0.00% (     0.00s) ....kohsuke.stapler.Function$InstanceFunction.getParamet()
   0.00% (     0.00s) com.jcraft.jzlib.InfCodes.inflate_fast()
   0.00% (     0.00s) hudson.PluginWrapper.getVersionOf()
   0.00% (     0.00s) com.thoughtworks.xstream.io.path.PathTracker.pushElement()
   0.00% (     0.00s) jenkins.util.VirtualFile$FileVF.isIllegalSymlink()
{code}

During the analysis I did not have any ec2 windows agents running so this seems a bit weird. Running Jenkins 2.32.1 and EC2 plugin 1.36.

johannes.ebke@tngtech.com (JIRA)

unread,
Mar 28, 2018, 3:24:02 AM3/28/18
to jenkinsc...@googlegroups.com

I observed this issue after the disconnect of Windows agents.

Looking at the code; it seems obvious that the Jenkins thread is stuck in a busy loop at https://github.com/jenkinsci/ec2-plugin/blob/0dc221934cbd087b2819b5660e9b778208c9f2dc/src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java#L126 since there is no escape for broken pipes. If spurious IOExceptions are really a problem there, as the comment claims, the retries should probably be limited to a sensible number.

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

johannes.ebke@tngtech.com (JIRA)

unread,
Mar 28, 2018, 3:30:02 AM3/28/18
to jenkinsc...@googlegroups.com

Looking at the github forks of this plugin; I found
https://github.com/jenkinsci/ec2-plugin/compare/master...karliszigurs:fix-intermittent-pipe-failure-in-windows
which would probably fix this issue.

johannes.ebke@tngtech.com (JIRA)

unread,
Mar 28, 2018, 4:00:02 AM3/28/18
to jenkinsc...@googlegroups.com
Johannes Ebke edited a comment on Bug JENKINS-35982
Looking at the github forks of this plugin; I found
https://github.com/jenkinsci/ec2-plugin/compare/master...karliszigurs:fix-intermittent-pipe-failure-in-windows
which would probably fix this issue.


Edit: The PR https://github.com/jenkinsci/ec2-plugin/pull/263 containing this change was already merged. Thanks!

mika.jo.karjalainen@gmail.com (JIRA)

unread,
Jul 30, 2018, 8:59:03 AM7/30/18
to jenkinsc...@googlegroups.com

We are now running Jenkins 2.121.2 and ec2-plugin 1.39 and still see this issue. Two days after Jenkins restart CPU load has climbed up to 60% and top 5 threads sorted by cpu time are all 'input copy: java -jar C:\Windows\Temp\slave.jar'
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

sdhoka@umd.edu (JIRA)

unread,
Jun 4, 2019, 4:53:03 PM6/4/19
to jenkinsc...@googlegroups.com
Shubham Dhoka updated an issue
 
Change By: Shubham Dhoka
Labels: ec2-plugin performance windows
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

raihaan.shouhell@autodesk.com (JIRA)

unread,
Aug 1, 2019, 2:20:01 AM8/1/19
to jenkinsc...@googlegroups.com

raihaan.shouhell@autodesk.com (JIRA)

unread,
Aug 1, 2019, 2:21:03 AM8/1/19
to jenkinsc...@googlegroups.com
Raihaan Shouhell assigned an issue to Raihaan Shouhell
 
Change By: Raihaan Shouhell
Assignee: Francis Upton Raihaan Shouhell
Reply all
Reply to author
Forward
0 new messages