[JIRA] (JENKINS-57138) Performance issue due to the bundle anonymization

[fbelzunc@gmail.com (JIRA)]

Félix Belzunce Arcos created an issue

Jenkins / JENKINS-57138 Performance issue due to the bundle anonymization Issue Type: Bug Assignee: Emilio Escobar Components: support-core-plugin Created: 2019-04-22 12:22 Priority: Major Reporter: Félix Belzunce Arcos This is a follow-up of JENKINS-54999 In the latest release of the support-core plugin {{support-core:2.56:not-pinned }} we have seen performance issues Without anonymization (I have launch 3 times) ~35seconds, and I see at max ~100% CPU usage With anonymization (I have launch 1 time regarding the time) ~11 minutes, and I see at max ~360% CPU usage The relevant stacktraces are: ThreadDump 1 "Handling POST /XXXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3de0000] java.lang.Thread.State: RUNNABLE at java.util.zip.Deflater.deflateBytes(Native Method) at java.util.zip.Deflater.deflate(Deflater.java:444) - locked <0x0000000719156578> (a java.util.zip.ZStreamRef) at java.util.zip.Deflater.deflate(Deflater.java:366) at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate(StreamCompressor.java:246) at org.apache.commons.compress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded(StreamCompressor.java:241) at org.apache.commons.compress.archivers.zip.StreamCompressor.writeDeflated(StreamCompressor.java:222) at org.apache.commons.compress.archivers.zip.StreamCompressor.write(StreamCompressor.java:190) at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.write(ZipArchiveOutputStream.java:900) at org.apache.commons.compress.archivers.ArchiveOutputStream.write(ArchiveOutputStream.java:109) at java.io.FilterOutputStream.write(FilterOutputStream.java:77) at java.io.FilterOutputStream.write(FilterOutputStream.java:125) at java.io.FilterOutputStream.write(FilterOutputStream.java:97) at org.apache.commons.io.IOUtils.write(IOUtils.java:1527) at org.apache.commons.io.IOUtils.write(IOUtils.java:1552) at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:122) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314) at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205) at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77) at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:130) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:95) at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:70) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:68) at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:43) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) ThreadDump 2 "Handling POST /XXXXXXe/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000] java.lang.Thread.State: RUNNABLE at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30) at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66) at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43) at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:121) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314) at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205) at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77) at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:130) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:95) at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:70) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:68) at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:43) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) ThreadDump 3 "Handling POST /XXXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000] java.lang.Thread.State: RUNNABLE at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30) at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66) at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43) at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:121) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314) at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205) ThreadDump 5 "Handling POST /usinerecette/support/generateAllBundles from 10.1.5.71 : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000] java.lang.Thread.State: RUNNABLE at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30) at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66) at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43) at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:130) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314) at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205) at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[fbelzunc@gmail.com (JIRA)]

Félix Belzunce Arcos updated an issue

Jenkins / JENKINS-57138 Performance issue due to the bundle anonymization

Change By: Félix Belzunce Arcos

This is a follow-up of JENKINS-54999 In the latest release of the support-core plugin {{support-core:2.56:not-pinned }} we have seen performance issues

{code:java}

Without anonymization (I have launch 3 times) ~35seconds, and I see at max ~100% CPU usage With anonymization (I have launch 1 time regarding the time) ~11 minutes, and I see at max ~360% CPU usage

{code} The relevant stacktraces are: h4. ThreadDump 1 {code:java}

{code} h4. ThreadDump 2 {code:java}

{code} h4. ThreadDump 3 {code:java}

"Handling POST /XXXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]    java.lang.Thread.State: RUNNABLE at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30) at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66) at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43) at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:121) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314) at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205)

{code} h4. ThreadDump 5 {code:java} "Handling POST / usinerecette XXX /support/generateAllBundles from 10 X . 1 X . 5 X . 71 X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]

java.lang.Thread.State: RUNNABLE at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30) at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66) at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43) at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:130) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314) at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205) at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)

{code}

Add Comment

[fbelzunc@gmail.com (JIRA)]

h4. ThreadDump 4 {code:java} Handling POST /XXXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio= 5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]    java.lang.Thread.State: RUNNABLE at hudson.ExtensionList.get(ExtensionList.java:149) at com.cloudbees.jenkins.support.filter.ContentMappings.get(ContentMappings.java:68) at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:56) at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43) at com.cloudbees.jenkins.support.filter.FilteredWriter.filterFlushLines(FilteredWriter.java:137) at com.cloudbees.jenkins.support.filter.FilteredWriter.write(FilteredWriter.java:100) - locked <0x0000000744dd6d90> (a com.cloudbees.jenkins.support.filter.FilteredWriter) at java.io.FilterWriter.write(FilterWriter.java:91) at java.io.Writer.write(Writer.java:157) at java.io.PrintWriter.newLine(PrintWriter.java:480) - locked <0x0000000744dd6d78> (a com.cloudbees.jenkins.support.util.IgnoreCloseWriter) at java.io.PrintWriter.println(PrintWriter.java:629) at com.cloudbees.jenkins.support.impl.LoadStats$CsvContent.printTo(LoadStats.java:238) at com.cloudbees.jenkins.support.api.PrintedContent.writeTo(PrintedContent.java:56) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:316)

at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205) at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH)

{code } h4. ThreadDump 5 {code

:java} "Handling POST /XXX/support/generateAllBundles from X.X.X.X : qtp2007328737-1361430" #1361430 prio=5 os_prio=0 tid=0x00007efcedfc0000 nid=0x12c4 runnable [0x00007efcb3ddf000]

java.lang.Thread.State: RUNNABLE at com.cloudbees.jenkins.support.util.WordReplacer.replaceWord(WordReplacer.java:133) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWords(WordReplacer.java:67) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:51) at com.cloudbees.jenkins.support.util.WordReplacer.replaceWordsIgnoreCase(WordReplacer.java:30) at com.cloudbees.jenkins.support.filter.SensitiveContentFilter.filter(SensitiveContentFilter.java:66) at com.cloudbees.jenkins.support.filter.AllContentFilters.filter(AllContentFilters.java:43)

at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:130) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:314) at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:205) at java.lang.invoke.LambdaForm$DMH/695376641.invokeVirtual_L3_V(LambdaForm$DMH) at java.lang.invoke.LambdaForm$BMH/1360969330.reinvoke(LambdaForm$BMH) at java.lang.invoke.LambdaForm$MH/318927722.invoker(LambdaForm$MH) at java.lang.invoke.LambdaForm$MH/1073698081.invokeExact_MT(LambdaForm$MH) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) {code}

Add Comment

[bmathus+ossjira@cloudbees.com (JIRA)]

Baptiste Mathus commented on JENKINS-57138

Re: Performance issue due to the bundle anonymization Hola Félix Belzunce Arcos! Can you please provide the version of the support-core plugin you were using for these tests? And can you retry on the latest version? Many improvements were done in last versions around performance, and hence I think taking a new test is important so we focus on what is still remaining, if anything. Gracias!

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[aheritier@apache.org (JIRA)]

Arnaud Héritier commented on JENKINS-57138

Re: Performance issue due to the bundle anonymization

FYI Allan BURDAJEWICZ

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon commented on JENKINS-57138

Re: Performance issue due to the bundle anonymization

Sorry Félix Belzunce Arcos there is no enough information here to investigate if there is some issue with the anonymization. It's a heavy operation where all the content written to the zip is parsed to replace sensitive information, so it's usual to see such numbers of CPU usage. Without any further data, I recommend: Remove all unneeded log files from the instance to avoid them to be parsed and included in the bundle Update to the latest version of support-core because we have done some improvements regarding the performance Please update the ticket with more information to be able to reproduce this behavior (configuration, number of users, jobs, labels, folders, ... in the instance) or any interesting information we can further analyze. I will keep it open for a while to allow you to provide the requested information or just close it if you agree. Thanks.

Add Comment