[JIRA] (JENKINS-61078) Groovy based parser thread stuck, causing high CPU load on master

6 views
Skip to first unread message

daniel@bachran.de (JIRA)

unread,
Feb 13, 2020, 8:12:04 AM2/13/20
to jenkinsc...@googlegroups.com
Daniel Bachran created an issue
 
Jenkins / Bug JENKINS-61078
Groovy based parser thread stuck, causing high CPU load on master
Issue Type: Bug Bug
Assignee: Ulli Hafner
Components: warnings-ng-plugin
Created: 2020-02-13 13:11
Environment: Jenkins 2.204.2
warnings-ng plugin 7.3.0

_________________________________________
System Properties

awt.toolkit sun.awt.windows.WToolkit
executable-war D:\Jenkins\jenkins.war
file.encoding Cp1252
file.encoding.pkg sun.io
file.separator \
hudson.lifecycle hudson.lifecycle.WindowsServiceLifecycle
hudson.model.DirectoryBrowserSupport.CSP default-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'
hudson.tasks.MailSender.SEND_TO_USERS_WITHOUT_READ true
java.awt.graphicsenv sun.awt.Win32GraphicsEnvironment
java.awt.headless true
java.awt.printerjob sun.awt.windows.WPrinterJob
java.class.path D:\Jenkins\jenkins.war
java.class.version 52.0
java.endorsed.dirs D:\Jenkins\jre\lib\endorsed
java.ext.dirs D:\Jenkins\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
java.home D:\Jenkins\jre
java.io.tmpdir C:\Windows\TEMP\
java.library.path D:\Jenkins\jre\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;d:\Tools\Git-2.18.0\bin;;.
java.runtime.name Java(TM) SE Runtime Environment
java.runtime.version 1.8.0_111-b14
java.specification.name Java Platform API Specification
java.specification.vendor Oracle Corporation
java.specification.version 1.8
java.vendor Oracle Corporation
java.vendor.url http://java.oracle.com/
java.vendor.url.bug http://bugreport.sun.com/bugreport/
java.version 1.8.0_111
java.vm.info mixed mode
java.vm.name Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name Java Virtual Machine Specification
java.vm.specification.vendor Oracle Corporation
java.vm.specification.version 1.8
java.vm.vendor Oracle Corporation
java.vm.version 25.111-b14
javamelody.About Monitoring https://wiki.jenkins-ci.org/display/JENKINS/Monitoring
javamelody.csrf-protection-enabled true
javamelody.custom-reports Jenkins Info,About Monitoring
javamelody.gzip-compression-disabled true
javamelody.http-transform-pattern /\d+/|(?<=/static/|/adjuncts/|/bound/)[\w\-]+|(?<=/ws/|/user/|/testReport/|/javadoc/|/site/|/violations/file/|/cobertura/).+|(?<=/job/).+(?=/descriptorByName/)
javamelody.Jenkins Info /systemInfo
javamelody.maven-repositories C:\Windows\system32\config\systemprofile/.m2/repository,http://repo1.maven.org/maven2,http://repo.jenkins-ci.org/public
javamelody.no-database true
javamelody.storage-directory /D:\Jenkins\monitoring
javamelody.system-actions-enabled true
jetty.git.hash e1bc35120a6617ee3df052294e433f3a25ce7097
jna.loaded true
jnidispatch.path C:\Windows\TEMP\jna-1079805582\jna6949693829815813595.dll
line.separator
mail.smtp.sendpartial true
mail.smtps.sendpartial true
os.arch amd64
os.name Windows Server 2012 R2
os.version 6.3
path.separator ;
pid 1300
sun.arch.data.model 64
sun.awt.enableExtraMouseButtons true
sun.boot.class.path D:\Jenkins\jre\lib\resources.jar;D:\Jenkins\jre\lib\rt.jar;D:\Jenkins\jre\lib\sunrsasign.jar;D:\Jenkins\jre\lib\jsse.jar;D:\Jenkins\jre\lib\jce.jar;D:\Jenkins\jre\lib\charsets.jar;D:\Jenkins\jre\lib\jfr.jar;D:\Jenkins\jre\classes
sun.boot.library.path D:\Jenkins\jre\bin
sun.cpu.endian little
sun.cpu.isalist amd64
sun.desktop windows
sun.io.unicode.encoding UnicodeLittle
sun.java.command D:\Jenkins\jenkins.war --httpPort=80 --webroot=D:\Jenkins\war
sun.java.launcher SUN_STANDARD
sun.jnu.encoding Cp1252
sun.management.compiler HotSpot 64-Bit Tiered Compilers
sun.os.patch.level
svnkit.http.methods Basic
svnkit.ssh2.persistent false
user.country US
user.dir D:\Jenkins
user.home C:\Windows\system32\config\systemprofile
user.language en
user.name [hostname removed]$
user.script
user.timezone Europe/Berlin
user.variant

_________________________________________
Environment Variables

ALLUSERSPROFILE C:\ProgramData
APPDATA C:\Windows\system32\config\systemprofile\AppData\Roaming
BASE D:\Jenkins
CommonProgramFiles C:\Program Files\Common Files
CommonProgramFiles(x86) C:\Program Files (x86)\Common Files
CommonProgramW6432 C:\Program Files\Common Files
COMPUTERNAME [hostname removed]
ComSpec C:\Windows\system32\cmd.exe
DEFLOGDIR C:\ProgramData\McAfee\Endpoint Security\Logs
FP_NO_HOST_CHECK NO
JENKINS_HOME D:\Jenkins
LOCALAPPDATA C:\Windows\system32\config\systemprofile\AppData\Local
NUMBER_OF_PROCESSORS 2
OS Windows_NT
Path C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;d:\Tools\Git-2.18.0\bin;
PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE AMD64
PROCESSOR_IDENTIFIER Intel64 Family 6 Model 58 Stepping 0, GenuineIntel
PROCESSOR_LEVEL 6
PROCESSOR_REVISION 3a00
ProgramData C:\ProgramData
ProgramFiles C:\Program Files
ProgramFiles(x86) C:\Program Files (x86)
ProgramW6432 C:\Program Files
PSModulePath C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC C:\Users\Public
Server Yes
SERVICE_ID Jenkins
SystemDrive C:
SystemRoot C:\Windows
TEMP C:\Windows\TEMP
Terminalserver No
TMP C:\Windows\TEMP
USERDOMAIN [domain removed]
USERNAME [hostname removed]$
USERPROFILE C:\Windows\system32\config\systemprofile
windir C:\Windows
WINSW_EXECUTABLE D:\Jenkins\jenkins.exe
WINSW_SERVICE_ID Jenkins

_________________________________________
Plugins

ace-editor 1.1 true
active-directory 2.16 true
analysis-core 1.96 true
analysis-model-api 7.0.4 true
ant 1.11 true
antisamy-markup-formatter 1.8 true
apache-httpcomponents-client-4-api 4.5.10-2.0 true
artifactory 3.5.0 true
authentication-tokens 1.3 true
bouncycastle-api 2.18 true
branch-api 2.5.5 true
build-monitor-plugin 1.12+build.201809061734 true
build-timeout 1.19 true
calendar-view 0.3.1 true
cloudbees-folder 6.11.1 true
cobertura 1.15 true
code-coverage-api 1.1.3 true
command-launcher 1.4 true
conditional-buildstep 1.3.6 true
config-file-provider 3.6.3 true
configurationslicing 1.47 true
credentials 2.3.1 true
credentials-binding 1.20 true
display-url-api 2.3.2 true
docker-commons 1.16 true
docker-java-api 3.0.14 true
docker-plugin 1.1.9 true
docker-workflow 1.21 true
downstream-build-cache 1.5.1 true
durable-task 1.33 true
email-ext 2.68 true
extended-read-permission 3.0 true
external-monitor-job 1.7 true
forensics-api 0.6.1 true
git 4.1.1 true
git-client 3.1.1 true
git-server 1.9 true
gradle 1.36 true
greenballs 1.15 true
groovy 2.2 true
handlebars 1.1.1 true
htmlpublisher 1.21 true
icon-shim 2.0.3 true
ivy 2.1 true
jackson2-api 2.10.2 true
javadoc 1.5 true
jdk-tool 1.4 true
job-dsl 1.76 true
jquery 1.12.4-1 true
jquery-detached 1.2.1 true
jsch 0.1.55.2 true
junit 1.28 true
ldap 1.21 true
locale 1.4 true
lockable-resources 2.7 true
log-parser 2.1 true
mail-watcher-plugin 1.16 true
mailer 1.30 true
mapdb-api 1.0.9.0 true
matrix-auth 2.5 true
matrix-project 1.14 true
maven-plugin 3.4 true
momentjs 1.1.1 true
monitoring 1.81.0 true
naginator 1.18 true
nodelabelparameter 1.7.2 true
Office-365-Connector 4.12.2 true
pam-auth 1.6 true
parameterized-trigger 2.36 true
periodic-reincarnation 1.13 true
piketec-tpt 8.7 true
pipeline-build-step 2.11 true
pipeline-graph-analysis 1.10 true
pipeline-input-step 2.11 true
pipeline-milestone-step 1.3.1 true
pipeline-model-api 1.5.1 true
pipeline-model-declarative-agent 1.1.1 true
pipeline-model-definition 1.5.1 true
pipeline-model-extensions 1.5.1 true
pipeline-rest-api 2.13 true
pipeline-stage-step 2.3 true
pipeline-stage-tags-metadata 1.5.1 true
pipeline-stage-view 2.13 true
plain-credentials 1.7 true
rebuild 1.31 true
run-condition 1.2 true
scm-api 2.6.3 true
script-security 1.69 true
sectioned-view 1.25 true
ssh-credentials 1.18.1 true
ssh-slaves 1.30.2 true
structs 1.20 true
subversion 2.13.0 true
tasks 4.53 true
throttle-concurrents 2.0.1 true
timestamper 1.10 true
token-macro 2.10 true
trilead-api 1.0.5 true
uno-choice 2.2.2 true
warnings 5.0.1 true
warnings-ng 7.3.0 true
windows-slaves 1.6 true
workflow-aggregator 2.6 true
workflow-api 2.39 true
workflow-basic-steps 2.19 true
workflow-cps 2.78 true
workflow-cps-global-lib 2.15 true
workflow-durable-task-step 2.35 true
workflow-job 2.36 true
workflow-multibranch 2.21 true
workflow-scm-step 2.10 true
workflow-step-api 2.22 true
workflow-support 3.4 true
yet-another-build-visualizer 1.10 true
Priority: Major Major
Reporter: Daniel Bachran

We have defined custom Groovy based warning parsers, for example to parse custom Matlab Simulink warnings output.

When adding the following text snippet to the "Example Log Message" of our custom Matlab Warnings parser, we observe excessively high CPU load on the Jenkins master. This load comes from a thread which seems to be stuck and is displayed by JavaMelody as:

  • Handling POST /descriptorByName/io.jenkins.plugins.analysis.warnings.groovy.GroovyParser/checkExample from [ip address removed] : qtp2052001577-666216

This thread (and possible sub-threads) is constantly consuming CPU time, easily causing something between 50-90% CPU load on our Jenkins master. Rendering the system almost unable to process something else properly.

Workaround: Killing the thread via JavaMelody reduces the CPU load to normal (around 5-10%).

The thread reappears once the Jenkins configuration page is opened again, e.g. https://jenkins/configure

This bug probably first occured after upgrading Jenkins LTS and the Warnings NG plugin sometime in October/November 2019. Unfortunately, I cannot tell anymore which versions were involved, as it took until now to realize the root cause of the sporadic high CPU load of our master...

And yes, this Groovy script and/or the regex does not work properly. But still, Jenkins should not run into such issues in this case.

Regular Expression

^\[.(Warning|Info|Caused by):(.*\n.*)*?\].\n\[.>(.*\n.*)*?\]

Mapping Script

 

import edu.hm.hafner.analysis.Severity
String type = matcher.group(1)
String message = matcher.group(2) + "\n" + matcher.group(3)
// Severity has: ERROR, WARNING_HIGH, WARNING_NORMAL, WARNING_LOW
switch (type) {
 case 'Warning':
 severity = Severity.ERROR
 break
 case 'Info':
 severity = Severity.WARNING_HIGH
 break
 case 'Caused by':
 severity = Severity.WARNING_LOW
 break
 default:
 severity = Severity.WARNING_NORMAL // unknown, should not happen
 break
}
String filename = ''
if (message =~ m/file ["'](.+)['"]/i) {
 filename = $1
}
return builder.setSeverity(severity)
 .setType(type)
 .setFileName(filename)
 .setMessage(message)
 .buildOptional()

 

Example Log Message

[ Warning: Minimum '0' and maximum '255' from 'Path/ModelElement' are mapped to a portion of the wide
signal coming from output port 1 of 'Path/SubPath/Switch' and have been ignored during range checking. To
enable the range checking for the minimum and maximum values, insert a Signal Conversion block with its
'Output' parameter set to 'Signal copy' in front of 'Path/ModelElement'.] 
[ > In slbus_get_struct
 In sl (line 15)
 In Simulink.Bus.createObject>createObjectFromBlks (line 169)
 In Simulink.Bus.createObject (line 149)
 In CqsModelCreateBusObjectsMissing/createBusObjects (line 301)
 In CqsModelCreateBusObjectsMissing/processWholeModel (line 86)
 In run_05_prepareImpModelStart (line 142)]

Java Thread Dump

"Handling POST /descriptorByName/io.jenkins.plugins.analysis.warnings.groovy.GroovyParser/checkExample from [ip address removed] : qtp2052001577-803368" prio=5 RUNNABLE
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match0(Unknown Source)}}
{{ java.util.regex.Pattern$Curly.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$LazyLoop.matchInit(Unknown Source)}}
{{ java.util.regex.Pattern$Prolog.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupTail.match(Unknown Source)}}
{{ java.util.regex.Pattern$BranchConn.match(Unknown Source)}}
{{ java.util.regex.Pattern$Slice.match(Unknown Source)}}
{{ java.util.regex.Pattern$Branch.match(Unknown Source)}}
{{ java.util.regex.Pattern$GroupHead.match(Unknown Source)}}
{{ java.util.regex.Pattern$CharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$BmpCharProperty.match(Unknown Source)}}
{{ java.util.regex.Pattern$Caret.match(Unknown Source)}}
{{ java.util.regex.Pattern$Start.match(Unknown Source)}}
{{ java.util.regex.Matcher.search(Unknown Source)}}
{{ java.util.regex.Matcher.find(Unknown Source)}}
{{ io.jenkins.plugins.analysis.warnings.groovy.GroovyParser$DescriptorImpl.parseExample(GroovyParser.java:381)}}
{{ io.jenkins.plugins.analysis.warnings.groovy.GroovyParser$DescriptorImpl.doCheckExample(GroovyParser.java:343)}}
{{ java.lang.invoke.LambdaForm$DMH/1622225974.invokeVirtual_L4_L(LambdaForm$DMH)}}
{{ java.lang.invoke.LambdaForm$BMH/822603816.reinvoke(LambdaForm$BMH)}}
{{ java.lang.invoke.LambdaForm$MH/2096875522.invoker(LambdaForm$MH)}}
{{ java.lang.invoke.LambdaForm$MH/2072658356.invokeExact_MT(LambdaForm$MH)}}
{{ java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)}}
{{ org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)}}
{{ org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)}}
{{ org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)}}
{{ org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)}}
{{ org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)}}
{{ org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)}}
{{ org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)}}
{{ org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)}}
{{ org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)}}
{{ org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)}}
{{ org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)}}
{{ org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)}}
{{ org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)}}
{{ org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)}}
{{ org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)}}
{{ org.kohsuke.stapler.Stapler.service(Stapler.java:238)}}
{{ javax.servlet.http.HttpServlet.service(HttpServlet.java:790)}}
{{ org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)}}
{{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)}}
{{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)}}
{{ jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)}}
{{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)}}
{{ com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)}}
{{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)}}
{{ net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)}}
{{ net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)}}
{{ net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)}}
{{ org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)}}
{{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)}}
{{ hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)}}
{{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)}}
{{ hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)}}
{{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)}}
{{ jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)}}
{{ hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)}}
{{ hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)}}
{{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)}}
{{ hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)}}
{{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)}}
{{ hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)}}
{{ jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)}}
{{ org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)}}
{{ org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)}}
{{ org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)}}
{{ jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)}}
{{ org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)}}
{{ hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)}}
{{ hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)}}
{{ hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)}}
{{ hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)}}
{{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)}}
{{ org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)}}
{{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)}}
{{ hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)}}
{{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)}}
{{ org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)}}
{{ org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)}}
{{ org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)}}
{{ org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)}}
{{ org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)}}
{{ org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)}}
{{ org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)}}
{{ org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)}}
{{ org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)}}
{{ org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)}}
{{ org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)}}
{{ org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)}}
{{ org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)}}
{{ org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)}}
{{ org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)}}
{{ org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)}}
{{ org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)}}
{{ org.eclipse.jetty.server.Server.handle(Server.java:505)}}
{{ org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)}}
{{ org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)}}
{{ org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)}}
{{ org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)}}
{{ org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)}}
{{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)}}
{{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)}}
{{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)}}
{{ org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)}}
{{ org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)}}
{{ org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)}}
{{ org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)}}
{{ java.lang.Thread.run(Unknown Source)}}
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

daniel@bachran.de (JIRA)

unread,
Feb 13, 2020, 8:13:04 AM2/13/20
to jenkinsc...@googlegroups.com
Daniel Bachran updated an issue
Change By: Daniel Bachran
We have defined custom Groovy based warning parsers, for example to parse custom Matlab Simulink warnings output.

When adding the following text snippet to the "Example Log Message" of our custom Matlab Warnings parser, we observe excessively high CPU load on the Jenkins master. This load comes from a thread which seems to be stuck and is displayed by JavaMelody as:
* Handling POST /descriptorByName/io.jenkins.plugins.analysis.warnings.groovy.GroovyParser/checkExample from [ip address removed] : qtp2052001577-666216


This thread (and possible sub-threads) is constantly consuming CPU time, easily causing something between 50-90% CPU load on our Jenkins master. Rendering the system almost unable to process something else properly.

*Workaround*: Killing the thread via JavaMelody reduces the CPU load to normal (around 5-10%).


The thread reappears once the Jenkins configuration page is opened again, e.g. [https://jenkins/configure]

This bug probably first occured after upgrading Jenkins LTS and the Warnings NG plugin sometime in October/November 2019. Unfortunately, I cannot tell anymore which versions were involved, as it took until now to realize the root cause of the sporadic high CPU load of our master...

And yes, this Groovy script and/or the regex does not work properly. But still, Jenkins should not run into such issues in this case.

*Regular Expression*
{code:java}
^\[.(Warning|Info|Caused by):(.*\n.*)*?\].\n\[.>(.*\n.*)*?\]{code}
*Mapping Script*

 
{code:java}
{code}
 

*Example Log Message*
{code:java}

[ Warning: Minimum '0' and maximum '255' from 'Path/ModelElement' are mapped to a portion of the wide
signal coming from output port 1 of 'Path/SubPath/Switch' and have been ignored during range checking. To
enable the range checking for the minimum and maximum values, insert a Signal Conversion block with its
'Output' parameter set to 'Signal copy' in front of 'Path/ModelElement'.]
[ > In slbus_get_struct
In sl (line 15)
In Simulink.Bus.createObject>createObjectFromBlks (line 169)
In Simulink.Bus.createObject (line 149)
In CqsModelCreateBusObjectsMissing/createBusObjects (line 301)
In CqsModelCreateBusObjectsMissing/processWholeModel (line 86)
In run_05_prepareImpModelStart (line 142)] {code}
*Java Thread Dump*
{ { code:java}
{{ java.lang.Thread.run(Unknown Source) {code } }

ullrich.hafner@gmail.com (JIRA)

unread,
Feb 13, 2020, 10:27:02 AM2/13/20
to jenkinsc...@googlegroups.com
Ulli Hafner assigned an issue to Unassigned
Change By: Ulli Hafner
Priority: Major Minor
Labels: help-wanted
Assignee: Ulli Hafner
Reply all
Reply to author
Forward
0 new messages