Jenkins web interface doesn't respond in high load

[Viacheslav Dubrovskyi]

Hi.

Out Jenkins installation use 24CPU 60G RAM GCE node. And connected about
400+ slaves. Mainly used pipeline in .

Some time web interface doesn't respond. It looks like 504 Gateway
Time-out. So respond time very long. Usually happens at high load (LA >
30).

Have such java and jenkins settings:

JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT
--httpKeepAliveTimeout=6000000 "
JENKINS_ARGS+=" -Dpermissive-script-security.enabled=true "
JAVA_ARGS+=" -Xmx40600m -Xms15240m"
JAVA_ARGS+=" -server -XX:+AlwaysPreTouch"
JAVA_ARGS+=" -Dhudson.DNSMultiCast.disabled=true"
JAVA_ARGS+="
-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=40
"
JAVA_ARGS+=" -Xloggc:$JENKINS_HOME/logs/gc-%t.log
-XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=20m
-XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails
-XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution
-XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy"
JAVA_ARGS+=" -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent
-XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication
-XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1"

My questions:

1. Please look to my settings and advice some changes for jetty
responsiveness?

2. Is it possible separate Jetty (webserver) and jenkins backend on
different servers?

3. How can I add logging to jetty to see only errors and warnings? Is it
correct?

Add this JAVA_ARGS and file

JAVA_ARGS+="
-Djava.util.logging.config.file=/var/lib/jenkins/jul.properties "

# cat /var/lib/jenkins/jul.properties

handlers=java.util.logging.ConsoleHandler
.level=INFO
org.eclipse.jetty.level=ERROR
java.util.logging.ConsoleHandler.level=FINEST

4. Is it possible to restart jetty thread from console?


Thank you!

--
WBD,
Viacheslav Dubrovskyi