Very slow view render

[Péter Major]

Hi,

I have a Jenkins instance (1.609.1) with 200-300 jobs or so, and I find the view rendering rather slow. It is so slow, that sometimes the Apache reverse proxy in front of my Tomcat just gives up and returns an HTTP 500 instead. I've had a look at some random jstacks, and the following appears to be interesting:

at hudson.model.Run.reload(Run.java:332)

at hudson.model.Run.<init>(Run.java:320)

at hudson.model.AbstractBuild.<init>(AbstractBuild.java:178)

at hudson.maven.AbstractMavenBuild.<init>(AbstractMavenBuild.java:51)

at hudson.maven.MavenBuild.<init>(MavenBuild.java:128)

at sun.reflect.GeneratedConstructorAccessor1318.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:158)

at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:135)

at hudson.model.RunMap.retrieve(RunMap.java:222)

at hudson.model.RunMap.retrieve(RunMap.java:57)

at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:465)

- locked <0x3aa6cff8> (a hudson.model.RunMap)

at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:448)

at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)

at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)

at jenkins.model.lazy.LazyBuildMixIn.getNearestOldBuild(LazyBuildMixIn.java:254)

at hudson.model.AbstractProject.getNearestOldBuild(AbstractProject.java:991)

at hudson.maven.MavenModuleSetBuild.getModuleLastBuilds(MavenModuleSetBuild.java:487)

at hudson.maven.MavenModuleSetBuild.computeResult(MavenModuleSetBuild.java:230)

at hudson.maven.MavenModuleSetBuild.getResult(MavenModuleSetBuild.java:221)

- locked <0x8b0248a8> (a java.lang.Object)

at hudson.model.Run.getIconColor(Run.java:743)

at hudson.model.Job.getBuildStabilityHealthReport(Job.java:1129)

at hudson.model.Job.getBuildHealthReports(Job.java:1104)

For me this means that in order to get back a health report of a single job, Jenkins will try to load the previous builds to be able to determine the right weather, but that takes somewhat long.

Looking at MavenModuleSetBuild#computeResult I don't really understand why it's necessary to go through all the MavenModules just to come up with the final result, the build's end result should be saved somewhere in the MavenModuleSetBuild, or am I missing something? Since most of my build jobs are Maven builds (few of which has 100+ modules) I wonder if the problem is somewhere around the Maven plugin, and not necessarily with the weather report.

Removing the weather report from the column list helped a bit, but unfortunately then the status column produces similar slowness:

at hudson.model.AbstractProject.getNearestOldBuild(AbstractProject.java:991)

at hudson.maven.MavenModuleSetBuild.getModuleLastBuilds(MavenModuleSetBuild.java:487)

at hudson.maven.MavenModuleSetBuild.computeResult(MavenModuleSetBuild.java:230)

at hudson.maven.MavenModuleSetBuild.getResult(MavenModuleSetBuild.java:221)

- locked <0x532c9de8> (a java.lang.Object)

at hudson.model.Run.getIconColor(Run.java:743)

at hudson.model.Job.getIconColor(Job.java:1058)

at hudson.model.AbstractProject.getIconColor(AbstractProject.java:734)

Disabling status column as well improved the performance quite a bit, but:

* the status is actually quite helpful to see on a view

* I couldn't find a way yet to remove Status+Weather columns from the default "All" view just yet

I have bumped into the following JIRA issues:

https://issues.jenkins-ci.org/browse/JENKINS-25078

https://issues.jenkins-ci.org/browse/JENKINS-25075

Neither of them are really good matches for my description though.. I got to wonder though, how does any of this work when a given job is tied to a certain build slave? Is the master trying to read all the build data from the remote instance, or is it outsourcing this task to the slave (Not quite sure how slaves really work yet..)?

Could it be that the view rendering should be multithreaded (each thread collecting the necessary information for a collection of jobs)? Whilst that wouldn't resolve the underlying slowness problem, at least Jenkins would be a lot more faster.

Any insight on this would be very much appreciated.

Regards,

Peter

[milki milk]

I've had this problem with every version I've tried in Jenkins. From what I can tell, Jenkins is parsing through every single build directory and parsing the build xml files as part of either cleanup, or sanity checking, or some other automated task. The logs are quite activate but the render only says "Wait while Jenkins is loading..." Only once Jenkins finished this does the start rendering, but this could be 30 minutes or more.