[1.509.1 LTS] Massive performance issues

[sbreit]

Hi everyone!

After updating to 1.509.1 LTS (from 1.480.3) we're facing massive performance issues. Performance was OK before, but now loading pages takes significantly longer than before. Certain jobs also have a way longer runtime than before.

Sometimes, Jenkins even won't respond any more. We're using Tomcat 6.0.36 as container. Tomcat itself still was responsive, but Jenkins was hanging for about 10 minutes. This occurred twice so far. When this happens, load and I/O on the machine itself are low, so we don't suspect issues with the machine or the OS itself. There isn't something of significance in the logs, either.

We tried to get to the bottom of this issue, but we weren't successful yet. Monitoring the JVM or the OS didn't bring up anything.

Anyone noticed a similar behaviour yet?

Regards

  Steffen

[Stephen Connolly]

From what I know, this is a side-effect of the new lazy loading of build data.

Bad plugins force Jenkins to load all the build data. Because of lazy loading, Jenkins is able to throw away the lazy loaded data (not sure if it is or isn't because I have been too busy with other things to look at the code). So if you have a bad plugin you basically force loading of the data on page request, gc purges after page displayed, lather, rince, repeat.

One way to isolate the issue is to try disabling half the plugins, see if that restores performance vs trying the other half disabled...

A few mixes of disabling half of the plugins, e.g.

1. disable first half, enable second half

2. enable first half, disable second half

3. disable every even plugin, enable every odd

4. enable every even plugin, disable every odd

That should give a reasonably small intersection set. Now you can enable all the plugins not on the intersection set and repeat the process for those on the intersection set until you find all culprets

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Arnaud Héritier]

Hi

I have such issue and downgraded back to 1.480.3 LTS. It was the same with community versions since the introduction of the lazy loading. From my point of view the most culprit plugin in my case is the dashboard plugin and all it's portlets loading all the history to display various statistics. 

You can see how I'm using it on https://ci.exoplatform.org

Sadly I found no time for now to try to upgrade and disable this plugin to prove it, nor to analyze deeply what is wrong in this plugin. 

Arnaud. 

—
Sent from Mailbox for iPhone

[jcsirot]

Hello Steffen, 

Yes, I'm facing the same hanging issue. It seems that it appeared with the patch providing the lazy-loading build data (around 1.485). Like you I tried to look for the reason but I don't have much time to spend on this issue.

7 month ago, I opened a JIRA ticket https://issues.jenkins-ci.org/browse/JENKINS-15601. Unresolved so far :(

-- 

Jean-Christophe

[alex ouzounis]

Hi all,

Was facing the same issue with lazy-loading. Updated to 1.511 which includes a fix for issue 16845 and the issue is gone for me.

Alex

[Steffen Breitbach]

Hi Arnaud!

On 19.05.2013 11:48, Arnaud H�ritier wrote:
> I have such issue and downgraded back to 1.480.3 LTS.

I'm also looking into this as a possible fix for our current situation.

Did you face any issues after downgrading to 1.480.3 LTS ?

Regards
Steffen

[Vladimir Zak]

Hi,

I think it is because of the dashboard plugin.

Is this problem already reported?

Vladimir

2013/5/21 Steffen Breitbach <Steffen....@1und1.de>

Hi Arnaud!

On 19.05.2013 11:48, Arnaud Héritier wrote:

I have such issue and downgraded back to 1.480.3 LTS.

I'm also looking into this as a possible fix for our current situation.

Did you face any issues after downgrading to 1.480.3 LTS ?

Regards
  Steffen

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.