Jenkins takes too long for loading recent changes of current job (15min approx)

[Renuka Pampana]

Hi,

We have migrated recently to Jenkins new stable version. After migration, Jenkins takes too long for displaying the changes of the current job (approx. 15min). 

Can you please provide me some suggestions to improve the speed?

Thanks,

Renuka

[Daniel Beck]

When asking about a possible regression, it helps to mention both the version you're using now, as well as the version you were using before.

Try applying the workaround mentioned for SECURITY-243 here:

https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2016-05-11

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/283e63c8-5cd2-459c-8161-0dc951383333%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> <Capture.PNG>

[Ravalika]

Thank you,

Old Jenkin version is 1.6.3 migrated to jenkins stable version 2.7.1

[Ravalika]

I have tried applying the workaround mentioned for SECURITY-243, but no luck

When i click on recent changes, still takes approx 15 min to display the messages.

Thank you

Renuka

[Mark Waite]

The job in the original picture showed only 7 items in its history.  Are some of your jobs retaining many, many history entries?  If so, that will slow the Jenkins startup time, and might slow the time to compute changes.  The configuration slicing plugin is very effective at showing (and changing) the amount of history retained for all jobs in the system.

Has the file system changed any significant characteristics (switched from a local fast file system to a USB 2.0 file system, or to a storage area network or to an NFS mount, or some other form of remote file system)?

Is the slower performance specific to a few jobs, or general to all jobs?  If specific to a few jobs, then you might investigate if the workspace directories for those jobs would benefit by being "wiped" (using the "Wipe workspace" additional behaviour of the git plugin) and reconstructed the next time the job runs.

The job in the original picture showed "Git polling log" as one of the links.  If you're using git, did you change the git version at the same time you updated Jenkins?  Command line git is generally very fast to compute changes, so this is an unlikely scenario.

You could check the "raw git" performance in those repositories by changing into the workspace directory of one or more jobs and executing "git log" commands of various forms to confirm that command line git is performing well.

Mark Waite

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d4ebe6bc-9eba-4e0e-bc1a-1af6be7cf0af%40googlegroups.com.

[Ravalika]

Hi Mark,

Thank you,

answers inline

On Thursday, August 25, 2016 at 6:09:53 PM UTC+5:30, Mark Waite wrote:

The job in the original picture showed only 7 items in its history.  Are some of your jobs retaining many, many history entries?  If so, that will slow the Jenkins startup time, and might slow the time to compute changes.  The configuration slicing plugin is very effective at showing (and changing) the amount of history retained for all jobs in the system.

We have around 10 disabled jobs and 20 active running jobs. For some jobs the history is more. 

Is the  configuration slicing plugin required to show the changes?

Has the file system changed any significant characteristics (switched from a local fast file system to a USB 2.0 file system, or to a storage area network or to an NFS mount, or some other form of remote file system)?

It is same filesystem, We are not accessing nfs mount data. Initially some old builds used to access data from the network, but now we have copied all necessary data in the same server, All data stored in same server  

Is the slower performance specific to a few jobs, or general to all jobs?  If specific to a few jobs, then you might investigate if the workspace directories for those jobs would benefit by being "wiped" (using the "Wipe workspace" additional behaviour of the git plugin) and reconstructed the next time the job runs.

No, The slower performance is not specific to few jobs, general to all jobs.

The job in the original picture showed "Git polling log" as one of the links.  If you're using git, did you change the git version at the same time you updated Jenkins?  Command line git is generally very fast to compute changes, so this is an unlikely scenario.

No, we haven''t upgraded the git version in new server 

You could check the "raw git" performance in those repositories by changing into the workspace directory of one or more jobs and executing "git log" commands of various forms to confirm that command line git is performing well.

I have tried git log combinations in the workspaces. git performance is very fast through command line 

Is there anyway we can debug, why the slow performance apart from jenkins.log?

Thank you very much for your help

[Mark Waite]

On Fri, Aug 26, 2016 at 12:33 AM Ravalika <pre...@gmail.com> wrote:

Is there anyway we can debug, why the slow performance apart from jenkins.log?

You can increase the logging level of the various classes in the git plugin and the git client plugin in hopes that the logging time stamps and messages will give some indication of areas that are slow.  Refer to https://wiki.jenkins-ci.org/display/JENKINS/Logging for instructions to increase log levels from the user interface.  The git client plugin source code (https://github.com/jenkinsci/git-client-plugin) and the git plugin source code (https://github.com/jenkinsci/git-plugin) will show you the classes you can monitor and the logging levels.

You could try capturing periodic stack traces of the Jenkins process while it is computing those changes.  The jvmtop (http://stackoverflow.com/questions/6846049/profiling-a-running-java-application-in-command-line) script seems like it could help there, without requiring recompilation or a profiler.

You could  try running your Jenkins process with VisualVM as a profiler (https://visualvm.java.net/profiler.html).

Mark Waite