Extremely LONG lag time when scheduling pipelines

[Lee Benhart]

We are seeing extremely low scheduling times when trying to start pipelines.  It can take upwards of an hour for a pipeline to successfully schedule.  The only significant change made was to add some pipelines to our configuration file yesterday, which has the overall config file size near 3MB.

[srinivas upadhya]

Can you post <go-server>/go/api/support & <go-server>/go/api/process_list here? It should tell us whats going on hopefully.

Note: Sanitize files for any sensitive information before posting.

[Lee Benhart]

Actually, this seems to be related to large material list.  There was a large checkin of 40,000+ files.  Has the issue of better handling large checkins been addressed?

We are running 14.1.0(18882-d0272e1d227b5e).

[srinivas upadhya]

Analysis:

There are no blocked threads.

In the attached file I see these settings:

cruise.material.update.interval=300000

cruise.produce.build.cause.delay=300000

cruise.material.update.delay=300000

Defaults are:

cruise.material.update.interval=30000

cruise.produce.build.cause.delay=10000

cruise.material.update.delay=10000

This could the cause for long delays in scheduling. Could you please revert those settings & see if things get better?

On Wed, Mar 11, 2015 at 7:44 PM, Lee Benhart <lee.b...@gmail.com> wrote:

Actually, this seems to be related to large material list.  There was a large checkin of 40,000+ files.  Has the issue of better handling large checkins been addressed?

​Its not handled yet.​ Can you please create an issue for that as well. Thanks!

We are running 14.1.0(18882-d0272e1d227b5e).

On Wednesday, March 11, 2015 at 9:44:10 AM UTC-4, srinivas upadhya wrote:

Can you post <go-server>/go/api/support & <go-server>/go/api/process_list here? It should tell us whats going on hopefully.

Note: Sanitize files for any sensitive information before posting.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aravind SV]

The delays are fine, I think, since they affect only startup. I think you can even keep those. The interval can be made smaller, probably 60000. Otherwise the commit check happens only once every 5 minutes.

But, I think Lee is right. The delay is probably because of the huge commit, not because of these settings.

[Lee Benhart]

We need to delay the material polling to our perforce server, it was causing significant issues.  I am ok with restoring defaults, but we want to maintain 5 minutes polling interval.  Even this is a drain on the servers since ALL pipelines poll a the same time. Which are the correct settings to insure delayed source repository polling?

Lee

[Lee Benhart]

Issue created #946