pipline slow vs flow

[Chris Overend]

So I am doing testing on our current system.

We use flows now.

So I have a flow and a pipeline doing the same task:

• 100 executors
  
• 2000 builds
• build - echo 'Hello World'

flow time = 2 min 33 sec

pipeline time = 5 min 11 sec

After all the echos are complete the pipeline seems to hang before completing

pipeline script

#!groovy import hudson.model.* stage('many_jobs') { parallel (get_many_jobs(2000)) } def get_many_jobs(num_builds) { // The map we'll store the parallel steps in before executing them. def stepsForParallel = [:] for (int i = 0; i < num_builds; i++) { def sub_run_config = 'job_' + i stepsForParallel[sub_run_config] = { build job: 'sq_echo', parameters: [string(name: 'job_name', value: sub_run_config)] } } return stepsForParallel }

Thanks 

[Daniel Beck]

> On 13.02.2017, at 18:42, Chris Overend <chris....@vectorcast.com> wrote:
>
> So I am doing testing on our current system.

If your use case is to print "Hello World" a lot, it looks like you're better off sticking with Build Flow.

Seriously though -- I wonder what you're trying to say here. Pipeline probably isn't optimized for startup time. Why should it be?

Note that Jesse (Pipeline main developer) keeps saying to do real work in actual build scripts and build tools and to just glue them together with Pipeline. If you do that, any performance differences will be negligible.

[Chris Overend]

I am not trying to say anything I just found it curious. 

We are doing testing to determine the optimal setup for Jenkins.

We currently use flows calling flows.

We believe we have too many builds in the build queue (3500).

In the process of testing we saw this behavior and I was hoping for insight.

We have seen the more builds we do the less efficient pipeline became.

going from 1500 builds to 2000 builds the pipeline took twice as long.

At 3500 builds

• pipeline - 23 mins
  
• flow - 8 min
  

So how does this become negligible if the jobs are doing work?

Does this overhead disappear?

[Baptiste Mathus]

It /might/ be the price of durability, since in Pipeline each steps is serialized/checkpointed and sent from the master to somewhere to run on. This enables it to restart in case of crash for instance, which build flow can't do at all. And that price is probably more visible if each step basically does nothing.

Also, you should be aware that Build Flow is deprecated de facto. His creator abandoned it, and it gave many good feedbacks to design Pipeline. And the last current maintainer I knew also abandoned it a few months ago.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d3dfdfd4-e5df-48e4-9aec-a11552c7040f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.