[workflow-plugin] stage concurrency not working outside of node scope?

[sebastien....@gmail.com]

Hello,

I'm experimenting with the "stage" command using the flow below, and running the job 4 times in a row (#59, #60, #61, #62).

My test show that the concurrency limit I set for stage 'build_and_test' is not accounted for, but the one for the 'test' stage is (great feature, thanks!).

Maybe limiting concurrency only works within a node scope? Or am I missing something?

Doc is quite sparce regarding the stage command.

stage name: 'sync'
node("qisrc") {
  echo "before sync"
  sh "sleep 10"
  echo "after sync"
}

stage name: 'build_and_test', concurrency: 1
node("qisrc") {
  stage name: 'build'
  echo "before build"
  sh "sleep 10"
  echo "build 1/3"
  sh "sleep 10"
  echo "build 2/3"
  sh "sleep 10"
  echo "build 3/3"

  stage name: 'test', concurrency: 1
  echo "before test"
  sh "sleep 10"
  echo "test 1/3"
  sh "sleep 10"
  echo "test 2/3"
  sh "sleep 10"
  echo "test 3/3"
}

job/workflow/59/console:

Started by user admin
Running: sync
Entering stage sync
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow
Running: Allocate node : Body : Start
Running: Print Message
before sync
Running: Shell Script
[workflow] Running shell script
+ sleep 10
Running: Print Message
after sync
Running: Allocate node : Body : End
Running: Allocate node : End
Running: build_and_test
Entering stage build_and_test
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow
Running: Allocate node : Body : Start
Running: build
Entering stage build
Proceeding
Running: Print Message
before build
Running: Shell Script
[workflow] Running shell script
+ sleep 10
Running: Print Message
build 1/3
Running: Shell Script
[workflow] Running shell script
+ sleep 10
Running: Print Message
build 2/3
Running: Shell Script
[workflow] Running shell script
+ sleep 10
Running: Print Message
build 3/3
Running: test
Entering stage test
Proceeding
Running: Print Message
before test
Running: Shell Script
[workflow] Running shell script
+ sleep 10
Running: Print Message
test 1/3
Running: Shell Script
[workflow] Running shell script
+ sleep 10
Running: Print Message
test 2/3
Running: Shell Script
[workflow] Running shell script
+ sleep 10
Running: Print Message
test 3/3
Running: Allocate node : Body : End
Running: Allocate node : End
Running: End of Workflow
Finished: SUCCESS

job/workflow/60/console:

Started by user admin
Running: sync
Entering stage sync
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow@2
Running: Allocate node : Body : Start
Running: Print Message
before sync
Running: Shell Script
[workflow@2] Running shell script
+ sleep 10
Running: Print Message
after sync
Running: Allocate node : Body : End
Running: Allocate node : End
Running: build_and_test
Entering stage build_and_test
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow@2
Running: Allocate node : Body : Start
Running: build
Entering stage build
Proceeding
Running: Print Message
before build
Running: Shell Script
[workflow@2] Running shell script
+ sleep 10
Running: Print Message
build 1/3
Running: Shell Script
[workflow@2] Running shell script
+ sleep 10
Running: Print Message
build 2/3
Running: Shell Script
[workflow@2] Running shell script
+ sleep 10
Running: Print Message
build 3/3
Running: test
Entering stage test
Waiting for builds [59]
Canceled since #61 got here
Running: Allocate node : Body : End
Running: Allocate node : End
Running: End of Workflow
Superseded by #61
Finished: NOT_BUILT

http://agility-test.local/job/workflow/61/console

Started by user admin
Running: sync
Entering stage sync
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow@3
Running: Allocate node : Body : Start
Running: Print Message
before sync
Running: Shell Script
[workflow@3] Running shell script
+ sleep 10
Running: Print Message
after sync
Running: Allocate node : Body : End
Running: Allocate node : End
Running: build_and_test
Entering stage build_and_test
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow@3
Running: Allocate node : Body : Start
Running: build
Entering stage build
Proceeding
Running: Print Message
before build
Running: Shell Script
[workflow@3] Running shell script
+ sleep 10
Running: Print Message
build 1/3
Running: Shell Script
[workflow@3] Running shell script
+ sleep 10
Running: Print Message
build 2/3
Running: Shell Script
[workflow@3] Running shell script
+ sleep 10
Running: Print Message
build 3/3
Running: test
Entering stage test
Canceling older #60
Waiting for builds [59]
Canceled since #62 got here
Running: Allocate node : Body : End
Running: Allocate node : End
Running: End of Workflow
Superseded by #62
Finished: NOT_BUILT

http://agility-test.local/job/workflow/62/console

Started by user admin
Running: sync
Entering stage sync
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow@4
Running: Allocate node : Body : Start
Running: Print Message
before sync
Running: Shell Script
[workflow@4] Running shell script
+ sleep 10
Running: Print Message
after sync
Running: Allocate node : Body : End
Running: Allocate node : End
Running: build_and_test
Entering stage build_and_test
Proceeding
Running: Allocate node : Start
Running on qisrc in /home/jenkins/jenkins/workspace/workflow@4
Running: Allocate node : Body : Start
Running: build
Entering stage build
Proceeding
Running: Print Message
before build
Running: Shell Script
[workflow@4] Running shell script
+ sleep 10
Running: Print Message
build 1/3
Running: Shell Script
[workflow@4] Running shell script
+ sleep 10
Running: Print Message
build 2/3
Running: Shell Script
[workflow@4] Running shell script
+ sleep 10
Running: Print Message
build 3/3
Running: test
Entering stage test
Canceling older #61
Waiting for builds [59]
Unblocked since #59 finished
Running: Print Message
before test
Running: Shell Script
[workflow@4] Running shell script
+ sleep 10
Running: Print Message
test 1/3
Running: Shell Script
[workflow@4] Running shell script
+ sleep 10
Running: Print Message
test 2/3
Running: Shell Script
[workflow@4] Running shell script
+ sleep 10
Running: Print Message
test 3/3
Running: Allocate node : Body : End
Running: Allocate node : End
Running: End of Workflow
Finished: SUCCESS

 

[sebastien....@gmail.com]

Hi again,

I still wonder wether this is a bug, which I should file, or not.
Maybe I shall detail a bit further, see below.

Thanks !

On Friday, February 20, 2015 at 1:06:38 PM UTC+1, sebastien....@gmail.com wrote:

Hello,

I'm experimenting with the "stage" command using the flow below, and running the job 4 times in a row (#59, #60, #61, #62).

My test show that the concurrency limit I set for stage 'build_and_test' is not accounted for, but the one for the 'test' stage is (great feature, thanks!).

Maybe limiting concurrency only works within a node scope? Or am I missing something?

Doc is quite sparce regarding the stage command.

stage name: 'sync'
node("qisrc") {
  echo "before sync"
  sh "sleep 10"
  echo "after sync"
}

stage name: 'build_and_test', concurrency: 1

Here I expect the first job to continue, and the others to wait, in a stack of size one (a newcomer kicks the older one off).
However it did not happen that way (see below) they all continue to the next stage irrespective of the concurrency constraint.

node("qisrc") {
  stage name: 'build'
  echo "before build"
  sh "sleep 10"
  echo "build 1/3"
  sh "sleep 10"
  echo "build 2/3"
  sh "sleep 10"
  echo "build 3/3"

  stage name: 'test', concurrency: 1

I expect this "concurrency constraint" to be superfluous since the previous stage command constrained the concurrency to one already.
However, in practice, since the previous get ignored, the concurrent jobs do wait here.
 

This is expected since this is the first job (#59).

This is expected since this is the first job (#59).

This is unexpected since this the first job (#59) is still running.

So, at this step, we (job #60) aknowledge that job #59 is running and wait. Weird, it should have happened ate the build_and_test stage instead.

Canceled since #61 got here

Idem, it should have happened ate the build_and_test stage instead.

Jobs 61 and 62 below behave the same way, they wait at stage test instead of build_and_test.

[Jesse Glick]

On Tuesday, February 24, 2015 at 12:55:38 PM UTC-5, sebastien....@gmail.com wrote:

I still wonder wether this is a bug, which I should file, or not.

I did recently fix a problem with the `stage` step not honoring `concurrency` in some cases, which will go into 1.3.

https://issues.jenkins-ci.org/browse/JENKINS-27052

[sebastien....@gmail.com]

Hi Jesse,

thank you for answering.

 based on the commit logs, I'm not sure this is the same issue.

Did you test my simple workflow?

Or is there as way for me to test it on a daily build of the plugin?

Thanks!

[Jesse Glick]

On Thu, Feb 26, 2015 at 3:59 PM, <sebastien....@gmail.com> wrote:
> Did you test my simple workflow?

No, sorry, was just guessing that your issue was a duplicate.

> Or is there as way for me to test it on a daily build of the plugin?

$ git clone https://github.com/jenkinsci/workflow-plugin.git
$ mvn -f workflow-plugin -DskipTests install
$ mvn -f workflow-plugin/aggregator hpi:run
$ <your-browser> http://localhost:8080/jenkins/

[sebastien....@gmail.com]

Hi Jesse,

I just tested with workflow-1.3-39-g694a0f0, and cannot reproduce the issue.

Thank you!

[niristotle okram]

Please Excuse & advise, if this is not appropriate to continue post in an old post. 

We are using the cloudbees jenkins. And are optimizing the workflows (the new name: pipeline) to get some concurrency going. The way i understand concurrency is the simultaneous execution. So a value of 

concurrency: 1 --> allow only one instance for the stage

concurrency: n --> allow 'n' number of instance for the stage

  

Am i am interpreting it correctly? 

What if the concurrency is not specified (and the option to execute concurrently in the workflow config is not checked)? Will the jobs run serially, meaning stay as pending until the first job is over? 

What is concurrency: 0 (a came accross that 0 means unlimited concurrency! is this correct)?

And to get some clarity, i used the workflow given by Sebastien here. But to get some clarity on the timelines, i entered the date command to mark the timelines for the stage's start and end. below is the script. 

stage name: 'sync'

node("master") {

  echo "before sync"

  sh "sleep 10"

  echo "after sync"

  sh "date"

}

stage name: 'build_and_test', concurrency: 1

node("master") {

  stage name: 'build'

  echo "before build"

  sh "date"

  sh "sleep 10"

  echo "build 1/3"

  sh "sleep 10"

  echo "build 2/3"

  sh "sleep 10"

  echo "build 3/3"

  sh "date"

  stage name: 'test', concurrency: 1

  echo "before test"

  sh "date"

  sh "sleep 10"

  echo "test 1/3"

  sh "sleep 10"

  echo "test 2/3"

  sh "sleep 10"

  echo "test 3/3"

  sh "date"

}

I still notice the issue reported by Sebastien. i have set up 7 executors and fired 4 jobs on the same workflow. 

1. All the 4 jobs triggered the first stage of sync

2. All the 4 jobs triggered the second stage of build_and_test which have a concurrency: 1

3. NOT all the 4 triggers the proceeds to the 'stage name: 'test', concurrency: 1' 

Why are the jobs running concurrently for 'stage name: 'build_and_test', concurrency: 1' ??

IS my understanding on the concurrency flawed? 

job console output:

http://pastebin.com/n6tJJ0Zu

http://pastebin.com/x9tryKUV

http://pastebin.com/6hsan1KY

http://pastebin.com/kPbtxKbj

This is a simple timeline for all the 4 jobs

Stage		Sync		'build_and_test', concurrency: 1				'test', concurrency: 1
	26	Fri Jan 29 16:20:39 UTC 2016	Fri Jan 29 16:20:40 UTC 2016		Fri Jan 29 16:21:10 UTC 2016		Fri Jan 29 16:21:11 UTC 2016		Fri Jan 29 16:21:41 UTC 2016
	27	Fri Jan 29 16:20:43 UTC 2016	Fri Jan 29 16:20:43 UTC 2016		Fri Jan 29 16:21:14 UTC 2016		Entering stage test/Waiting for builds [26]/Canceled since #28 got here
	28	Fri Jan 29 16:20:46 UTC 2016	Fri Jan 29 16:20:47 UTC 2016		Fri Jan 29 16:21:17 UTC 2016		Entering stage test/Canceling older #27/Waiting for builds [26]/Canceled since #29 got here
	29	Fri Jan 29 16:20:51 UTC 2016	Fri Jan 29 16:20:51 UTC 2016		Fri Jan 29 16:21:22 UTC 2016		Fri Jan 29 16:21:42 UTC 2016		Fri Jan 29 16:22:12 UTC 2016

[niristotle okram]

Things looks much better with the flow below:

stage name: 'sync'

node("master") {

  echo "before sync"

  sh "sleep 10"

  echo "after sync"

  sh "date"

}

node("master") {

  stage name: 'build', concurrency: 1

  echo "before build"