[JIRA] [workflow-plugin] (JENKINS-33925) Test framework for Jenkinsfile

29 views
Skip to first unread message

jglick@cloudbees.com (JIRA)

unread,
Mar 31, 2016, 7:17:01 AM3/31/16
to jenkinsc...@googlegroups.com
Jesse Glick created an issue
 
Jenkins / New Feature JENKINS-33925
Test framework for Jenkinsfile
Issue Type: New Feature New Feature
Assignee: Jesse Glick
Components: workflow-plugin
Created: 2016/Mar/31 11:16 AM
Labels: testing
Priority: Major Major
Reporter: Jesse Glick

It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

Mock framework

Inspired by Job DSL (example).

We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.

Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).

Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.

Quick and flexible, but low fidelity to real behavior.

JenkinsRule-style

Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.

Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.

Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Mar 31, 2016, 7:19:02 AM3/31/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
Change By: Jesse Glick
It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

h3. Mock framework

Inspired by Job DSL ([example|https://github.com/sheehan/job-dsl-gradle-example/blob/e240056da6691bf0a2fdc99e5aab33bc49e42b2f/src/test/groovy/com/dslexample/GrailsCiJobBuilderSpec.groovy#L34-L49]).

We could set up a {{GroovyShell}} in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for {{currentBuild}}.

Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with {{workflow-cps}}).


Should be easy to pass it through the Groovy sandbox (if requested), though the live {{Whitelist.all}} from Jenkins would be unavailable, so we would be limited to known static whitelists, the {{@Whitelisted}} annotation, and perhaps some custom additions.

Quick and flexible, but low fidelity to real behavior.

h3. {{JenkinsRule}}-style

Use an embedded Jenkins server, as per {{JenkinsRule}} in {{jenkins-test-harness}}, and actually create a {{WorkflowJob}} with the specified definition. Can use for example {{mock-slave}} to create nodes.

Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), {{acceptance-test-harness}}, etc.

Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage.  The tests for Pipeline (and Pipeline steps) themselves use this technique.

andrew.bayer@gmail.com (JIRA)

unread,
May 13, 2016, 6:08:01 PM5/13/16
to jenkinsc...@googlegroups.com
Andrew Bayer commented on New Feature JENKINS-33925
 
Re: Test framework for Jenkinsfile

I'm strongly +1 on the JenkinsRule approach. I think the challenge really entirely lies in the "dry-run" matter - how do we determine what behaviors are the ones we should mock/skip/whatever we do? How do we mock/skip/whatever those behaviors? How is that extensible and easy for plugins to support?

pwolf@cloudbees.com (JIRA)

unread,
May 19, 2016, 1:57:02 PM5/19/16
to jenkinsc...@googlegroups.com
Patrick Wolf updated an issue
Change By: Patrick Wolf
Labels: 3.0 testing

pwolf@cloudbees.com (JIRA)

unread,
May 19, 2016, 1:59:06 PM5/19/16
to jenkinsc...@googlegroups.com
Patrick Wolf updated an issue
Change By: Patrick Wolf
Labels: 3.0 followup  testing

andrew.bayer@gmail.com (JIRA)

unread,
May 19, 2016, 2:17:19 PM5/19/16
to jenkinsc...@googlegroups.com

I've been thinking about this a bit more - some thoughts on what I'd like to see in the end result:

  • An easy way to run a test - I think my ideal would be to have a job type that was basically identical to the existing Pipeline job type (with both inline and from-SCM options) that would run the Pipeline script in the test environment.
  • Guaranteeing no side effects - this is the hairy one, obviously. I'm not sure how we'd be able to make sure everything that could have a side effect doesn't - i.e., would we need to interfere and override sh and bat? And def foo = docker.build ... ; foo.push ...? And who knows what else?
  • Going along with that - how do we deal with things like testing if errors happened, or steps that depend on generated output from previous steps that we may have mocked out/overridden?

jglick@cloudbees.com (JIRA)

unread,
May 24, 2016, 4:33:01 PM5/24/16
to jenkinsc...@googlegroups.com

a job type that was basically identical to the existing Pipeline job type (with both inline and from-SCM options) that would run the Pipeline script in the test environment

Hmm, this would be something very different from the JenkinsRule proposal. I do not think this kind of setup would fly. It would be fine for interactive testing but it would not work well for automated testing.

[…] steps that depend on generated output from previous steps that we may have mocked out/overridden

Perhaps the mocking facility could be used in the JenkinsRule scenario, as a kind of hybrid approach. So you could declare that, for example, all mail steps, or the third node step, or any sh step taking the script argument mvn clean test, etc., should print the following output, or return the following result object, or fail with the following error message. Unmentioned steps would run “for real” by default, or you could declare that by default these would fail the test, but you could explicitly ask for some to follow the real behavior.

Some of these facilities would actually be useful in Pipeline plugin tests, too, which would help validate the concept and iron out bugs.

kong.shijun@gmail.com (JIRA)

unread,
Jun 23, 2016, 9:41:01 AM6/23/16
to jenkinsc...@googlegroups.com

I believe a lightweight mock pipeline engine is better than embedded jenkins approach. For the speed of feedback.

I like externalize build process in Jenkinsfile, which I could put into VCS. However, the burden comes when trying out new "step", make sure both syntax and semantic are correct. The entire feedback loop today is too long: commit to VCS, Jenkins pulling the updated Jenkinsfile, running the build for multiple stages, and finally hit the new step, failed.

A
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Aug 29, 2016, 5:02:06 PM8/29/16
to jenkinsc...@googlegroups.com
Jesse Glick updated an issue
Change By: Jesse Glick
Component/s: workflow-cps-plugin
Component/s: pipeline

grose.matthewf@gmail.com (JIRA)

unread,
Aug 30, 2016, 3:33:05 PM8/30/16
to jenkinsc...@googlegroups.com

Has anyone made any movement on this front? Whether related to this ticket or elsewhere?

francois.genois@wolterskluwer.com (JIRA)

unread,
Nov 25, 2016, 1:47:05 PM11/25/16
to jenkinsc...@googlegroups.com

Any updates?

jglick@cloudbees.com (JIRA)

unread,
Mar 6, 2017, 2:37:07 PM3/6/17
to jenkinsc...@googlegroups.com

JenkinsPipelineUnit, developed recently, looks promising. I have not personally used or reviewed it.
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

bill.dennis@gmail.com (JIRA)

unread,
Mar 8, 2017, 5:32:04 AM3/8/17
to jenkinsc...@googlegroups.com

I have used JenkinsPipelineUnit to unit test declarative pipelines and shared code (DSL style implementations in 'vars') with Spock based unit tests in a Gradle project for some work I am doing. I needed to extend one of the classes to register many of the declarative pipeline methods but I have had good success with it. To write unit tests that assert that say certain builds or things are called you need to be well versed with Groovy and how to mock / assert closure calls with Spock.

domi@icereed.net (JIRA)

unread,
Mar 8, 2017, 6:03:12 AM3/8/17
to jenkinsc...@googlegroups.com

Bill Dennis, would it be possible to share your setup on github? Maybe we can derive a minimalistic example project of it.

bill.dennis@gmail.com (JIRA)

unread,
Mar 8, 2017, 7:05:03 AM3/8/17
to jenkinsc...@googlegroups.com

Dominik Schroeter What I have is in a private repo but I can try and create a simple minimal gradle project in my github account if I can get time this week.

domi@icereed.net (JIRA)

unread,
Mar 8, 2017, 9:03:03 AM3/8/17
to jenkinsc...@googlegroups.com

This would be fantastic!
I think this could be a very good starting point for everybody who wants to ensure quality in a pipeline.

bill.dennis@gmail.com (JIRA)

unread,
Mar 9, 2017, 12:24:05 AM3/9/17
to jenkinsc...@googlegroups.com

Dominik Schroeter OK - here is what I have on my Github: https://github.com/macg33zr/pipelineUnit. It my be a bit raw but should give a good leg-up. Gradle project is unit testing the Jenkinsfile of the Gradle project with Spock and Groovy.

--Bill

dpr@ted.com (JIRA)

unread,
Mar 14, 2017, 8:43:03 PM3/14/17
to jenkinsc...@googlegroups.com

Bill Dennis A big +1 and thank you.  I'm still figuring out the quirks, but this is a massive improvement over the cobbled-together tests I had before.

jglick@cloudbees.com (JIRA)

unread,
Mar 1, 2018, 5:56:11 PM3/1/18
to jenkinsc...@googlegroups.com

Jenkinsfile Runner was recently published, which attacks the problem from the opposite end of the spectrum, literally using JenkinsRule internally.

o.v.nenashev@gmail.com (JIRA)

unread,
Nov 20, 2018, 4:33:05 AM11/20/18
to jenkinsc...@googlegroups.com

The latest version of Jenkinsfile Runner does not use JenkinsRule anymore, but yes it is one of the possible approaches to testing nowadays
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

s.rademacher@live.de (JIRA)

unread,
Mar 5, 2019, 5:27:04 AM3/5/19
to jenkinsc...@googlegroups.com

The Jenkins Pipeline docs only refer to JenkinsPipelineUnit.

On Jenkins World 2018 there was a talk about [jenkins-spock|https://github.com/homeaway/jenkins-spock.|https://github.com/homeaway/jenkins-spock],]

I would like to find out, which library is the best choice, as I am starting to unittest my Jenkins Pipelines just now.

s.rademacher@live.de (JIRA)

unread,
Mar 5, 2019, 7:00:15 AM3/5/19
to jenkinsc...@googlegroups.com
Stefan Rademacher edited a comment on New Feature JENKINS-33925
The Jenkins Pipeline docs only refer to [JenkinsPipelineUnit|https://github.com/jenkinsci/JenkinsPipelineUnit].

On Jenkins World 2018 there was a talk about [jenkins-spock| [ https://github.com/homeaway/jenkins-spock]. |
(see [ https:// github www . youtube. com/ homeaway/jenkins watch?v=4PZ - spock UFBexIE ] ,]  )

I would like to find out, which library is the best choice, as I am starting to unittest my Jenkins Pipelines just now.

jglick@cloudbees.com (JIRA)

unread,
Mar 5, 2019, 8:29:18 AM3/5/19
to jenkinsc...@googlegroups.com

jenkinsfile-runner-test-framework can be used for this purpose, again if you are interested in something closer to an integration test than a unit test.

s.rademacher@live.de (JIRA)

unread,
Mar 5, 2019, 8:54:30 AM3/5/19
to jenkinsc...@googlegroups.com

Thanks! I think unit tests are the best way to test the behavior of my custom steps. I just want to check, if all mocked steps are called as expected.
For that scenario 'jenkins-spock' seems to be a notable alternative to JenkinsPipelineUnit.

If you agree, it might be a good idea to mention it in the docs. Might be interesting, especially for Spock users.

jglick@cloudbees.com (JIRA)

unread,
Mar 5, 2019, 8:57:23 AM3/5/19
to jenkinsc...@googlegroups.com

it might be a good idea to mention it in the docs

If you have field experience doing this successfully, you probably know as much as anyone about the topic, so contributions are welcomed.

o.v.nenashev@gmail.com (JIRA)

unread,
Mar 5, 2019, 8:58:32 AM3/5/19
to jenkinsc...@googlegroups.com

Stefan Rademacher I suggest bringing up the framework comparison topics in https://jenkins.io/sigs/pipeline-authoring/ . It this the venue with the ongoing discussion about Pipeline development tools. CC Andrew Bayer Liam Newman

 

 

 

sparshev@griddynamics.com (JIRA)

unread,
Nov 12, 2019, 7:30:05 PM11/12/19
to jenkinsc...@googlegroups.com

Guys, why we're not testing pipelines directly using JenkinsRule?
I don't think it's worth to support some separated engine to run the pipelines, because they always will have some deviation from the actual engine (for example JenkinsPipelineEngine CPS issue and related CPS issue of find/findAll and the other closure functions that was fixed in workflow-cps-plugin years ago).

I'm checking a way to test like in JenkinsPipelineEngine (same interface) but using JenkinsRule and workflow-cps-plugin and seems it's possible, but probably will require some hooks from CpsScript.invokeMethod (at least I did not found a way without copy to groovy src). When it will work somehow - I will share the changes and hopefully they will work well)
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

sparshev@griddynamics.com (JIRA)

unread,
Nov 12, 2019, 7:33:05 PM11/12/19
to jenkinsc...@googlegroups.com
Sergei Parshev edited a comment on New Feature JENKINS-33925
Guys, why we're not unit- testing pipelines directly using JenkinsRule?
I don't think it's worth to support some separated engine to run the pipelines, because they always will have some deviation from the actual engine (for example JenkinsPipelineEngine [CPS issue|https://github.com/jenkinsci/JenkinsPipelineUnit/issues/159] and related CPS issue of find/findAll and the other closure functions that was fixed in workflow-cps-plugin years ago).


I'm checking a way to test like in JenkinsPipelineEngine (same interface) but using JenkinsRule and workflow-cps-plugin and seems it's possible, but probably will require some hooks from CpsScript.invokeMethod (at least I did not found a way without copy to groovy src). When it will work somehow - I will share the changes and hopefully they will work well)

jglick@cloudbees.com (JIRA)

unread,
Nov 12, 2019, 8:35:27 PM11/12/19
to jenkinsc...@googlegroups.com

Tip: if a script defines a method named, say, node, it will take precedence over a Step of the same name.

sparshev@griddynamics.com (JIRA)

unread,
Nov 12, 2019, 8:58:06 PM11/12/19
to jenkinsc...@googlegroups.com

Jesse Glick I think the good thing of JenkinsPipelineUnit - user specifically chooses what kind of steps is allowed. Unfortunately just redefine still leaves a way to execute some not described step. I think it's a potential issue, so without this strict rule proper unit tests will be impossible.

sparshev@griddynamics.com (JIRA)

unread,
Mar 5, 2020, 1:00:18 AM3/5/20
to jenkinsc...@googlegroups.com

Hi Jesse Glick,

I just completed my research & implementation of JenkinsRule+Groovy-CPS interceptor Jenkinsfile unit testing framework (link) - right now it's integrated to MPL and it's working, but looks not so good as I thought:

  • It uses java path overrides to setup a couple of files to use my invokers: link - not sure it's a good idea... Maybe there is a way to provide some hooks from Jenkins side, wdyt?
  • JenkinsRule produces too much logs - there is a way how to disable the logging? My attempts were unsuccessful: link

If you could provide some advice - that will be really great!

Thank you
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Mar 10, 2020, 4:34:19 PM3/10/20
to jenkinsc...@googlegroups.com

Sergei Parshev Interesting but I doubt maintainers of groovy-sandbox / groovy-cps can take on any testability enhancements—keeping production code running is more than enough work.

sparshev@griddynamics.com (JIRA)

unread,
Mar 10, 2020, 5:54:05 PM3/10/20
to jenkinsc...@googlegroups.com

Jesse Glick Ok, got it. Will work on preparing the changes for groovy-cps unit testing in this intercepting way, if you think it's a good idea.

jglick@cloudbees.com (JIRA)

unread,
Mar 10, 2020, 6:27:17 PM3/10/20
to jenkinsc...@googlegroups.com

I am not maintaining this code so I cannot comment.

sparshev@griddynamics.com (JIRA)

unread,
Mar 10, 2020, 10:16:05 PM3/10/20
to jenkinsc...@googlegroups.com

Ok, just added issue and PR with a sample realization to groovy-cps. Hopefully we will find some way to simplify the execution of JenkinsRule-based unit tests for the shared libraries.
Reply all
Reply to author
Forward
0 new messages