Generate/Execute Pipeline from plugin programatically?

[Matt Q]

Is there an existing Java API for generating Jenkins Pipelines programmatically? 

I tried to execute a simple declarative pipeline as a Groovy Postbuild step:

#!/usr/bin/env groovy
pipeline {
agent any

stages {

stage('Test Stage') {

steps {

echo '----- In Test Stage------'

}

}

}

}

in Java code as follows:

import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript;
import org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder;
import hudson.model.Project;

...

String pipeScript = new String("script above as a String");
SecureGroovyScript secureScript = new SecureGroovyScript(pipeScript, /*sandbox*/false, /*classpath*/null);
GroovyPostbuildRecorder groovy = new GroovyPostbuildRecorder(secureScript, /*behaviour*/2, /*matrix parent*/false);
myProject.getPublishersList().add(groovy);

This results in the stacktrace:

ERROR: Failed to evaluate groovy script.
groovy.lang.MissingMethodException: No signature of method: Script1.pipeline() is applicable for argument types: (Script1$_run_closure1) values: [Script1$_run_closure1@111b5ac]
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:58)
	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:75)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:68)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
	at org.kohsuke.groovy.sandbox.impl.Checker$checkedCall.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194)
	at Script1.run(Script1.groovy:2)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.run(GroovySandbox.java:139)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript.evaluate(SecureGroovyScript.java:161)
	at org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder.perform(GroovyPostbuildRecorder.java:362)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
	at hudson.model.Build$BuildExecution.post2(Build.java:185)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
	at hudson.model.Run.execute(Run.java:1766)
	at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:73)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Build step 'Groovy Postbuild' marked build as failure
Finished: FAILURE

I assume this is because the Domain Specific Language used for Pipelines is similar to but not actually Groovy. Is there an existing plugin or API to accomplish this?

Thanks,

-Matt

[Jesse Glick]

On Fri, Jan 25, 2019 at 5:10 PM Matt Q <matthew....@gmail.com> wrote:
> Is there an existing Java API for generating Jenkins Pipelines programmatically?

Not in the way you are looking for.

> I tried to execute a simple declarative pipeline as a Groovy Postbuild step:

You cannot. Why would you want to do that?

[Artur Szostak]

If you are actually looking for a way to populate Jenkins with Pipeline type jobs in an automated manner, then the best bet is to use the Job DSL plugin. i.e. the Job DSL and Pipeline combination will allow to have everything in SCM and you can bootstrap the full job creation from SCM. The Pipeline type job alone does not allow this. As far as I'm concerned that was a major oversight.

Best regards.

Artur


________________________________________
From: jenkin...@googlegroups.com <jenkin...@googlegroups.com> on behalf of Matt Q <matthew....@gmail.com>
Sent: 25 January 2019 22:38:12
To: Jenkins Developers
Subject: Generate/Execute Pipeline from plugin programatically?

Is there an existing Java API for generating Jenkins Pipelines programmatically?

I tried to execute a simple declarative pipeline as a Groovy Postbuild step:

#!/usr/bin/env groovy
pipeline {
agent any
stages {
stage('Test Stage') {
steps {
echo '----- In Test Stage------'
}
}
}
}


in Java code as follows:

import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript;
import org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder;
import hudson.model.Project;

...

String pipeScript = new String("script above as a String");
SecureGroovyScript secureScript = new SecureGroovyScript(pipeScript, /*sandbox*/false, /*classpath*/null);
GroovyPostbuildRecorder groovy = new GroovyPostbuildRecorder(secureScript, /*behaviour*/2, /*matrix parent*/false);
myProject.getPublishersList().add(groovy);


This results in the stacktrace:


ERROR: Failed to evaluate groovy script.

groovy.lang.MissingMethodException<http://stacktrace.jenkins-ci.org/search?query=groovy.lang.MissingMethodException>: No signature of method: Script1.pipeline() is applicable for argument types: (Script1$_run_closure1) values: [Script1$_run_closure1@111b5ac]
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:58)<http://stacktrace.jenkins-ci.org/search/?query=org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap&entity=method>

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com<mailto:jenkinsci-de...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/8acadf56-6c3b-40af-8567-f46c62fe8abb%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-dev/8acadf56-6c3b-40af-8567-f46c62fe8abb%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Mon, Jan 28, 2019 at 10:06 AM Artur Szostak <aszo...@partner.eso.org> wrote:
> the Job DSL and Pipeline combination will allow to have everything in SCM and you can bootstrap the full job creation from SCM. The Pipeline type job alone does not allow this.

Well…for the specific, common use case of creating one job per repo
and keeping branch-specific build instructions, “Pipeline-as-code” ~
`workflow-multibranch` plugin offers a simple solution. Clearly this
does not cover more complicated scenarios. JENKINS-37220 has
discussion.

[Matt Q]

Thanks for the replies.  I am able to create pipeline jobs by modifying an existing config.xml and posting it to the command line interface:so I am exploring that route. I was trying to avoid using the XML file but is it safe to assume that the config.xml files will be backwards compatible if we upgrade to a newer version of Jenkins in the future?

Similarly, this command works from the command line but not when executed from Java code:

java.nio.file.FileSystemException: C:\Users\mqu\.jenkins\jobs\AnotherNewPipeline\config.xml: The process cannot access the file because it is being used by another process.

This is unexpected since the file is being created, not deleted, and I mention it because sounds similar to this issue.

Any ideas/workarounds would be appreciated. In the meantime I am looking into leveraging the Jenkins API Client for Java.

[Jesse Glick]

On Thu, Feb 7, 2019 at 3:02 PM Matt Q <matthew....@gmail.com> wrote:
> I was trying to avoid using the XML file but is it safe to assume that the config.xml files will be backwards compatible if we upgrade to a newer version of Jenkins in the future?

Yes, we would make every attempt to retain `config.xml` compatibility,
since failing to due so would cause a critical problem for essentially
every Jenkins user.

> this command works from the command line but not when executed from Java code:
>
> java.nio.file.FileSystemException: C:\Users\mqu\.jenkins\jobs\AnotherNewPipeline\config.xml: The process cannot access the file because it is being used by another process.

Without knowing what the “Java code” in question was, I cannot help.
There is a supported set of (Java) APIs in Jenkins for creating new
jobs from various forms of input.

[Artur Szostak]

>> I was trying to avoid using the XML file but is it safe to assume that the config.xml
>> files will be backwards compatible if we upgrade to a newer version of Jenkins in the future?
>
> Yes, we would make every attempt to retain `config.xml` compatibility,
> since failing to due so would cause a critical problem for essentially
> every Jenkins user.

Sorry, but that it not quite true in reality. As an example, I have performed a fresh installation of Jenkins
about 4 months ago. Within that time, when I look at the management page for plugins is see 16
messages of the following form:

"Warning: This plugin requires dependent plugins be upgraded and at least one of these dependent plugins
claims to use a different settings format than the installed version. Jobs using that plugin may need to be
reconfigured, and/or you may not be able to cleanly revert to the prior version without manually restoring
old settings. Consult the plugin release notes for details."

This indicates to me that the XML format has likely changed in an incompatible manner.

I have also maintained a customised jenkins-job-builder version in the past and experienced a number
of cases where I was forced to update the generated XML format due to changes in Jenkins plugins.

It may be a goal for some of the developers to maintain backwards compatibility, but it is clearly not
being achieved. My experience indicates that for any sizeable and non-trivial Jenkins setup you cannot
expect long term backwards compatibility for all the plugins. You will run into problems sooner rather
than later, which is why regular backups of all the configurations files in Jenkins are so important. That
is the only certain way to rollback a plugin upgrade. The built in mechanism in the Jenkins GUI are brittle.

[Jesse Glick]

On Tue, Feb 12, 2019 at 4:41 AM Artur Szostak <aszo...@partner.eso.org> wrote:
> when I look at the management page for plugins is see 16
> messages of the following form:
>
> "Warning: This plugin requires dependent plugins be upgraded and at least one of these dependent plugins
> claims to use a different settings format than the installed version. Jobs using that plugin may need to be
> reconfigured, and/or you may not be able to cleanly revert to the prior version without manually restoring
> old settings. Consult the plugin release notes for details."
>
> This indicates to me that the XML format has likely changed in an incompatible manner.

Likely not. First of all, it says _may_ need to be configured. In fact
in most cases, it is only the second clause which applies—that a
`config.xml` saved with the _new_ plugin version may not be properly
read by the _old_ plugin version. Which is not a concern for Matt’s
usage. In any case, this is merely a precautionary notice; the details
in the plugin changelog will say what you actually need to pay
attention to.

> I have also maintained a customised jenkins-job-builder version in the past and experienced a number
> of cases where I was forced to update the generated XML format due to changes in Jenkins plugins.

In such cases please file bug reports for the affected plugins.
Obviously the quality of plugins varies according to how diligent
their maintainers are about this topic. At any rate, compatibility at
the XML level remains much more likely than compatibility using some
internal Java API.

> regular backups of all the configurations files in Jenkins are so important.

That is certainly good advice.