Generic DSL support for (almost) all plugins

273 views
Skip to first unread message

Daniel Spilker

unread,
Apr 1, 2016, 5:28:53 AM4/1/16
to job-dsl...@googlegroups.com
Hi everyone!

Some days ago Kohsuke released a new plugin which can be used by DSL plugins to generate DSL syntax from metadata provided by Descriptors[2]. I started to explore how Job DSL can use that plugin to complement the existing Job DSL ContextExtensionPoint and opened a pull request [3] to show the progress.

I'm very happy with the intermediate results. Let's take some of the currently open pull requests [4][5][6][7][8] as an example. The following DSL will work by just applying my changes and it does neither require specific support for those plugins in the DSL nor any changes in the plugins themselves.

job('example') {
  steps {
    phingBuilder {
      name('Phing 1.8')
      useModuleRoot(false)
      properties('KEY=VALUE')
      targets('test')
      options('--debug')
      buildFile('dir1/build.xml')
    }
    exporterBuilder()
    unity3dBuilder {
      unstableReturnCodes('2,3')
      argLine('-batchmode')
    }
    msBuildBuilder {
      msBuildName('MSBuild 1.8')
      msBuildFile('dir1/build.proj')
      cmdLineArgs('check')
      buildVariablesAsProperties(true)
      continueOnBuildFailure(true)
      unstableIfWarnings(true)
    }
  }
  publishers {
    logParserPublisher {
       projectRulePath('unitybuilder.parser')
       useProjectRule(true)
       unstableOnWarning(true)
       failBuildOnError(true)
    }
  }
}

The DSL syntax can be derived from the config XML. Field/property names can be taken from the XML directly and class names are shortened by using the uncapitalized class name only. The following config XML corresponds to the phingBuilder DSL above:

<project>
  <builders>
    <hudson.plugins.phing.PhingBuilder>
      <buildFile>dir1/build.xml</buildFile>
      <name>Phing 1.8</name>
      <targets>test</targets>
      <properties>KEY=VALUE</properties>
      <useModuleRoot>false</useModuleRoot>
      <options>--debug</options>
    </hudson.plugins.phing.PhingBuilder>
  </builders>
</project>

The syntax resembles the built-in DSL methods except for the step and publisher names where we would for example use phing instead of phingBuilder. But that can be solved by adding the new @Symbol annotation to the plugin, which is only a minimal change to the plugin. See [1] for details. As an example I added some symbols to the Gerrit Trigger plugin to get nicer method names for the trigger events, see [9].

So we will gain immediate support for almost all of the 1000+ plugins. Some will not work, like the Jython plugin, because they do not use the Describable/Descriptor pattern [2] correctly or use an older variant. But even those can be fixed easily.

To avoid naming collisions, the following order of precedence is implemented for resolving methods (highest to lowest):
* built-in DSL syntax
* methods provided by the ContextExtensionPoint
* symbolic names provided by the @Symbol annotation
* uncapitalized class names (e.g. msBuildBuilder for hudson.plugins.msbuild.MsBuildBuilder)

The only drawback is that all restrictions of the ContextExtensionPoint also apply to the generic DSL, e.g. no docs in the API viewer, no IDE support and limited testability. I would like to address these restrictions in the near future, e.g. we have been discussing [10] how to integrate the API viewer into Jenkins so that it can use runtime information to generate the docs.

What do you think?

Daniel

Victor Martinez

unread,
Apr 1, 2016, 3:19:23 PM4/1/16
to job-dsl-plugin
+1000

I really like this; I've found a bit tedious and slow process to support new plugins right now. It'll definitely help to standardise those DSL easily!

I've got a couple of questions:
a) How will it work between different versions of the same plugin? 
b) For instance, when a particular field/property is either deprecated or new, will it work straight away?

Awesome work!!

Daniel Spilker

unread,
Apr 4, 2016, 4:01:04 AM4/4/16
to job-dsl...@googlegroups.com
On Fri, Apr 1, 2016 at 9:19 PM, Victor Martinez <victormar...@gmail.com> wrote:
a) How will it work between different versions of the same plugin?

That is not a problem. The metadata is provided by the currently installed version of a plugin and only one version can be installed.
 
b) For instance, when a particular field/property is either deprecated or new, will it work straight away?

The behavior is similar to the configure block. Deprecation is not a problem, since the field/property should still be there. But you will not get a deprecation warning in the log. When a field/property has been removed, the DSL script will fail. Here the configure block behaves different since the DSL script would run successfully and you would get a "You have data stored in an older format and/or unreadable data." warning on the "Manage Jenkins" page, which may be unnoticed for a while. New fields will be populated with their default value, so it should be no problem.

Daniel

Victor Martinez

unread,
Apr 9, 2016, 3:54:56 PM4/9/16
to job-dsl-plugin
great!

another question, does it make sense to release that feature as 2.X version?

Daniel Spilker

unread,
Apr 11, 2016, 4:04:49 AM4/11/16
to job-dsl...@googlegroups.com
Yesterday I added support for optional and required parameters. A parameter is considered required when defined by a @DataBoundConstructor and optional when defined by a @DataBoundSetter. See https://github.com/jenkinsci/structs-plugin/blob/structs-parent-1.1/plugin/src/main/java/org/jenkinsci/plugins/structs/describable/DescribableParameter.java#L82-L84. Unfortunately some plugins primarily use @DataBoundConstructor, so more parameters than necessary must be specified. But that can be fixed easily in plugins by migrating to @DataBoundSetter for optional parameters. And introducing required parameters at a later stage will break a lot of scripts, so the generic DSL will check them from the beginning.

I think the generic DSL is ready to be merged. But it would be great if more people would do some testing with their favorite unsupported plugins and post the results to the PR. I already did some testing for several plugins, see https://github.com/jenkinsci/job-dsl-plugin/pull/816#issuecomment-207347635.

I thought about having a 2.0 release. The generic DSL is an add-on, not a breaking change. We could do the 2.0 like Jenkins mainly for marketing reasons. But then we must have support for generic DSL in the API viewer and must have IDE support, so that we achieve an excellent user experience. Or we save the 2.0 for breaking changes. I digged through JIRA and there are IMHO no feature requests or bug reports which would justify a breaking change. In context of the generic DSL I can imagine a 2.0 release where we aggressively reduce the built-in DSL support for plugins and rely on the generic DSL instead. But then again be need to have an API viewer, IDE support and most likely support for unit testing as well. Or at least have some built-in dry run mode for testing, see https://issues.jenkins-ci.org/browse/JENKINS-27182.

Daniel

--
You received this message because you are subscribed to the Google Groups "job-dsl-plugin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to job-dsl-plugi...@googlegroups.com.
To post to this group, send email to job-dsl...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/job-dsl-plugin/3623e410-9d64-4780-b268-4fdbd5091c72%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jeremy Marshall

unread,
Apr 22, 2016, 8:02:01 AM4/22/16
to job-dsl-plugin
 Hi

I love the flexibility this provides. I was looking at adding in the Matrix Groovy Execution Strategy (which I wrote). I spent a while wondering why it wasn't working - mainly because I was looking at the git diff output. I totally missed that the structs plugin is required especially as I was running it directly from gradle. I don't know if there is any way to warn people, or possibly download the struct plugin at build time as the failure is just a missing methods with closure parameters.

As for the execution strategy, this is what I got, which matches the example in its wiki

matrixJob('example') {
    axes {
        text('axis1', 'a', 'b', 'c')
        text('axis2', 'x', 'y', 'z')
    }
    steps {
        shell('sleep 2')
    }
    executionStrategy{
      groovyScriptMES {
        secureScript {
        script('''
combinations.each {
    if(it.axis2 == 'z')
        return
    result[it.axis2] = result[it.axis2] ?: []
    result[it.axis2] << it
}
result
''')
 sandbox(false)
        }
        scriptFile('')
 scriptType('script')
      }
    }
}

I think it will conflict with the 'classic' execution strategy which is mixed into the matrix job code, such as keystone builds and run sequentially but then these can be modified through generic support.

I'll do more work with it and add a PR if its worthwhile

Jeremy
Reply all
Reply to author
Forward
0 new messages