Pipeline as YAML GSoC project and Declarative

[Andrew Bayer]

Hi -

First of all, great work by Abhishek on https://jenkins.io/blog/2018/07/17/simple-pull-request-plugin/ - kudos! I need to review the code in more detail and suggest some possible streamlining/improvements in the interaction with Declarative proper, but that's just nitpicking stuff. =)

The main reason I'm sending this email is that we're probably going to be doing some form of Declarative YAML syntax as a native part of Declarative itself in the future - no specific details, plans, or timeline yet, but I'd expect a JEP to land in the fall once some other work of mine wraps up. This'll be similar from an authoring perspective to Abhishek's work, but not exactly - since it'll be pure Declarative (with the YAML being parsed into the same intermediate form we parse the Groovy pipeline {} block and JSON used with the Blue Ocean Pipeline Editor into, and then we'll transform that intermediate form into the actual execution AST at runtime, just as we do with the Groovy pipeline {} block), it'll cover the full range of Declarative syntax (except for, probably, script {} blocks and when { expression {} }). A Declarative YAML file will be able to be converted to a Groovy Jenkinsfile and vice versa - they'll really just be two different syntaxes for the exact same backend and execution engine.

This means that the eventual native Declarative YAML syntax won't support things that aren't part of Declarative proper - at a glance, that would mean that the PR configuration, reports, and archive management aspects of Abhishek's work wouldn't be part of the native Declarative YAML syntax, since they aren't Declarative directives. There also might be syntax detail differences - I haven't done any work yet on designing the native Declarative YAML syntax, so I don't yet know how we'll represent credentials, etc, and I'll try to have that design match Abhishek's as closely as I can, but for technical or usability reasons, there may need to be divergences.

So now I'm thinking some on how we can have compatibility for a YAML file between Abhishek's work and the eventual native Declarative implementation. Regrettably, I haven't spent enough time going through the code to see if there's a clean way to fit the PR/reports/archive functionality into the Declarative syntax model, but I have a feeling there isn't a good way to do so - since they result in autogenerated stages, meaning they're not Declarative options, wrappers, etc. I'm also a bit nervous over the use of Jenkinsfile.yaml as the "marker" file - that'd mean that unless a YAML file written for Abhishek's work could be read and handled by the eventual native Declarative implementation, we'd need a different name for the Declarative YAML file, creating which could be confusing for users.

I don't have a solid plan/suggestion/etc for what to do here, I'll confess. It might make the most sense for Abhishek's work to use a less "official" sounding filename than Jenkinsfile.yaml - perhaps a name that shows that file is going to be read and used by the Simple Pull Request plugin's implementation. Something similar in terms of display names would probably also make sense - if this isn't going to be part of the "core" Pipeline suite, its naming/terminology/etc should probably make its purpose and origin more clear?

Again, I have nothing but praise for your work, Abhishek, and I am really sorry to be requesting changes like this. I'm just trying to think about futureproofing to prevent conflicting behavior, user confusion, etc in the future.

A.

[Andrew Bayer]

Oh, just remembered something - https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/260 is a work-in-progress on an extension point for plugins to use to contribute directives to Declarative. If the PR/reports/archives stuff in Abhishek's work was nested within a new top-level directive, it would definitely be possible down the road for the Simple Pull Request plugin to contribute a directive that, at the time of transformation from the intermediate model to the runtime model, added stages for PR/reports/archives (depending on configuration, obviously), which would get rid of the biggest compatibility difference between Abhishek's syntax and the eventual native Declarative syntax. If I can persuade y'all to do that (maybe have the new directive's key in the YAML be something distinct to this plugin, like "simplePullRequest: ..."), then when the native Declarative implementation is ready to go, I can help Abhishek refactor the YAML parsing/execution aspects out of this plugin and instead hook it in with Declarative directly. Then we could still keep the same filename (though I'd suggest Jenkinsfile.yml rather than Jenkinsfile.yaml - the .yml file extension seems to be more commonly used) and once a user upgrades to the Declarative version that introduces native YAML syntax and the updated version of Simple Pull Request that switches to building on top of Declarative's native YAML, things should just work (well, along with some migration logic for autoconverting from the Simple Pull Request flow definition/project factory implementations to the eventual Declarative equivalents).

Thoughts? 

A.

[Jeff Knurek]

Andrew, thanks for the input. 

If I'm not mistaken, we are still working on a name change of the plugin, but I think renaming Jenkinsfile.yaml to something more plugin specific shouldn't be a problem. Also, the user has the ability to change the name of the yaml file to anything they like, it just happens to default to Jenkinsfile.yaml (so this is mostly just a documentation change, and doesn't impact the current/future usability).

I think the idea of using the future extension point to implement the directives directly to Declarative is a great idea. I'd suggest that be added as a future task, that if possible could be added to this last phase of the GSoC project, but that of course depends on your PR to be merged.

Regarding the extension of .yaml and .yml, I've actually experienced that in regards to k8s, and especially Helm, the .yaml extension is more common. But I think the most important thing is that anything implemented should support either extension. 

/Jeff

[Andrew Bayer]

Hi Jeff -

In re: the directive - that PR probably won't be merged in the immediate future, but I think it'd make sense to futureproof a bit - given that the configuration, archiveArtifacts, and reports top-level keys in the YAML are the only ones that don't correspond directly to an existing Declarative directive (i.e., stages, agent, tools, environment, post), it'd be great if those could all be nested under a single top-level key, like:

simplePullRequest:

    archiveArtifacts: ...

    configuration: ...

    reports: ....

with simplePullRequest parsing however works best for y'all. I just want to make sure we don't end up with this plugin going out of alpha/beta with a syntax that will definitely not be compatible with the eventual native Declarative YAML so that users won't hit compatibility problems in the future.

A.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6d4694b3-8167-4a12-936f-95295e0b830d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Abhishek Gautam]

Hi Andrew,

Thanks for taking time and providing all the suggestions.

[Jesse Glick]

On Wed, Jul 18, 2018 at 10:03 AM Andrew Bayer <andrew...@gmail.com> wrote:

> the full range of Declarative syntax (except for, probably, script {} blocks and when { expression {} })

I suppose if you wanted it would be easy enough to say

script: |
if (…) {…}

> the PR configuration, reports, and archive management aspects of Abhishek's work wouldn't be part of the native Declarative YAML syntax, since they aren't Declarative directives

One of my comments from the last presentation was that the PR
configuration stuff really makes more sense as a separate feature,
perhaps in a plugin depending on `github-branch-source` defining a
`JobProperty`, since it does not seem to be specific to Declarative
(or perhaps even Pipeline). This also seemed to be true of some other
features like Git pushes. They may cohere as a higher-level user
experience (i.e., in a documented usage pattern), but architecturally
it would be better to divide these things into independent plugins
that can be composed naturally with other Jenkins features.

> a less "official" sounding filename than Jenkinsfile.yaml

BTW I would suggest the more conventional naming pattern of `jenkins.yaml`, etc.

[nicolas de loof]

Be careful jenkins.yaml is the default name used by configuration as code for master's configuration.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0yfJgRdNwcckPbjDtKTpVrLe4fVmS7GMfuYuxJtVvWEw%40mail.gmail.com.

[martinda]

On Thursday, July 19, 2018 at 4:56:24 PM UTC-4, nicolas de loof wrote:

Be careful jenkins.yaml is the default name used by configuration as code for master's configuration.

Yes, a few non-conflictual names have names have been floated in the gitter chat, so we're pretty sure we'll be safe here. We're naming the pluing yaml file after the name of the plugin itself, so it won't be a generic broad name anymore.

Martin

[martinda]

Hi Andrew,

Would you be available to participate in a hangout with Abhishek and his mentors on Aug 1 (or around that date)? We'd like to know a bit more on how we could proceed WRT your suggestions on extracting these features into extensions.

Thanks in advance,

Martin

[Andrew Bayer]

Sure thing - I'm in EDT, and probably won't be available from like noon to 3pm, but otherwise am free.

A.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ebbfb6f6-9b73-4dd5-bd48-4e61c64cd205%40googlegroups.com.