Exposing a step as global variable (or calling it implicitly)

[thomas.w...@de.amadeus.com]

Hi all,

I have written a custom step (let's call it "foo") that allows pipeline authors
to configure aspects of their build. The step can be used with our without a
body:

# applies the config to the executed block
foo(config: ...) {
}

# applies the config to the current block
stage("s") {
    def config = foo()
    config.setting = 1

    # or
    foo().setting = 1
}

This works very nicely.
The configuration is expected to be called many times in a pipeline definition
and for this reason I would like to increase the ergonomics of the syntax
(even more).

# desired syntax
stage("s") {
    foo.setting = 1
}

The logic in the "foo" step needs access to the current StepContext, so I can't
implement it completely as a global variable, as those don't have access to the
context.

Ideas:
a) implement it with a global variable, that directly call the step via the CpsScript.
   However the CPS engine always resolves symbol look-ups by looking at global
   variables, so my global variable calls itself.
   (I have a PoC that allows GlobalVariables to be exempt from method look-ups)
b) Allow steps to declare that they also want to be exposed as a global
   variable, which when accessed will call the Step without arguments?


Do you think this functionality would be in scope of Jenkins?

Thanks,
Thomas

[Jesse Glick]

On Tue, Mar 5, 2019 at 1:07 PM <thomas.w...@de.amadeus.com> wrote:
> The step can be used with our without a body

Huh? That is not possible. I given `StepDescriptor` specifies whether
it `takesImplicitBlockArgument` or not. If you want two syntaxes, you
need two step names.

> # applies the config to the current block

There is no such concept in Pipeline.

> I would like to increase the ergonomics of the syntax
>

> # desired syntax
> stage("s") {
> foo.setting = 1

This is not possible using steps.

> implement it with a global variable

As noted in the `GlobalVariable` Javadoc, you should avoid this API.

> Do you think this functionality would be in scope of Jenkins?

No. I would stick to the block-scoped step and leave it at that.

[Thomas Weißschuh]

Hi Jesse,

thanks for the quick response!

On Tue, Mar 05, 2019 at 02:21:03PM -0500, Jesse Glick wrote:
> On Tue, Mar 5, 2019 at 1:07 PM <thomas.w...@de.amadeus.com> wrote:
> > The step can be used with our without a body
>
> Huh? That is not possible. I given `StepDescriptor` specifies whether
> it `takesImplicitBlockArgument` or not. If you want two syntaxes, you
> need two step names.

I think it is:

From the docs of StepDescriptor:

/**
* Return true if this step can accept an implicit block argument.
* (If it can, but it is called without a block, {@link StepContext#hasBody} will be false.)
* @see StepContext#newBodyInvoker()
*/

Also official steps like "stage" accept both forms.
(Yes, the form without step is deprecated, but for other reasons)

Also this part of the plugin already works quite well.

>> # applies the config to the current block
> There is no such concept in Pipeline.

To be more precise it refers to the datastructure associated with
the current FlowNode.
(I can elaborate on the specifics if wanted)

>> I would like to increase the ergonomics of the syntax
>>
>> # desired syntax
>> stage("s") {
>> foo.setting = 1
>
> This is not possible using steps.
>
>> implement it with a global variable
>
> As noted in the `GlobalVariable` Javadoc, you should avoid this API.

Yes, I would like to avoid it.

OTOH I don't think it is *that* bad in case of this specific plugin:

* The plugin is *very* generic.
(it only looks at the build as a series of nodes/steps, without any
assumptions about their content)
* It does not have to be compatible with declarative pipeline.
(It would be better for its ergonomics to implement a dedicated
extensionpoint for declarative pipelines)
* The variable is only syntactic sugar that is meant to save two braces in Cps
pipelines.
* The groovy code it executes is literally:

public Object getValue(@Nonnull CpsScript script) throws Exception {
return script.evaluate("steps.foo()");

}

>> Do you think this functionality would be in scope of Jenkins?
>
> No. I would stick to the block-scoped step and leave it at that.

What do you think of having steps (optionally, decided by the step author)
available as something like global variables?
This would allow plugin authors to provide this functionality
(for which they currently do implement GlobalVariables)
without the disadvantages and without a tight dependency on Cps.

Thanks,
Thomas

[Jesse Glick]

On Wed, Mar 6, 2019 at 3:59 AM Thomas Weißschuh
<thomas.w...@de.amadeus.com> wrote:
> * (If it can, but it is called without a block, {@link StepContext#hasBody} will be false.)
>

> Also official steps like "stage" accept both forms.

Ah forgot that this was introduced for the `stage` step trick. My
advice had actually been to keep `stage` as is and introduce
block-scoped `label`. Best to not use this facility and keep each step
either exclusively block-scoped or non-block-scoped.

> The plugin is *very* generic.
> (it only looks at the build as a series of nodes/steps, without any
> assumptions about their content)

So what is it doing, exactly? This smells like something that (if
needed at all) should be built into the syntax, not done as a `Step`.

> What do you think of having steps (optionally, decided by the step author)
> available as something like global variables?
> This would allow plugin authors to provide this functionality
> (for which they currently do implement GlobalVariables)
> without the disadvantages and without a tight dependency on Cps.

Even if there were not a literal compile-and-link dependency on
`workflow-cps` the semantics would be inherently tied to details of
that model.

[Thomas Weißschuh]

On Wed, Mar 06, 2019 at 12:25:10PM -0500, Jesse Glick wrote:

> On Wed, Mar 6, 2019 at 3:59 AM Thomas Weißschuh wrote:
>> * (If it can, but it is called without a block, {@link StepContext#hasBody} will be false.)
>>
>> Also official steps like "stage" accept both forms.
>
> Ah forgot that this was introduced for the `stage` step trick. My
> advice had actually been to keep `stage` as is and introduce
> block-scoped `label`. Best to not use this facility and keep each step
> either exclusively block-scoped or non-block-scoped.

Is there a chance of it breaking in the future, now that it has been published
and documented?

>> The plugin is *very* generic.
>> (it only looks at the build as a series of nodes/steps, without any
>> assumptions about their content)
>
> So what is it doing, exactly? This smells like something that (if
> needed at all) should be built into the syntax, not done as a `Step`.

Now that you are asking :-)

The plugin exposes the internal processes of Jenkins
(runs, queues, workflows) to a general distributed tracing system.
The goal is to have a complete, end-to-end linked trace over the whole CI/CD pipeline,
including the SCM system, all the different steps inside Jenkins, all the steps
of the build systems executed by Jenkins, asynchronous processes triggered by
the build, executors spun up for the build, etc...

You may remember our discussion on this list about having per step environment
variables and the resulting implementation of the StepEnvironmentContributor
Extensionpoint. This was for the same plugin and allows the plugin to
communicate with other plugins in maven, SonarQube, docker etc.

The goal is to have increased transparency for the users, better monitoring and accounting.

Currently inside Jenkis Pipelines all Steps are recorded (in addition to the
other parts), enriched with information from within Jenkins and added to the
tracing system.
For this reason it keeps a database of currently running steps and their
associated tracing datastructure
(care is taken not to leak memory references to core Jenkins objects).

However there are cases where the information available out of the box is not
sufficient.

For this usecase the custom step provides two functionalities, when called with
or without a block.

It can be used to "synthesize" a step for the tracing system, which is useful
if there is no native step with the desired scope available at the moment.
For example when a new stage is not desired, or for use in a global
pipeline library, as steps from libraries do not end up as new FlowNodes and so
are not traced explicitly out of the box.

trace(tags: ["command.type": "something"]) {
sh '...'
}

The other syntax (without a block) allows users and pipeline library authors to
enrich the currently active tracing scope with custom information.
This helps to keep the amount of tracing scopes smaller, also it allows for a
better search experience.

# all nested steps modify the tags of the "stage" scope
stage("someStage") {
# same way as above
trace(tags: ["foo": 1])

# it also returns an object that can be used from groovy
trace().setTag("foo", 1)
trace().tags["bar"] = 2
}

For this last part I wanted to implement a shorthand syntax without the
required braces.

Can you elaborate on "building something into the syntax"?
Is it something I can already do?

>> What do you think of having steps (optionally, decided by the step author)
>> available as something like global variables?
>> This would allow plugin authors to provide this functionality
>> (for which they currently do implement GlobalVariables)
>> without the disadvantages and without a tight dependency on Cps.

> Even if there were not a literal compile-and-link dependency on
> `workflow-cps` the semantics would be inherently tied to details of
> that model.

As it would only be syntactic sugar, this would be fine for my particular
usecase. If it does not fit the general semantics then I will have to do
without.

The other case I mentioned, where GlobalVariables override Steps during lookup
does actually match the behaviour of normal Groovy. Somehow I got my previous
tests there wrong.

So as for the triply overloaded symbol, I will have to do with the current state.

[Jesse Glick]

On Thu, Mar 7, 2019 at 9:29 AM Thomas Weißschuh
<thomas.w...@de.amadeus.com> wrote:
>> Best to not use this facility and keep each step
>> either exclusively block-scoped or non-block-scoped.
>
> Is there a chance of it breaking in the future, now that it has been published
> and documented?

I suppose not, but better not use it regardless. It was added to
support this one hack.

> It can be used to "synthesize" a step for the tracing system, which is useful

> if there is no native step with the desired scope available at the moment. […]

>
> trace(tags: ["command.type": "something"]) {
> sh '...'
> }

Other than `Map`s not being well supported by `structs` (so for
example that `tags` syntax will be illegal in Declarative Pipeline and
would cause headaches for `Snippetizer`), this much is fine—it is akin
to `stage`, except for tags rather than display labels.

> The other syntax (without a block) allows users and pipeline library authors to

> enrich the currently active tracing scope with custom information. […]

>
> # all nested steps modify the tags of the "stage" scope
> stage("someStage") {
> # same way as above
> trace(tags: ["foo": 1])

Would be OK if you just picked a different step name (and, as above,
avoided types not supported by Jenkins databinding).

> # it also returns an object that can be used from groovy
> trace().setTag("foo", 1)
> trace().tags["bar"] = 2

Avoid these Groovy-specific idioms.

> Can you elaborate on "building something into the syntax"?
> Is it something I can already do?

Not without patching fundamental plugins, which would not be wise.