Modifying Environment variables for each workflow step from a plugin

[thomas.w...@de.amadeus.com]

Is there a way to modify the value of an environment variable for a single step
or block in Pipeline from a GraphListener?
The value would have to change for each node. (For the background read on)

So far I have tried to add a custom EnvironmentContributingAction in
GraphListener.onNewHead(). But those actions seem to be only evaluated before
each build in AbstractBuild.getEnvironment().

I also tried EnvActionImpl.forRun((Run) flowNode.getExection().getOwner().getExecutable())
in GraphListener.onNewHead() which only changes it for the *next* step, as
apparently the StepContext has already been set up.
(It is probably also not great in case of parallel jobs)

EnvironmentContributor is also documented to affect the whole build.

The best way forward seems to somehow get hold of the StepContext from
GraphListener.onNewHead() but so far I could not find a way to do so.


I would be grateful for any pointers.


Background:

I am trying to integrate an OpenTracing [0] tracer into Jenkins.
It will record executions for all Pipeline runs / queueing processes etc...
The tracing inside Jenkins works (mostly) fine so far.
Now the issue is to propagate the tracing context from within Jenkins to the
executed steps. in this case Maven for which a similar plugin was created.
The probably best way of propagating this information is via environment
variables.
Obviously the value of this environment variable will have to change often
during the course of a single build.
It would also be nice if the plugin could work without any changes to the
Pipeline definition itself.
(It is planned to add optional custom steps/methods)

PS: it seems the usage of env vars with a dash ("-") in their keys is broken.
Even when set directly via the Java APIs, they don't show up in the "sh"
pipeline step.

Thanks,
Thomas

[0] http://opentracing.io/

[Jesse Glick]

On Mon, Apr 30, 2018 at 10:09 AM, <thomas.w...@de.amadeus.com> wrote:
> Is there a way to modify the value of an environment variable for a single
> step
> or block in Pipeline from a GraphListener?
> The value would have to change for each node.

I do not think so. As of JENKINS-42499, an `EnvironmentContributor`
could change a value dynamically. This is not scoped to a step; it
just means that if and when a step requests access to environment
variables, your plugin will be asked anew for contributions.

> before
> each build in AbstractBuild.getEnvironment().

`Run.getEnvironment` I guess you mean. Pipeline builds are not `AbstractBuild`s.

> I also tried EnvActionImpl.forRun((Run) flowNode.getExection().getOwner().getExecutable())

I am not sure what you are trying to accomplish here, but stop. Do not
declare a (`compile`-scoped) dependency on `workflow-cps` at all. You
may use `workflow-step-api` and, if necessary, `workflow-api` as
dependencies.

> Now the issue is to propagate the tracing context from within Jenkins to the
> executed steps. in this case Maven for which a similar plugin was created.
> The probably best way of propagating this information is via environment
> variables.
> Obviously the value of this environment variable will have to change often
> during the course of a single build.

Just define a step which would provide the current tracing context
(whatever that is) as its return value. That will work fine in
Scripted. If you require Declarative compatibility, your best bet is
to define a block-scoped step which would define some environment
variable(s) for its nested steps. That is my best recommendation based
on my very limited grasp of what it is you are trying to do.

> it seems the usage of env vars with a dash ("-") in their keys is
> broken.
> Even when set directly via the Java APIs, they don't show up in the "sh"
> pipeline step.

https://unix.stackexchange.com/a/23714/26736

[thomas.w...@de.amadeus.com]

Hi Jesse,

thanks for the quick response!

On Tuesday, May 1, 2018 at 3:14:48 PM UTC+2, Jesse Glick wrote:
On Mon, Apr 30, 2018 at 10:09 AM,  <thomas.w...@de.amadeus.com> wrote:

>> Is there a way to modify the value of an environment variable for a single
>> step
>> or block in Pipeline from a GraphListener?
>> The value would have to change for each node.

> I do not think so. As of JENKINS-42499, an `EnvironmentContributor`
> could change a value dynamically. This is not scoped to a step; it
> just means that if and when a step requests access to environment
> variables, your plugin will be asked anew for contributions.

Thanks for the hint I will take a look at it.

>> before
>> each build in AbstractBuild.getEnvironment().
>
> `Run.getEnvironment` I guess you mean. Pipeline builds are not `AbstractBuild`s.

Yes.

>> I also tried EnvActionImpl.forRun((Run) flowNode.getExection().getOwner().getExecutable())

> I am not sure what you are trying to accomplish here, but stop. Do not
> declare a (`compile`-scoped) dependency on `workflow-cps` at all. You
> may use `workflow-step-api` and, if necessary, `workflow-api` as
> dependencies.

Thanks for the hint. The dependency on `workflow-api` was enough for all other
ways forward I tried.

>> Now the issue is to propagate the tracing context from within Jenkins to the
>> executed steps. in this case Maven for which a similar plugin was created.
>> The probably best way of propagating this information is via environment
>> variables.
>> Obviously the value of this environment variable will have to change often
>> during the course of a single build.
>
> Just define a step which would provide the current tracing context
> (whatever that is) as its return value. That will work fine in
> Scripted. If you require Declarative compatibility, your best bet is
> to define a block-scoped step which would define some environment
> variable(s) for its nested steps. That is my best recommendation based
> on my very limited grasp of what it is you are trying to do.

This worked best so far, only a few issues remain.
There seems to be a slight API-assumption mismatch between Jenkins and the
component I am trying to integrate it with, which I have to deal with.

The fact that this requires explicit adaptions to the pipeline definition is a
bit unfortunate, though.

The goal I am trying to achieve is a deep insight into the CI/build/etc
pipeline.
So if for example a user of the platform complains about slow builds, we can
see exactly where time is spent.
Waiting for executors, running a certain shell script, a single maven step and
so on.
And all of this would be stored in a uniform way for queries and analysis.

>> it seems the usage of env vars with a dash ("-") in their keys is
>> broken.
>> Even when set directly via the Java APIs, they don't show up in the "sh"
>> pipeline step.

> https://unix.stackexchange.com/a/23714/26736

As everything goes through a shell fair enough.
I somehow thought `sh` would be like plain `execve` which was quite far
fetched.

Thanks again.

[Jesse Glick]

On Wed, May 2, 2018 at 10:50 AM, <thomas.w...@de.amadeus.com> wrote:
> The goal I am trying to achieve is a deep insight into the CI/build/etc
> pipeline.
> So if for example a user of the platform complains about slow builds, we can
> see exactly where time is spent.
> Waiting for executors, running a certain shell script, a single maven step
> and
> so on.
> And all of this would be stored in a uniform way for queries and analysis.

All of that metadata is already available via `workflow-api` metadata
calls. It is unclear to me why it would need to be made available to
steps inside the build itself as environment variables, as opposed to
via REST endpoints on the build URL or something (akin to what Blue
Ocean uses).

> As everything goes through a shell fair enough.
> I somehow thought `sh` would be like plain `execve`

There is JENKINS-44231, though the most straightforward implementation
would not handle prohibited environment variables.

[Thomas Weißschuh]

Hi Jesse,

thanks again for the fast response!

On Wed, May 02, 2018 at 06:20:34PM -0400, Jesse Glick wrote:
> On Wed, May 2, 2018 at 10:50 AM, <thomas.w...@de.amadeus.com> wrote:
> > The goal I am trying to achieve is a deep insight into the CI/build/etc
> > pipeline.
> > So if for example a user of the platform complains about slow builds, we can
> > see exactly where time is spent.
> > Waiting for executors, running a certain shell script, a single maven step
> > and
> > so on.
> > And all of this would be stored in a uniform way for queries and analysis.
>
> All of that metadata is already available via `workflow-api` metadata
> calls. It is unclear to me why it would need to be made available to
> steps inside the build itself as environment variables, as opposed to
> via REST endpoints on the build URL or something (akin to what Blue
> Ocean uses).

The steps are supposed to report their own internal information too.
By exposing the id of the current state from Jenkins to the build step both
executions can be correlated and an end-to-end trace of the complete
build-infrastructure can be shown.
For this it is important that each single step has its own identity, that can
be accessed from within the step (eg maven in the `sh` step).
I am honestly not sure how it would be possible to get information for a
specific step from the REST API without having a way of knowing the identity of
the current step in the first place.

By passing this identity via environment variables I hoped to keep the changes
to the code of the users of build-infrastructure minimal to non-existent.
At the same time it would provide optional extensibility for the users to also
instrument their own code and hook it up to the existing information about the
Jenkins build.

The whole data collection can be extended far before and after the build pipeline,
including the original trigger of the build, external message queues, non-Jenkins
resource provisioning etc.

> > As everything goes through a shell fair enough.
> > I somehow thought `sh` would be like plain `execve`
>
> There is JENKINS-44231, though the most straightforward implementation
> would not handle prohibited environment variables.

The restriction is not an issue. It was a problem of understanding on my part.

Thanks for the pointer though, as I personally much prefer executing things
without having the mess of a shell in between.
(Except in the many valid cases where it is necessary)

[Jesse Glick]

On Thu, May 3, 2018 at 2:38 AM, Thomas Weißschuh
<thomas.w...@de.amadeus.com> wrote:
> For this it is important that each single step has its own identity, that can
> be accessed from within the step (eg maven in the `sh` step).
> I am honestly not sure how it would be possible to get information for a
> specific step from the REST API without having a way of knowing the identity of
> the current step in the first place.

Well, at least for `sh` steps you can identify the command line (in
most cases) via `ArgumentsAction`.

Note that my suggestion about defining environment variables in a
custom block-scoped step may not fit your needs since the environment
variables need to be defined before the body starts, so you could not
include timing information about this step itself.

From my (again, weak) understanding of your goals, it might work
better to use the `STAGE_NAME` environment variable from inside
external processes to see where you are in the build, which can then
be mapped back to build metadata via the flow graph. Or scripts could
be amended slightly to just use `withEnv` to specify some arbitrary
ID, accessible of course to external processes via environment,
correlatable to the flow graph via `ArgumentsAction`.

I suppose we could also amend `DefaultStepContext.get` to include a
special variable bound to the current `FlowNode.id`; and/or more
generally introduce an `ExtensionPoint` similar to
`EnvironmentExpander` but accepting a `StepContext` argument (to be
called from a new overload of `getEffectiveEnvironment`), so you could
bind such a variable yourself. The latter would be an Enhancement with
`api` label in `workflow-step-api-plugin` + `workflow-support-plugin`.

[Thomas Weißschuh]

On Thu, May 03, 2018 at 02:27:32PM -0400, Jesse Glick wrote:
> Well, at least for `sh` steps you can identify the command line (in
> most cases) via `ArgumentsAction`.

Thanks for the hint. I would prefer to have proper IDs for correlation, but if
nothing else works this will probably have to do.

> Note that my suggestion about defining environment variables in a
> custom block-scoped step may not fit your needs since the environment
> variables need to be defined before the body starts, so you could not
> include timing information about this step itself.

Yes this is the case. Unfortunately currently it is not possible to create the
timer in a non-started state.

> From my (again, weak) understanding of your goals, it might work
> better to use the `STAGE_NAME` environment variable from inside
> external processes to see where you are in the build, which can then
> be mapped back to build metadata via the flow graph. Or scripts could
> be amended slightly to just use `withEnv` to specify some arbitrary
> ID, accessible of course to external processes via environment,
> correlatable to the flow graph via `ArgumentsAction`.

It would have to be a custom step, because the environment variable has to be
computed by the library. But otherwise, this is what I have currently.

> I suppose we could also amend `DefaultStepContext.get` to include a
> special variable bound to the current `FlowNode.id`;

This is actually currently implemented in `CpsStepContext`,
and exposed via the generic `StepContext.get`.
It includes the complete FlowNode.
This is currently used by my mentioned custom step.

> and/or more
> generally introduce an `ExtensionPoint` similar to
> `EnvironmentExpander` but accepting a `StepContext` argument (to be
> called from a new overload of `getEffectiveEnvironment`), so you could
> bind such a variable yourself. The latter would be an Enhancement with
> `api` label in `workflow-step-api-plugin` + `workflow-support-plugin`.

I assume it should be similar to `EnvironmentContributor`, right?

Something like this would be great.

If this feature has a chance of being merged I would like to take a stab at
implementing it.
Should I open a proper ticket to discuss the details or are concrete prototypes
preferred?

[Jesse Glick]

On Fri, May 4, 2018 at 3:22 AM, Thomas Weißschuh
<thomas.w...@de.amadeus.com> wrote:
>> I suppose we could also amend `DefaultStepContext.get` to include a
>> special variable bound to the current `FlowNode.id`;
>
> This is actually currently implemented in `CpsStepContext`,
> and exposed via the generic `StepContext.get`.
> It includes the complete FlowNode.

By “variable” here I mean “environment variable”: to include the value
of the node ID in `context.get(EnvVars.class)`.

>> introduce an `ExtensionPoint` similar to
>> `EnvironmentExpander` but accepting a `StepContext` argument (to be
>> called from a new overload of `getEffectiveEnvironment`), so you could
>> bind such a variable yourself. The latter would be an Enhancement with
>> `api` label in `workflow-step-api-plugin` + `workflow-support-plugin`.
>
> I assume it should be similar to `EnvironmentContributor`, right?

Roughly, yes, but with a finer-grained context: an
`EnvironmentContributor` only knows which build is running, nothing
Pipeline-specific.

> If this feature has a chance of being merged

@svanoort & @abayer could speak to that.

> I would like to take a stab at
> implementing it.
> Should I open a proper ticket to discuss the details or are concrete prototypes
> preferred?

Both.

[Thomas Weißschuh]

On Fri, May 04, 2018 at 11:39:12AM -0400, Jesse Glick wrote:
> By “variable” here I mean “environment variable”: to include the value
> of the node ID in `context.get(EnvVars.class)`.

Ok, makes sense.

>> If this feature has a chance of being merged
>
> @svanoort & @abayer could speak to that.

I mentioned them on the ticket.

>> I would like to take a stab at
>> implementing it.
>> Should I open a proper ticket to discuss the details or are concrete prototypes
>> preferred?

> Both.

https://issues.jenkins-ci.org/browse/JENKINS-51170