Reading console log of parallel stages

[Ullrich Hafner]

If a pipeline contains parallel stages, then the console log will contain an unordered mix of stdout messages of all parallel stages. This makes it impossible to decided which log message is from which parallel stage if I access the log using Run.getLogReader().

Since a couple of weeks the console log view shows in the frame border which log message is from which parallel stage, so it is somehow possible to obtain that information. Which API can I use to read the individual console log information of a parallel stage?

[Jesse Glick]

On Mon, Jan 14, 2019 at 6:34 PM Ullrich Hafner <ullrich...@gmail.com> wrote:
> Which API can I use to read the individual console log information of a parallel stage?

There is not an API exactly for that, but you can use various
techniques in `workflow-api` like `FlowScanner` to get the set of
`FlowNode`s in the branch, and you can use `LogAction` (if present) to
retrieve any log text associated with that node.

I think I mentioned at some point that a simpler and more efficient
way to process log messages is to use a block-scoped step, which can
then contextualize a `ConsoleLogFilter` (freestyle-compatible but
clunky) or `TaskListenerDecorator` (Pipeline-specific but cleaner)
that passes through log lines after tallying them for warnings or
whatever, with no need to ever read the build log later.

[Ullrich Hafner]

I finally managed it to implement a block scoped step. Currently I invoke the body of the step using this API

                getContext().newBodyInvoker()
                        .withContext(new ConsoleLogSplitter(…)) // implements TaskListenerDecorator and writes the output to a temporary file
                        .withCallback(new RecordIssuesCallback(…)) // extends BodyExecutionCallback and reads the temporary file to extract the warnings 
                        .start();

My TaskListenerDecorator instance is now invoked for every line in the body of my step. Is there a similar API to get the whole log of the body instead? Or do I need to concatenate the log on my own? 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1F11wKsJtfsUWy4h3ErgvxmyM4BQVrY-gGNZtb9WwgQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Tue, Oct 22, 2019 at 5:44 AM Ullrich Hafner <ullrich...@gmail.com> wrote:
> .withContext(new ConsoleLogSplitter(…)) // implements TaskListenerDecorator and writes the output to a temporary file
> .withCallback(new RecordIssuesCallback(…)) // extends BodyExecutionCallback and reads the temporary file to extract the warnings

This sort of defeats the purpose of using `TaskListenerDecorator`, I
think. You should be able to extract warnings on the fly without
making a copy of the log.

Note that to behave properly with `sh`/`bat`/`powershell` steps in
`org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING=true`
mode (including when `pipeline-cloudwatch-logs` is installed), as well
as to handle some other steps like `checkout`, the
`TaskListenerDecorator` must be prepared to run remotely inside the
agent JVM. (See JEP-210 for background.) Under the assumption that
warnings are occasional low-bandwidth events, this is most easily done
by calling an exported interface method. The `timeout` step
implementation has an example.

[Ullrich Hafner]

> Am 22.10.2019 um 15:20 schrieb Jesse Glick <jgl...@cloudbees.com>:
>
> On Tue, Oct 22, 2019 at 5:44 AM Ullrich Hafner <ullrich...@gmail.com> wrote:
>> .withContext(new ConsoleLogSplitter(…)) // implements TaskListenerDecorator and writes the output to a temporary file
>> .withCallback(new RecordIssuesCallback(…)) // extends BodyExecutionCallback and reads the temporary file to extract the warnings
>
> This sort of defeats the purpose of using `TaskListenerDecorator`, I
> think. You should be able to extract warnings on the fly without
> making a copy of the log.

Hmm, this is currently not possible since all parsers (XML, JSON parsers and console line parsers) use a different API at the moment: they basically work on a reader instance and not on single lines. I think it might be feasible to change that at least for those parsers that read the console log (and do not work on whole files) but this requires some extra work.

>
> Note that to behave properly with `sh`/`bat`/`powershell` steps in
> `org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING=true`
> mode (including when `pipeline-cloudwatch-logs` is installed), as well
> as to handle some other steps like `checkout`, the
> `TaskListenerDecorator` must be prepared to run remotely inside the
> agent JVM. (See JEP-210 for background.) Under the assumption that
> warnings are occasional low-bandwidth events, this is most easily done
> by calling an exported interface method. The `timeout` step
> implementation has an example.
>

If the TaskListenerDecorator method might be called on the agent how can I transfer results back to the master from it?
Or is the callback (from withCallback) then called on the agent as well?

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2fMM1tFQsOhyROt3D5i%2BUHQpwZABqueP%3DCoNXJVRB8Bw%40mail.gmail.com.

[Jesse Glick]

On Tue, Oct 22, 2019 at 1:48 PM Ullrich Hafner <ullrich...@gmail.com> wrote:
> If the TaskListenerDecorator method might be called on the agent how can I transfer results back to the master from it?

Really best to look at the example implementation in `TimeoutStep` to
see how to use `Channel.export`.

> Or is the callback (from withCallback) then called on the agent as well?

No, callbacks are run inside the Pipeline engine on the master.