GitHub workflow - identifying FAILure from WARNings

[Nathan Summers]

Hi,

Using zaproxy/action-api-scan or zaproxy/action-full-scan results in a successful step always, unless setting fail_action: true.

When the action does fail, it fails when the scan result exits with 1, 2 or 3.

https://www.zaproxy.org/docs/docker/baseline-scan/#exit-value

I would not expect 2 (WARNings) to result in triggering a workflow failure.

I'd suggest this is a bug, or at least misleading wording of parameters.

(NB also, in the docs of fail_action, I read the two sentences as contradictions - may be worth reviewing).

Suggestion 1:

Expose more outputs in the GH action, sharing the evaluated FAIL/WARNings.  eg at least the exit code as above.  This will help subsequent decision making in automation.

Suggestion 2:

Extend or replace the existing 'fail_action' with something like

on_fail_behaviour: 'do_fail_action'

on_warn_behaviour: 'do_nothing'

(NB my own aversion to booleans)

Suggestion 3:

Bundle the evaluation of the pass/FAIL/WARN in with the reports.

In my GH workflow I could download the report and parse the Highs, Mediums, etc, but I could not find a way of accessing the important evaluation of pass or fail.

It's really important our workflows fail, fail only when necessary., and scan results are not habitually dismissed.

All help really appreciated!

[Simon Bennetts]

Hiya,

Have a look at the Automation Framework (AF) Action: https://github.com/zaproxy/action-af

ZAP has a huge number of options, and we've decided to try to keep the packaged scans more restricted and make sure that the AF Action provides all of the flexibility needed.

I'm not saying it will do everything that you want, but we're much more likely to add new options to that Action rather than the other ones :)

I'm also not saying that we wont change the packaged scan options (sorry for the double negatives:/ ) but we will probably need to consider and changes to those much more carefully.

Cheers,

Simon

[Nathan Summers]

Thanks for responding, much appreciated.

Looking at the source of action-af, it seems to suffer the same issue as action-full-scan and action-api-scan.  None of them declare outputs, so effectively swallow the docker return value.

I really think if these are off-the-shelf actions to be used by the general public, they should at least pass some outputs back to the calling github workflow?  Unless I've misunderstood and they are example workflows for people to copy+paste and build upon?

[Simon Bennetts]

That sounds like a bug to me, we really want these actions to be usable "as is".

Thanks for letting us know!