Automation Framework "monitor" job test proposal

[psiinon]

I've been thinking about how to fail an Automation Framework plan early, for example if the authentication fails.

My plan is to create a new "monitor" type of job test.

It would be similar to the "stats" job test but instead of running after a job it would run _during_ a job.

It would only be supported by specific 'long running' jobs like the spider, spiderAjax, active scan etc

The config would include a statistic name and a threshold - if that threshold is exceeded at any point during the job then the job will fail. The action being performed (spider / active scan etc) will be stopped.

Whether or not the plan fails will depend on the plan config.

The idea is that this could then be used to check for things like auth failures and exit a job/plan early rather than potentially waiting hours for it to finish and then fail.

Any thoughts, suggestions or feedback?

Cheers,

Simon

[Ananda Krishna]

This sounds like a good idea - we have something similar implemented. Have currently implemented such a threshold using global variables - and it works well in production across multiple sites.

Like you have suggested, would be great if we can specify in the plan config whether the plan should fail or continue scanning without trying to authenticate again. This would be a little more flexible, and allow the site to be scanned (although without authentication) as it would have normally done.

Currently when using script based auth (and selenium), and an incorrect auth script - ZAP retries auth multiple times in a span of few seconds causing the browser to respawn every time and hog resources.

[psiinon]

The PR for this enhancement is https://github.com/zaproxy/zap-extensions/pull/4005 - feedback always appreciated :)

Right now theres no option to "continue scanning with no auth", partly because this is completely generic and works off any stats, and partly because that sort of feature would require core changes.

However I'm also wondering how useful it would actually be?

I would prefer to know that the authentication failed and that the job terminated.

Note that if you are using script based auth you can implement this in your scripts, which I think would be a cleaner approach...

Thoughts anyone?