Checking for ERROR and WARN log messages

[Jernej Rus]

Hello. ZAP Automation Framework jobs might log unexpected ERROR or WARN messages. Examples:

1794825 [ZAP-ActiveScanner-16] WARN org.parosproxy.paros.core.scanner.VariantJSONQuery - Failed to parse the request body for url [MASKED] : Input is invalid JSON; does not start with '{' or '[', c=34 java.lang.IllegalArgumentException: Input is invalid JSON; does not start with '{' or '[', c=34 at org.parosproxy.paros.core.scanner.VariantJSONQuery.parseObject(VariantJSONQuery.java:139) ~[zap-2.16.1.jar:2.16.1] at org.parosproxy.paros.core.scanner.VariantJSONQuery.parseContent(VariantJSONQuery.java:89) ~[zap-2.16.1.jar:2.16.1] at org.parosproxy.paros.core.scanner.VariantAbstractRPCQuery.setRequestContent(VariantAbstractRPCQuery.java:148) ~[zap-2.16.1.jar:2.16.1] at org.parosproxy.paros.core.scanner.VariantAbstractRPCQuery.setMessage(VariantAbstractRPCQuery.java:60) [zap-2.16.1.jar:2.16.1] at org.parosproxy.paros.core.scanner.AbstractAppParamPlugin.scan(AbstractAppParamPlugin.java:91) [zap-2.16.1.jar:2.16.1] at org.parosproxy.paros.core.scanner.AbstractPlugin.run(AbstractPlugin.java:391) [zap-2.16.1.jar:2.16.1] at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

1465072 [BiDi Connection] ERROR org.zaproxy.zap.ZAP.UncaughtExceptionLogger - Exception in thread "BiDi Connection"

I don't want to disregard such messages, even if the job passes anyway. At the end of the ZAP plan, I want to check if the amount of ERROR and WARN messages is greater than 0. Do any ZAP statistics track these amounts? Or is there a better way to do this?

[Simon Bennetts]

Hiya,

We do have the statistic "stats.error.core.uncaught" which you could check?

Sadly not mentioned on https://www.zaproxy.org/docs/internal-statistics/ so we should fix that asap.

But it would be more useful if we had other stats for the errors and warnings as well.

We'll have a think and see what we can come up with..

Cheers,

Simon

[Jernej Rus]

Thank you for the reply. I'll check out stats.error.core.uncaught. I was also thinking of creating an appender or a filter for rootLogger that monitors for LogEvent with level equal to or greater than WARN. Do you see any issues here?

[Simon Bennetts]

This is what we do in the dev add-on: https://github.com/zaproxy/zap-extensions/blob/main/addOns/dev/src/main/java/org/zaproxy/addon/dev/error/LoggedErrorsHandler.java
But our assumption is that most people wont install this, so we're looking at something that will be available to everyone for any new stats we add.

Cheers,

Simon

[Jernej Rus]

I see. Maybe I can use that handler too. I think I have enough resources now to continue.

OK, this conversation is now resolved. Thank you for your help.

[Jernej Rus]

For anyone interested - I ended up writing a ZAP extender Jython script that creates a custom log4j appender. The appender has a ThresholdFilter with Level WARN, and an append() method that stores the messages in a thread-safe deque. The script starts the appender and adds it to RootLogger. When the script is uninstalled and stored messages exist, the appender logs them again and fails with sys.exit(1).

[Simon Bennetts]

The new Insights add-on will report if ZAP logs and errors or warnings :)

It will be included in ZAP 2.17 and is already in the weekly/nightly releases.

There will be more enhancements v soon, including adding the insights to the reports.

For a quick overview see https://www.zaproxy.org/blog/2025-11-25-50-million-errors-in-one-day/#insights

Cheers,

Simon

[Jernej Rus]

Looks very useful, thanks for letting me know. Will this be available in Automation Framework too?

[Simon Bennetts]

Yes :) 

The reports will be just the same, but we are also planning on adding a new job to configure the insights - that will give you more control on if & when the plan is ended early.

Cheers,

Simon

[Jernej Rus]

Awesome! Thanks for letting me know..