"Required array size too large" Java runs of memory during report download

[Mark Ibrahim]

Hello, I have a few scans that fail while trying to download traditional-json-plus-report. I have seen the OutOfMemoryError before, but never with " Required array size too large." I am wondering if there is any recommended way to resolve this other than increasing the java heap size, since it is already 24gb. Below I have included the error, which is the only log after the active scan completes.

Thank you for your time.

18745412 [ZAP-IO-Server-1-82] ERROR org.zaproxy.addon.network.internal.server.http.MainServerHandler - An error occurred while notifying a handler:
java.lang.OutOfMemoryError: Required array size too large
at java.nio.file.Files.readAllBytes(Unknown Source) ~[?:?]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOtherImpl(CoreAPI.java:1531) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOther(CoreAPI.java:1346) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:576) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleApiRequest(ZapApiHandler.java:111) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleRequest(ZapApiHandler.java:85) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleMessage(ZapApiHandler.java:70) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.notifyMessageHandlers(MainServerHandler.java:151) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.processMessage(MainServerHandler.java:131) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.LocalServerHandler.processMessage(LocalServerHandler.java:67) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.process(MainServerHandler.java:94) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.lambda$channelRead0$0(MainServerHandler.java:82) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [network-beta-0.14.0.zap:?]
at java.lang.Thread.run(Unknown Source) [?:?]
18745445 [ZAP-IO-Server-1-82] ERROR org.zaproxy.addon.network.internal.server.http.MainServerHandler - An error occurred while notifying a handler:
java.lang.OutOfMemoryError: Required array size too large
at java.nio.file.Files.readAllBytes(Unknown Source) ~[?:?]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOtherImpl(CoreAPI.java:1531) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOther(CoreAPI.java:1346) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:576) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleApiRequest(ZapApiHandler.java:111) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleRequest(ZapApiHandler.java:85) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleMessage(ZapApiHandler.java:70) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.notifyMessageHandlers(MainServerHandler.java:151) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.processMessage(MainServerHandler.java:131) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.LocalServerHandler.processMessage(LocalServerHandler.java:67) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.process(MainServerHandler.java:94) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.lambda$channelRead0$0(MainServerHandler.java:82) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [network-beta-0.14.0.zap:?]
at java.lang.Thread.run(Unknown Source) [?:?]
18745448 [ZAP-IO-Server-1-82] ERROR org.zaproxy.addon.network.internal.server.http.MainServerHandler - An error occurred while notifying a handler:
java.lang.OutOfMemoryError: Required array size too large
at java.nio.file.Files.readAllBytes(Unknown Source) ~[?:?]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOtherImpl(CoreAPI.java:1531) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOther(CoreAPI.java:1346) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:576) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleApiRequest(ZapApiHandler.java:111) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleRequest(ZapApiHandler.java:85) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleMessage(ZapApiHandler.java:70) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.notifyMessageHandlers(MainServerHandler.java:151) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.processMessage(MainServerHandler.java:131) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.LocalServerHandler.processMessage(LocalServerHandler.java:67) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.process(MainServerHandler.java:94) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.lambda$channelRead0$0(MainServerHandler.java:82) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [network-beta-0.14.0.zap:?]
at java.lang.Thread.run(Unknown Source) [?:?]
18745450 [ZAP-IO-Server-1-82] ERROR org.zaproxy.addon.network.internal.server.http.MainServerHandler - An error occurred while notifying a handler:
java.lang.OutOfMemoryError: Required array size too large
at java.nio.file.Files.readAllBytes(Unknown Source) ~[?:?]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOtherImpl(CoreAPI.java:1531) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.CoreAPI.handleApiOther(CoreAPI.java:1346) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:576) ~[zap-2.14.0.jar:2.14.0]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleApiRequest(ZapApiHandler.java:111) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleRequest(ZapApiHandler.java:85) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.handlers.ZapApiHandler.handleMessage(ZapApiHandler.java:70) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.notifyMessageHandlers(MainServerHandler.java:151) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.processMessage(MainServerHandler.java:131) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.LocalServerHandler.processMessage(LocalServerHandler.java:67) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.process(MainServerHandler.java:94) ~[?:?]
at org.zaproxy.addon.network.internal.server.http.MainServerHandler.lambda$channelRead0$0(MainServerHandler.java:82) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [network-beta-0.14.0.zap:?]
at java.lang.Thread.run(Unknown Source) [?:?]
18560842 [ZAP-Automation] INFO org.parosproxy.paros.CommandLine - Job activeScan finished, time taken: 04:40:13

[Simon Bennetts]

I've not noticed that one, but the answer is the same I'm afraid - add more memory.

However I'd also check the ZAP log - at the start it should tell you how much memory is available to ZAP. Double check its the value you expect.

Cheers,

Simon

[Mark Ibrahim]

Thanks for your response. Unfortunately, I am indeed using 24gb of heap, so I may disable recording traffic in the report if it helps.

I appreciate your help.

[kingthorin+zap]

If you're really exhausting 24GB of RAM it seems like you're scanning in a very inefficient way. Is this a catalog or store site? Or do you have some functionality that has absolutely gigantic responses?

[thc...@gmail.com]

Raise an issue, that API endpoint should be changed to send the response
chunked.

As kingthorin poninted out, would be interesting to know the average
size of the messages (or more details of the contents of the report).

Best regards.

[Mark Ibrahim]

Hello all, yes, the sites that run into this issue are large stores/catalogs. Almost all other scans I perform are much less memory hungry.

Not sure what information to include in an issue post, but I'll link it here. Thanks for your continued support. Like I said, I'll be downloading the reports without traffic to try to resolve this issue.

[thc...@gmail.com]

Thank you for the information and raising the issue.

Best regards.

On 01/03/2024 00:12, Mark Ibrahim wrote:
> Hello all, yes, the sites that run into this issue are large
> stores/catalogs. Almost all other scans I perform are much less memory
> hungry.
>
> Not sure what information to include in an issue post, but I'll link it here

> <https://github.com/zaproxy/zaproxy/issues/8371>. Thanks for your continued