Client Spider - Blog Post and Video

[Simon Bennetts]

In ZAP 2.16.0 we introduced a new client Spider.

This blog post and video explain why we did that, how it works, and where it’s going:

https://www.zaproxy.org/blog/2025-01-31-client-spider/

Any feedback you have about the Client Spider would be much appreciated on this thread!

Many thanks,

Simon

[zinw elzl]

When will be "We also need to actually use the Client Map when attacking sites"

[Simon Bennetts]

Hopefully sooner rather than later, but we dont have a specific date yet.

Cheers,

Simon

[Cruzcat]

Simon, 

Tks for the video and details. 

A quick question. Will it be possible to use the Client Spider in lieu of the Ajax Spider with the Zap client SDK or APIs ?.

Cheers.

[kingthorin+zap]

Not yet, but yes ultimately that is our hope/plan

[Cruzcat]

Perfect. Thanks.

[tenaz 3]

Hi Simon, I've been playing around Client Spider API. But I got some timeouts on the websites that I scanned. Is it something wrong or Am I missing configurations?

The message says: Navigation timed out after 1000 ms Build info: version: '4.30.0', revision: '509c7f17cc*' System info: os.name: 'Mac OS X', os.arch: 'aarch64', os.version: '15.3.2', java.version: '17.0.14' Driver info: org.openqa.selenium.firefox.FirefoxDriver Command: The rest of the command I did not manage to get from the ZAP UI (2.16.1)

[tenaz 3]

Another question I have, running zap client spider, the spider finished with 100% but still have told me is not finished, how can I check properly if it was finished?

[tenaz 3]

I manage to get the exception:

2025-04-02 10:26:23 1138171 [ZAP-ClientSpiderThreadPool-0-thread-10] WARN org.zaproxy.addon.client.spider.ClientSpiderTask - Task 758 failed Navigation timed out after 1000 ms

2025-04-02 10:26:23 Build info: version: '4.30.0', revision: '509c7f17cc*'

2025-04-02 10:26:23 System info: os.name: 'Linux', os.arch: 'aarch64', os.version: '6.10.14-linuxkit', java.version: '17.0.14'

2025-04-02 10:26:23 Driver info: org.openqa.selenium.firefox.FirefoxDriver

2025-04-02 10:26:23 Command: [13278670-bada-4a2b-a1b1-785a4b0104a7, get {url=https://www.mydomain/company/background-story/}]

2025-04-02 10:26:23 Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 128.8.0, moz:accessibilityChecks: false, moz:buildID: 20250224130137, moz:debuggerAddress: 127.0.0.1:12752, moz:geckodriverVersion: 0.36.0, moz:headless: true, moz:platformVersion: 6.10.14-linuxkit, moz:processID: 1980, moz:profile: /tmp/rust_mozprofilezUVoZ3, moz:shutdownTimeout: 60000, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, userAgent: Mozilla/5.0 (X11; Linux x86..., webSocketUrl: ws://127.0.0.1:12752/sessio...}

2025-04-02 10:26:23 Session ID: 13278670-bada-4a2b-a1b1-785a4b0104a7

2025-04-02 10:26:23 org.openqa.selenium.TimeoutException: Navigation timed out after 1000 ms

2025-04-02 10:26:23 Build info: version: '4.30.0', revision: '509c7f17cc*'

2025-04-02 10:26:23 System info: os.name: 'Linux', os.arch: 'aarch64', os.version: '6.10.14-linuxkit', java.version: '17.0.14'

2025-04-02 10:26:23 Driver info: org.openqa.selenium.firefox.FirefoxDriver

2025-04-02 10:26:23 Command: [13278670-bada-4a2b-a1b1-785a4b0104a7, get {url=https://www.mydomain/company/background-story/}]

2025-04-02 10:26:23 Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 128.8.0, moz:accessibilityChecks: false, moz:buildID: 20250224130137, moz:debuggerAddress: 127.0.0.1:12752, moz:geckodriverVersion: 0.36.0, moz:headless: true, moz:platformVersion: 6.10.14-linuxkit, moz:processID: 1980, moz:profile: /tmp/rust_mozprofilezUVoZ3, moz:shutdownTimeout: 60000, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, userAgent: Mozilla/5.0 (X11; Linux x86..., webSocketUrl: ws://127.0.0.1:12752/sessio...}

2025-04-02 10:26:23 Session ID: 13278670-bada-4a2b-a1b1-785a4b0104a7

2025-04-02 10:26:23 at jdk.internal.reflect.GeneratedConstructorAccessor152.newInstance(Unknown Source) ~[?:?]

2025-04-02 10:26:23 at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

2025-04-02 10:26:23 at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]

2025-04-02 10:26:23 at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.ErrorCodec.decode(ErrorCodec.java:167) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:138) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:50) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:215) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:216) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:174) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:545) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:313) ~[selenium-remote-driver-4.30.0.jar:?]

2025-04-02 10:26:23 at org.zaproxy.addon.client.spider.actions.OpenUrl.run(OpenUrl.java:40) ~[client-alpha-0.15.0.zap:?]

2025-04-02 10:26:23 at org.zaproxy.addon.client.spider.ClientSpiderTask.lambda$runImpl$0(ClientSpiderTask.java:119) ~[client-alpha-0.15.0.zap:?]

2025-04-02 10:26:23 at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]

2025-04-02 10:26:23 at org.zaproxy.addon.client.spider.ClientSpiderTask.runImpl(ClientSpiderTask.java:119) [client-alpha-0.15.0.zap:?]

2025-04-02 10:26:23 at org.zaproxy.addon.client.spider.ClientSpiderTask.run(ClientSpiderTask.java:81) [client-alpha-0.15.0.zap:?]

2025-04-02 10:26:23 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]

2025-04-02 10:26:23 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]

2025-04-02 10:26:23 at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

2025-04-02 10:26:23 1138179 [ZAP-IO-Server-1-67] WARN org.zaproxy.addon.network.internal.server.http.handlers.HttpSenderHandler - Failed to read https://www.mydomain/ within 20 seconds, check to see if the site is available and if so consider adjusting ZAP's read time out in the Connection options panel.

2025-04-02 10:26:23 1138185 [ZAP-IO-Server-1-70] WARN org.zaproxy.addon.network.internal.server.http.handlers.HttpSenderHandler - Failed to read https://www.mydomain/ within 20 seconds, check to see if the site is available and if so consider adjusting ZAP's read time out in the Connection options panel.

[Simon Bennetts]

The main reason for timeouts is because the target did not respond within the given time.

The default Page Load Time is 1 second, but you can increase that in the options.

The spider not finishing sounds like a bug though.

Can you reproduce that consistently, and if so how?

Many thanks,

Simon

[tenaz 3]

Ok, I will open a pr for increasing the timeout over API.

I was able to reproduce the bug once or twice through the ZAP UI. When running ZAP in headless mode, it still gets stuck, but I haven't confirmed if this is due to the same cause. I'm currently investigating to determine exactly when this issue occurs, as it appears to be intermittent at the moment.

[Simon Bennetts]

Its not a bug, its a configuration option.

You are instructing the spider to crawl a site and wait for 1 second for the page to load.

If it takes more than 1 second then you'll get timeouts, to fix it you just need to increase the timeout.

We dont set the timeout to an arbitrary high value because then ZAP will take much longer to crawl some web sites.

Its a trade off.

I'm all for adding configuration support via the API though :)

Cheers,

Simon

[tenaz 3]

Simon after a bit more Analysis:
The ZAP got stuck and even Shutdown or creating new session did not work. I check the thread dump.

Thread Dump Analysis:

• Thread Creation Failure: The system failed to create new threads due to resource exhaustion, likely caused by an OutOfMemoryError or an EAGAIN error from pthread_create.

• Timeouts in ZAP’s Client Spider with Selenium: The Firefox WebDriver encountered a TimeoutException, indicating that the pageLoad request exceeded its time limit. The error suggests a 1-second timeout (pageLoad = 1000 ms), which may be too short for pages requiring more time to load. (Already discussed)
  

Regarding the Thread Creation, it seems the client spider went a bit crazy on that. I see a loot of threads (> 1000), but since I dump the thread every 5 min I may missed something. Regardless do you have any suggestion how I can prevent it?

[thc...@gmail.com]

Hi,

The evidence provided does not show the spider finished (see the pause
and stop button still active), just that all currently available tasks
were completed (the 100%).

In that state the spider will still wait some time (configurable) for
any last events to be sent by the browser extension (e.g. page finished
loading and some new elements were identified) before stopping.

Best regards.

[thc...@gmail.com]

Hi,

It would be better to provide the thread dump.

Best regards.

On 03/04/2025 21:44, tenaz 3 wrote:
> Simon after a bit more Analysis:
> The ZAP got stuck and even Shutdown or creating new session did not work. I
> check the thread dump.
>
> Thread Dump Analysis:
>

> -
>
> *Thread Creation Failure*: The system failed to create new threads due
> to resource exhaustion, likely caused by an *OutOfMemoryError* or an
> *EAGAIN* error from pthread_create.
> -
>
> *Timeouts in ZAP’s Client Spider with Selenium*: The Firefox WebDriver
> encountered a *TimeoutException*, indicating that the pageLoad request
> exceeded its time limit. The error suggests a *1-second timeout
> (pageLoad = 1000 ms)*, which may be too short for pages requiring more

[tenaz 3]

Hi all,

Apologies for the delay.

I've collected the thread dump data along with the ZAP error logs. From what I can tell, the issue seems related to Thread Exhaustion.

To support further investigation, you can check the attached logs. I'm also sharing a snippet of how I typically run ZAP, and the adjusted command that helped me mitigate the issue. While this workaround resolved the problem for me, I don’t believe it’s a definitive solution—perhaps the ZAP team might want to take a closer look.

Usual command:

podman run --pod zap-pod --memory=15g --cpus=5.5 --restart=always -u zap --name owasp -e ZAP_PORT=8090 -d ghcr.io/zaproxy/zaproxy:stable ...

Adjusted command (this worked for me):

podman run --pod zap-pod --memory=15g --cpus=5.5 --pids-limit=3000 --restart=always -u zap --name owasp -e ZAP_PORT=8090 -d ghcr.io/zaproxy/zaproxy:stable ...

Adding --pids-limit=3000 was the key to keeping ZAP stable and preventing it from running out of resources.

[tenaz 3]

Hi again,

I’d also like to share some feedback and see if anyone can help with this.

I’m testing a Single Page Application (SPA) that includes multiple buttons and pages behind an authentication screen. The authentication flow is working fine—I’ve configured ZAP to use Selenium for authentication, and that part runs smoothly.

However, I noticed something interesting during the Client-Side Spider run:

When my app used raw HTML elements like <a> or <button>, the Client Spider wasn’t able to detect or follow the links properly. But after I switched to using NavLink from react-router-dom, ZAP was able to find and navigate those routes successfully.

So it seems that ZAP’s spider might be better tuned to detect NavLink navigation in React SPAs compared to raw HTML buttons or anchors.

Has anyone else run into this? Is there a recommended way to make raw elements more discoverable by ZAP?