ClientSpider/Ajax Spider not able to find elements in web pages

[MERIEM MELLOUS]

Hello, 

I have few questions concerning ZAP clientspider and ajaxspider: 

• Do crawlers see by default elements in shadowroot ? Is there a way to do so ?
• When it comes to Websites that take few time to execute Js scripts in pages and construct elements  , is there a way to configure ClientSpider to wait until the web page gets loaded and elements can be interactive?

I also have an issue concerning  authenticated Spidering.

 In fact, In my automation plan I set loginUrl , username and password as GlobalScriptVars and I add and run  the standalone Graal.js script beofre adding the authentication script. My authentication is configured of type "client".

jobs:

- type: script

parameters:

action: "add"

context: mycontext

type: "standalone"

engine: "ECMAScript : Graal.js"

name: "setGlobalVars"

inline: |

var ScriptVars = Java.type("org.zaproxy.zap.extension.script.ScriptVars");

ScriptVars.setGlobalVar("test.loginUrl","https://test_url.com");

ScriptVars.setGlobalVar("test.username","myUser");

ScriptVars.setGlobalVar("test.password","mypassword");

- type: script

parameters:

action: "run"

context: mycontext

type: "standalone"

name: "setGlobalVars"

In my Zest script I use ZestAssignGlobalVariable in order to replace the value of the GlobalVar in a local variable so that I can automatically request the loginUrl. Yet my Zest script doesn't seem to replace  the value of the loginUrl. All scripts are running in the same context.  

  

{
"globalVariableName": "test.loginUrl",
"variableName": "loginUrl",
"index": 2,
"enabled": true,
"elementType": "ZestAssignGlobalVariable"
},

{
"windowHandle": "windowHandle1",
"browserType": "chrome",
"url": "{{loginUrl}}",
"capabilities": "window-size\u003D1200,800\n",
"headless": false,
"profilePath": "",
"index": 3,
"enabled": true,
"elementType": "ZestClientLaunch"
}

The error is:

An error occurred while trying to execute the Client Script Authentication script: invalid argument

(Session info: chrome=147.0.7727.116)

Build info: version: '4.43.0', revision: 'dd0f534'

Driver info: org.openqa.selenium.chrome.ChromeDriver

Command: [dd132c8361d6f173332c08b099f8eb4f, get {url={{loginUrl}}}]

Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 147.0.7727.116, chrome: {chromedriverVersion: 147.0.7727.137 (68ba233a543..., userDataDir:

/tmp/org.chromium.Chromium....}, fedcm:accounts: true, goog:processID: 665979, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName:

linux, proxy: Proxy(manual, http=127.0.0...., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000},

unhandledPromptBehavior: dismiss and notify, webSocketUrl: ws://localhost:30265/sessio..., webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true,

webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}

Session ID: dd132c8361d6f173332c08b099f8eb4f

org.openqa.selenium.InvalidArgumentException: invalid argument

(Session info: chrome=147.0.7727.116)

Build info: version: '4.43.0', revision: 'dd0f534'

Driver info: org.openqa.selenium.chrome.ChromeDriver

Command: [dd132c8361d6f173332c08b099f8eb4f, get {url={{loginUrl}}}]

Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 147.0.7727.116, chrome: {chromedriverVersion: 147.0.7727.137 (68ba233a543..., userDataDir:

/tmp/org.chromium.Chromium....}, fedcm:accounts: true, goog:processID: 665979, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy:

Proxy(manual, http=127.0.0...., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior:

dismiss and notify, webSocketUrl: ws://localhost:30265/sessio..., webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength:

true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}

Session ID: dd132c8361d6f173332c08b099f8eb4f

at org.openqa.selenium.remote.ErrorCodec.decode(ErrorCodec.java:169)

at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:142)

at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)

at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:223)

at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:216)

at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:174)

at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:606)

at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:374)

at org.zaproxy.addon.authhelper.internal.ZestAuthRunner.handleClient(ZestAuthRunner.java:78)

at org.zaproxy.zest.impl.ZestBasicRunner.runStatement(ZestBasicRunner.java:240)

at org.zaproxy.addon.authhelper.internal.ZestAuthRunner.runStatement(ZestAuthRunner.java:101)

at org.zaproxy.zest.impl.ZestBasicRunner.run(ZestBasicRunner.java:157)

at org.zaproxy.zest.impl.ZestBasicRunner.run(ZestBasicRunner.java:118)

at org.zaproxy.addon.authhelper.ClientScriptBasedAuthenticationMethodType$ClientScriptBasedAuthenticationMethod.executeZestAuthScript(ClientScriptBasedAuthenticationMethodType.java:264)

at org.zaproxy.addon.authhelper.ClientScriptBasedAuthenticationMethodType$ClientScriptBasedAuthenticationMethod.authenticate(ClientScriptBasedAuthenticationMethodType.java:287)

at org.zaproxy.addon.authhelper.internal.AuthenticationBrowserHook.browserLaunched(AuthenticationBrowserHook.java:50)

at org.zaproxy.zap.extension.selenium.ExtensionSelenium.lambda$getWebDriver$5(ExtensionSelenium.java:945)

at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)

at java.base/java.util.Collections$SynchronizedCollection.forEach(Collections.java:2359)

at org.zaproxy.zap.extension.selenium.ExtensionSelenium.getWebDriver(ExtensionSelenium.java:942)

at org.zaproxy.zap.extension.selenium.ExtensionSelenium.getWebDriver(ExtensionSelenium.java:636)

at org.zaproxy.addon.client.spider.ClientSpider$WebDriverProcess.<init>(ClientSpider.java:830)

at org.zaproxy.addon.client.spider.ClientSpider.getWebDriverProcess(ClientSpider.java:340)

at org.zaproxy.addon.client.spider.ClientSpiderTask.runImpl(ClientSpiderTask.java:115)

at org.zaproxy.addon.client.spider.ClientSpiderTask.run(ClientSpiderTask.java:81)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)

at java.base/java.lang.Thread.run(Thread.java:1474)

(Notice: I see when I use authentication of type script I dont have the issue but my crawling is not performed in an authenticated way but when I use client authentication I see that the auth script is called but I have this issue)

Can someone help me understand and fix the issue please?

Thank you for you time!