ClientSpider/Ajax Spider not able to find elements in web pages

[MERIEM MELLOUS]

Hello, 

I have few questions concerning ZAP clientspider and ajaxspider: 

• Do crawlers see by default elements in shadowroot ? Is there a way to do so ?
• When it comes to Websites that take few time to execute Js scripts in pages and construct elements  , is there a way to configure ClientSpider to wait until the web page gets loaded and elements can be interactive?

I also have an issue concerning  authenticated Spidering.

 In fact, In my automation plan I set loginUrl , username and password as GlobalScriptVars and I add and run  the standalone Graal.js script beofre adding the authentication script. My authentication is configured of type "client".

jobs:

- type: script

parameters:

action: "add"

context: mycontext

type: "standalone"

engine: "ECMAScript : Graal.js"

name: "setGlobalVars"

inline: |

var ScriptVars = Java.type("org.zaproxy.zap.extension.script.ScriptVars");

ScriptVars.setGlobalVar("test.loginUrl","https://test_url.com");

ScriptVars.setGlobalVar("test.username","myUser");

ScriptVars.setGlobalVar("test.password","mypassword");

- type: script

parameters:

action: "run"

context: mycontext

type: "standalone"

name: "setGlobalVars"

In my Zest script I use ZestAssignGlobalVariable in order to replace the value of the GlobalVar in a local variable so that I can automatically request the loginUrl. Yet my Zest script doesn't seem to replace  the value of the loginUrl. All scripts are running in the same context.  

  

{
"globalVariableName": "test.loginUrl",
"variableName": "loginUrl",
"index": 2,
"enabled": true,
"elementType": "ZestAssignGlobalVariable"
},

{
"windowHandle": "windowHandle1",
"browserType": "chrome",
"url": "{{loginUrl}}",
"capabilities": "window-size\u003D1200,800\n",
"headless": false,
"profilePath": "",
"index": 3,
"enabled": true,
"elementType": "ZestClientLaunch"
}

The error is:

An error occurred while trying to execute the Client Script Authentication script: invalid argument

(Session info: chrome=147.0.7727.116)

Build info: version: '4.43.0', revision: 'dd0f534'

Driver info: org.openqa.selenium.chrome.ChromeDriver

Command: [dd132c8361d6f173332c08b099f8eb4f, get {url={{loginUrl}}}]

Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 147.0.7727.116, chrome: {chromedriverVersion: 147.0.7727.137 (68ba233a543..., userDataDir:

/tmp/org.chromium.Chromium....}, fedcm:accounts: true, goog:processID: 665979, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName:

linux, proxy: Proxy(manual, http=127.0.0...., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000},

unhandledPromptBehavior: dismiss and notify, webSocketUrl: ws://localhost:30265/sessio..., webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true,

webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}

Session ID: dd132c8361d6f173332c08b099f8eb4f

org.openqa.selenium.InvalidArgumentException: invalid argument

(Session info: chrome=147.0.7727.116)

Build info: version: '4.43.0', revision: 'dd0f534'

Driver info: org.openqa.selenium.chrome.ChromeDriver

Command: [dd132c8361d6f173332c08b099f8eb4f, get {url={{loginUrl}}}]

Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 147.0.7727.116, chrome: {chromedriverVersion: 147.0.7727.137 (68ba233a543..., userDataDir:

/tmp/org.chromium.Chromium....}, fedcm:accounts: true, goog:processID: 665979, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy:

Proxy(manual, http=127.0.0...., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior:

dismiss and notify, webSocketUrl: ws://localhost:30265/sessio..., webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength:

true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}

Session ID: dd132c8361d6f173332c08b099f8eb4f

at org.openqa.selenium.remote.ErrorCodec.decode(ErrorCodec.java:169)

at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:142)

at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)

at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:223)

at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:216)

at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:174)

at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:606)

at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:374)

at org.zaproxy.addon.authhelper.internal.ZestAuthRunner.handleClient(ZestAuthRunner.java:78)

at org.zaproxy.zest.impl.ZestBasicRunner.runStatement(ZestBasicRunner.java:240)

at org.zaproxy.addon.authhelper.internal.ZestAuthRunner.runStatement(ZestAuthRunner.java:101)

at org.zaproxy.zest.impl.ZestBasicRunner.run(ZestBasicRunner.java:157)

at org.zaproxy.zest.impl.ZestBasicRunner.run(ZestBasicRunner.java:118)

at org.zaproxy.addon.authhelper.ClientScriptBasedAuthenticationMethodType$ClientScriptBasedAuthenticationMethod.executeZestAuthScript(ClientScriptBasedAuthenticationMethodType.java:264)

at org.zaproxy.addon.authhelper.ClientScriptBasedAuthenticationMethodType$ClientScriptBasedAuthenticationMethod.authenticate(ClientScriptBasedAuthenticationMethodType.java:287)

at org.zaproxy.addon.authhelper.internal.AuthenticationBrowserHook.browserLaunched(AuthenticationBrowserHook.java:50)

at org.zaproxy.zap.extension.selenium.ExtensionSelenium.lambda$getWebDriver$5(ExtensionSelenium.java:945)

at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)

at java.base/java.util.Collections$SynchronizedCollection.forEach(Collections.java:2359)

at org.zaproxy.zap.extension.selenium.ExtensionSelenium.getWebDriver(ExtensionSelenium.java:942)

at org.zaproxy.zap.extension.selenium.ExtensionSelenium.getWebDriver(ExtensionSelenium.java:636)

at org.zaproxy.addon.client.spider.ClientSpider$WebDriverProcess.<init>(ClientSpider.java:830)

at org.zaproxy.addon.client.spider.ClientSpider.getWebDriverProcess(ClientSpider.java:340)

at org.zaproxy.addon.client.spider.ClientSpiderTask.runImpl(ClientSpiderTask.java:115)

at org.zaproxy.addon.client.spider.ClientSpiderTask.run(ClientSpiderTask.java:81)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)

at java.base/java.lang.Thread.run(Thread.java:1474)

(Notice: I see when I use authentication of type script I dont have the issue but my crawling is not performed in an authenticated way but when I use client authentication I see that the auth script is called but I have this issue)

Can someone help me understand and fix the issue please?

Thank you for you time! 

[Simon Bennetts]

Hiya,

Replies inline:

Hello, 

I have few questions concerning ZAP clientspider and ajaxspider: 

• Do crawlers see by default elements in shadowroot ? Is there a way to do so ?

Not right now.

We're looking into it, but it wont be easy as the available APIs are not great.

If we do add any support then its likely to be to the client spider.

• When it comes to Websites that take few time to execute Js scripts in pages and construct elements  , is there a way to configure ClientSpider to wait until the web page gets loaded and elements can be interactive?

Thats the client spider "Initial Page Load Time".

There was me thinking it was clearly labelled ;)

 

Re the following script variable problem, I'll have to have a play and see wjhats going on...

Cheers,

Simon

[MERIEM MELLOUS]

Hello Simon,

Thank you for your response !

I ll wait to hear back from you once you've had the time to investigate.

Cheers ,

Meriem