Ajax Spider fails to find all links

[ks k]

Hi there,

I am using Ajax spider to explore my web application and it works fine. But I found that some known links, which expected to be found, are not found by ajax spider. The scanning is started using API without UI. The only config for ajax spider is setting "Click Default Elements Only" as false. The links cannot be found are in <a> element.

 Is there any setting which helps ajax spider to find all links? 

And another question about ajax spider, does every scan by ajax spider always return the same result, if the configuration is kept unchanged? 

Thanks for the support 

[kingthorin+owaspzap]

Are you able to provide a full example of what isn't being found?

Have you checked if they're found by the traditional spider?

> And another question about ajax spider, does every scan by ajax spider always return the same result, if the configuration is kept unchanged? 

Yes, assuming the settings, starting location, and state of the application are the same.

[Aakash Gupta]

Here is what I would do:

1. You need to manually explore the app and export all the URLs found in the app.
2. Save those URLS in your context.
3. You need to use passive scan first
4. Finally run your ajax scan

[Simon Bennetts]

Examples of the links that are missed would be appreciated.

I am actually working on something that I think _should_ fix this problem, but I cant know for sure without examples, and no eta for release yet I'm afraid.

Manually exploring and importing those URLs is also worth a try, but definitely run the ajax spider as well (as suggested) otherwise if and when your app changes you wont pick up any new URLs.

Cheers,

Simon

[ks k]

Thanks all for the quick replies. 

The web application being scannned is a Angular SPA. The HTML content is generated by the browser, so traditional spider doesn't works well. 

I cannot share the exact html. but here is an example of the xpath of the missed link: /html/body/app-root/app-header/div/div/div/a

The link is in the href attribute in the <a> element.

From the user's point of view, it is an item of a drop down menu in the web page header. In order to click that missed link, the user has to:

1. Click an icon in the web page header, so that the hidden drop down menu will be shown
2. Click the item in the drop down menu and go to the target page

[Archana Mehta]

Hi All,

I am also facing similar issues. I am working on angular SPA application which has side menu bar with multiple items.

I have added ZEST authentication script which is working fine if I manually crawl through the application and perform active scan.

However, the spiders (traditional/ajax) are not able to discover the same items on the menu bar.

I am not sure if I am missing something.

Regards,

Archana

[QUENTIN GALLIOU]

Hi all,

I have the same issues with Vue.js and GWT. 

The spiders (traditional/ajax) are not able to discover the items on the menu bar.

Regards

Quentin

[Simon Bennetts]

Can you share stand alone examples?

Or point us to online examples?

Or just HTML / JS snippets?

We have to be able to reproduce problems in order to fix them.

Knowing that the spiders cannot crawl every single possible combination of HTML / JS doesnt really help at this stage.

Cheers,

Simon

[QUENTIN GALLIOU]

For vue.js, I have pwndoc (https://github.com/pwndoc/pwndoc) 

The only difference between my own version is the authentication method, I use CAS. 

Spiders don't capture URLs like /audits /vulnerabilities /data/* etc ....

Cheers,

Quentin

[QUENTIN GALLIOU]

Hi Simon,

Do you have enough information ?

Also, does Ajax Spider work with Shadow DOM?

Regards,

Quentin

[Simon Bennetts]

Hi Quentin,

The Ajax Spider works with the shadow DOM.

It doesnt actually interact with the DOM directly, it controls browsers and click on elements, fills in forms etc.

Are you suggestion https://pwndoc.github.io/pwndoc/#/ as a good target to try crawling?

I havnt tried that yet, but will try to do that asap :)

Cheers,

Simon

[QUENTIN GALLIOU]

Simon,

Ok thank's for your reply.

Yes, Pwndoc is one of my apps where the spider doesn't find URLs. 
Here are some URLs not found : /audits /vulnerabilities /data/*  ...

FYI : Pwndoc is easy to install, it uses docker-compose. :)

Cheers,

Quentin

[thc...@gmail.com]

Does the site link 3rd-party JavaScript scripts?

Best regards.

>>>>>>>> 1. Click an icon in the web page header, so that the hidden drop

>>>>>>>> down menu will be shown

>>>>>>>> 2. Click the item in the drop down menu and go to the target page

>>>>>>>>
>>>>>>>> On Friday, July 29, 2022 at 9:35:14 AM UTC+2 psi...@gmail.com wrote:
>>>>>>>>
>>>>>>>>> Examples of the links that are missed would be appreciated.
>>>>>>>>> I am actually working on something that I think _should_ fix this
>>>>>>>>> problem, but I cant know for sure without examples, and no eta for release
>>>>>>>>> yet I'm afraid.
>>>>>>>>>
>>>>>>>>> Manually exploring and importing those URLs is also worth a try,
>>>>>>>>> but definitely run the ajax spider as well (as suggested) otherwise if and
>>>>>>>>> when your app changes you wont pick up any new URLs.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Simon
>>>>>>>>>
>>>>>>>>> On Friday, 29 July 2022 at 07:24:45 UTC+2 aakash...@gmail.com
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Here is what I would do:
>>>>>>>>>>

>>>>>>>>>> 1. You need to manually explore the app and export all the

>>>>>>>>>> URLs found in the app.

>>>>>>>>>> 2. Save those URLS in your context.
>>>>>>>>>> 3. You need to use passive scan first
>>>>>>>>>> 4. Finally run your ajax scan

[QUENTIN GALLIOU]

Thc202,

No, the site doesn't link to 3rd-party JS scripts.

Best regards.

[QUENTIN GALLIOU]

Simon, thc202,

If you need any information, please don't hesitate to ask me. I'll be happy to help you solve this issue.

Best regards,

[Simon Bennetts]

Many thanks!

We should have enough info to make progress - the docker compose option should make this much easier to test :D

I'm focussing on other things right now, but its definitely on my list to look at asap.

And if anyone else fancies helping out with it then we can give all the advice and guidance needed.

Cheers,

Simon

[Tina]

Hi,

I would like to share a simple modification that helped me out while using the Ajax-Spider on a SPA.

I faced a similar issue and eventually added “configurationBuilder.crawlRules().clickElementsInRandomOrder(true);” to function createCrawljaxConfiguration() in SpiderThread (https://github.com/zaproxy/zap-extensions/blob/main/addOns/spiderAjax/src/main/java/org/zaproxy/zap/extension/spiderAjax/SpiderThread.java) (see also https://github.com/crawljax/crawljax/blob/master/core/src/main/java/com/crawljax/core/configuration/CrawlRules.java). Also, it helps in my case to use a relatively small Event (e.g. 50 ms) and Reload Wait (e.g. 2000 ms) and a sufficient Max Crawl Depth.

I think that this is the only important thing that I changed before reinstalling the spiderAjax add-on and running ZAP again.

Now, it finds all elements of the SPA that I am currently testing.

However, I somehow seem to be unable to reproduce the described issues regarding https://pwndoc.github.io/pwndoc/. As far as I can tell, audits, vulnerabilities and data under pwndoc seem to be found and added to the site tree both in my modified and standard ZAP version. I don't really know why I can't reproduce the problem you describe, sorry... :(

I hope that this may nevertheless help someone out…

Good luck,

Tina

P.S. I also tried to use CrawlRules’ addWaitCondition() and crawlHiddenAnchors() but that appears to not help in my case….

[Simon Bennetts]

Hi Tina,

Thanks for this feedback - its really helpful!

I must admit I thought we already supported the "clickElementsInRandomOrder" option, but I must have been thinking of the "Random value" option :/

We should look at supporting this asap.

I also tried the 'crawlHiddenAnchors()' option but that didnt help with the specific app I was testing either, I hope to find time to try it again soon with a wider range of apps.

Cheers,

Simon

[QUENTIN GALLIOU]

Hi, 

Thank you for your reply.
During your test on pwndoc, was ZAP able to crawl the audit items or the vulnerability items? In my case, the spider crawls the navbar but no more (So /vulnerabilities, /audits, /settings, data/collaborators but not /audits/XXXXXX/*).

Cheers,

[QUENTIN GALLIOU]

Hi,

I found a problem (about vue.js / SPA crawling in general?). The "for=" attribute (i.e. <label for="f_fabc0704-113a-484c-b70f-e90e04c48d92" ...> has an id part after "f_" that is generated every time you navigate to a page. If you reload a page, the id changes and Zap detects a new DOM even if it is in fact the same page. This creates a kind of crawling loop.

Is there a solution to this issue?

Cheers,

[Simon Bennetts]

Thanks for identifying this - thats very useful!

The AJAX spider is actually based on Crawljax https://github.com/crawljax/crawljax so any fix would need to be in their code.

Are you ok raising on issue in their repo or would you prefer us to do it?

Many thanks,

Simon

[Simon Bennetts]

Having discussed this with the team, it might be a bug in Crawljax or it might be a problem in our code.

I've raised this issue: https://github.com/zaproxy/zaproxy/issues/8045

You can subscribe to it to get any updates.

Cheers,

Simon

[QUENTIN GALLIOU]

Hi,

With pleasure.
If you need more information or help, don't hesitate to contact me . :)

FYI : I'm using the latest Zap weekly release (D-2023-08-29) and all my addons are up to date

Cheers,

nitek29 on github

[Satyajit Todankar]

We are also facing same issue here. 

Ajax Spider is not exploring all links inside DOM. What is the solution to this problem. We want to setup CI/CD plugin of zap but we cant rely on auto scan for sure. 

[Simon Bennetts]

Try out the Client Side Integration AJAX spider enhancement: https://www.zaproxy.org/docs/desktop/addons/client-side-integration/ajax-scan/

This is a temporary solution, but knowing if it helps will help us understand if this is a good direction for us to go in.

Cheers,

Simon