Best Approach to running both standard and AJAX spider

[Ruairi]

Hi,

I'm new to ZAP and looking for some advice.

I'm wondering what the best approach to running both a standard and AJAX spider, in order to maximise node discovery. Ie. I would like know if there is a "winner" from the following:

- running AJAX spider, then running standard spider

- running standard spider, then running AJAX spider

- running both spiders concurrently if there is no benefit to the two previous scenarios

I understand that there won't be a one-size-fits all approach here, due to the nature of web apps, but would like a general approach which is suited to exploring modern SPA applications.

I have done some testing on a local juiceshop, and found that the most nodes were discovered when running the AJAX spider followed by the standard spider, as opposed to the other scenarios I mentioned above. (I believe this was because URLs discovered by the AJAX spider were used as seeds for the standard spider).

If anyone with more experience has an opinion/advice on this, I would greatly appreciate any feedback.

Regards,

Ruairi

[Simon Bennetts]

Hi Ruairi,

Good question :)

The standard spider is a standard "crawler" which requests web pages and then analyses the HTML returned. Its pretty fast and works best for "traditional" web apps.

Modern apps tend to make heavy use of Java Script, which the standard spider does not understand.

To fully understand modern apps you really need a browser.

So the AJAX Spider works by launching browsers and clicking on things, filling out forms etc. It is much slower than the standard spider but works well with modern apps.

However .. it is still worth using the standard spider on modern web apps as it can still find some interesting things.

We test ZAP against the Google Security Crawl Maze and publish the results on https://www.zaproxy.org/docs/scans/crawlmaze/

You can see which of the spiders finds which pages :)

The Automation Framework ajaxSpider job (https://www.zaproxy.org/docs/desktop/addons/ajax-spider/automation/ ) has a "runOnlyIfModern" option.

This only works if you run the standard spider first and wait for the passive scan to finish.

ZAP can detect if an app is "modern" and will raise an informational alert for it - if that alert is present then the ajaxSpider job will run when you specific the "runOnlyIfModern" option.

I think this is a good generic approach if for some reason you cannot compare the options with any specific site.

Cheers,

Simon