Make ajax spider wait to load the webapp before crawling? (Through Python API)

[xiaozheng wu]

Hi,
I am currently running ZAP version 2.5.0, with the latest version of the Ajax spider add-on.
The problem i am having is that when I try to test my company's webapp using Ajax spider and Chrome, it starts spidering before the application is fully loaded.
This makes the spider stop earlier than it is supposed to and misses a bunch of information. 

Is there any workaround?
I am new to Zap and doing this in an internship, any help would be greatly appreciated :)

[Simon Bennetts]

Hi,

Yes, there is a workaround - dont start spidering your site until it has finished loading :)
ZAP just does what you tell it to do when you tell it to do it.
If you are automating ZAP via a script then you can either add a suitable 'sleep' command or you can poll a URL which indicates that your app has fully loaded, if you have a url like that.

Cheers,

Simon

[xiaozheng wu]

Thanks for the answer, I tried to do that, but it seems that the problem isn't with how long i wait before it starts spidering. Oddly enough, no matter the time I wait the application just shows the header bar, and when i try to click on a link it gives me a 404 not found error. This got me startled since the URL works perfectly on a chrome browser that I open up manually. However, whenever it is through the ajax spider selenium tool it won't load completely.

We are also using selenium for automated test cases for the functionality of the app and those seems to be working fine. Any suggestions to what might be the issue?

[Simon Bennetts]

I'm afraid not, but maybe someone else on this list will have some suggestions?
However if you have selenium test cases then why not proxy those through ZAP instead of (or as well as) using the Ajax Spider?
Real test cases are usually much better than automated spidering as they supply a good range of valid (and often invalid) input.

Cheers,

Simon

[thc...@gmail.com]

Hi.

Are all URLs in the spider scope (usually the same host)?
The AJAX Spider will block/ignore the requests that aren't, which might
well cause the behaviour that you are seeing.

Best regards.

> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/zaproxy-users/3dcdcf8e-a0f5-48b8-ae4f-8931f28c1f4c%40googlegroups.com
> <https://groups.google.com/d/msgid/zaproxy-users/3dcdcf8e-a0f5-48b8-ae4f-8931f28c1f4c%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[xiaozheng wu]

Thanks for the replies,
I tried adding the url to the context scope but it is still displaying the same issue.
Otherwise, is it possible to make the ajax Spider crawl on an already opened window? 

[thc...@gmail.com]

> I tried adding the url to the context scope but it is still displaying the same issue.

Did you specify to spider "in scope" when starting the spider?

You can check which requests are being excluded by adding the following
to log4j.properties file:
log4j.logger.org.zaproxy.zap.extension.spiderAjax.SpiderThread=DEBUG

(The file log4j.properties is in the ZAP's home directory. [1])

It would log to zap.log file and print to the console output something like:
> DEBUG org.zaproxy.zap.extension.spiderAjax.SpiderThread - Excluding request [http://example.com/] not on target site [somehost].

or, when spidering everything in scope:
> DEBUG org.zaproxy.zap.extension.spiderAjax.SpiderThread - Excluding request [http://example.com/] not in scope.

> Otherwise, is it possible to make the ajax Spider crawl on an already opened window?

It's not possible.


[1] https://github.com/zaproxy/zaproxy/wiki/FAQconfig

Best regards.

> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/zaproxy-users/aec0fc33-8992-478f-afa7-c8a641c4ef80%40googlegroups.com
> <https://groups.google.com/d/msgid/zaproxy-users/aec0fc33-8992-478f-afa7-c8a641c4ef80%40googlegroups.com?utm_medium=email&utm_source=footer>.