Spider does not crawl through multiple POST forms

[martin...@jpower8.cz]

Hi, we try to use ZAP for scan of Spring Webflow based application. The flow transitions are realized as submit the forms by POST method (browser client point of view). 

After some tweaks, I set successfully the flow id and flow transition id variables as Structural Parameters.

But whatever combination of settings I tried (process forms, process POST forms), the ZAP spider was not able to crawl more than one step through the flow. 

Is there any setting I did not find, or is this a bug? Any reason to not crawl through POSTs ?

[Simon Bennetts]

Does the site make heavy use of Java Script?
If so you may need to use the Ajax Spider: https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsSpiderAjaxConcepts
Note that there is a problem with the Ajax Spider and the latest versions of Firefox, so you should either use an older version of Firefox or another browser until this is fixed.

Cheers,

Simon

[martin...@jpower8.cz]

Hi Simon, 

thank you for your quick reply. I tried Ajax Spider, but it does not work what I need.

1) it cant be limited to url subtree, so it just exits the URL I need it to scan, and crawles through some linked js and css resources

2) it does not POST the form (after approx 10 000+ requests) - I expect it will not work this way

I'm pretty sure normal Spider is what I need, but with this crawl through POST feature. Is there any reason to not support this?

Regards

Martin

Dne čtvrtek 15. prosince 2016 11:24:36 UTC+1 Simon Bennetts napsal(a):

[Simon Bennetts]

Which browsers did you try to use with the Ajax Spider?

Re the 'standard' spider, its not really designed to handle flow transitions as is typically does not maintain enough state information.
Can you provide a sanitized version of the forms in the flow, or maybe point to an example application that implements something similar?

Even if ZAP could spider the flow it would still need some configuration in order to attack the flow correctly.
It would be worth checking out the (alpha) Sequence add-on which allows you to configure ZAP to understand these sort of flows.

Cheers,

Simon

[thc...@gmail.com]

Hi.

Latest version of AJAX spider add-on allows to limit the spider scan to
a subtree. [1] (It can be updated from within ZAP. [2])

Another option of interest might be "Click elements once" [3] to allow
the spider to click/follow the same page/element more than once (i.e.
submit/follow chained forms).
(Also, "Select elements to click during crawl (table)" if your form is
not using buttons to submit the forms.)


The normal spider does submit forms but not in sequence, that might
explain the behaviour that you are seeing?



[1]
https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsSpiderAjaxScandialog#spider-subtree-only
[2] https://github.com/zaproxy/zap-core-help/wiki/HelpUiDialogsManageaddons
[3]
https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsSpiderAjaxOptions

Best regards.

On 15/12/16 11:31, martin...@jpower8.cz wrote:
> Hi Simon,
> thank you for your quick reply. I tried Ajax Spider, but it does not work
> what I need.
>
> 1) it cant be limited to url subtree, so it just exits the URL I need it to
> scan, and crawles through some linked js and css resources
> 2) it does not POST the form (after approx 10 000+ requests) - I expect it
> will not work this way
>
>
> I'm pretty sure normal Spider is what I need, but with this crawl through
> POST feature. Is there any reason to not support this?
>
> Regards
>
> Martin
>
>
>
> Dne čtvrtek 15. prosince 2016 11:24:36 UTC+1 Simon Bennetts napsal(a):
>>
>> Does the site make heavy use of Java Script?
>> If so you may need to use the Ajax Spider:
>> https://github.com/zaproxy/zap-core-help/wiki/HelpAddonsSpiderAjaxConcepts
>> Note that there is a problem with the Ajax Spider and the latest versions
>> of Firefox, so you should either use an older version of Firefox or another
>> browser until this is fixed.
>>
>> Cheers,
>>
>> Simon
>>
>> On Thursday, 15 December 2016 10:06:26 UTC, martin...@jpower8.cz

[martin...@jpower8.cz]

I'm using the option HtmlUnit in the GUI dialog. However our we plan to use ZAP as part of automated security scan in continuous delivery process, so we will use Client API called from Bamboo job running on server, without possibility to launch any browser there.

The option to make you our app available would be a long , as it is a property of our customer from banking sector ;-)

I would give try to the Sequence add-on, could you provide me the link, please?

Thanks a lot for your help.

Martin

Dne čtvrtek 15. prosince 2016 12:46:02 UTC+1 Simon Bennetts napsal(a):

[martin...@jpower8.cz]

Hi THC, 

as you "The normal spider does submit forms but not in sequence, that might  explain the behaviour that you are seeing? " - This is exactly what blocks me from use spider with success.

I could try new version of Ajax spider, thanks for the tip.

M.

Dne čtvrtek 15. prosince 2016 12:46:08 UTC+1 thc202 napsal(a):

[Simon Bennetts]

You can get the Sequence add-on from the ZAP Marketplace which is integrated with ZAP.
Just click on the .Manage Add-ons' button as per https://github.com/zaproxy/zap-extensions/blob/master/README.md

Cheers,

Simon

[martin...@jpower8.cz]

Hi,

I finally realized that ZAP Spider actually DOES crawl through post forms, but the problem was in not checked option Edit->Enable Session Tracking (cookie)". Without correct jsessionid  the webflow app was failing down instantly.

I check this on and my the Spider is now able to crawl complete spring-webflow app. 

However, I did not found any way to switch this on from web API, is there any way to make it switch on by default? Or did I miss something?

Regards.

Martin

Dne čtvrtek 15. prosince 2016 13:21:53 UTC+1 martin...@jpower8.cz napsal(a):

[thc...@gmail.com]

For the record, answer(s) in:
https://groups.google.com/d/topic/zaproxy-users/Vc_zp-1WLTg/discussion

Best regards.

On 19/12/16 12:21, martin...@jpower8.cz wrote:
> Hi,
>
> I finally realized that ZAP Spider actually DOES crawl through post forms,
> but the problem was in not checked option Edit->Enable Session Tracking
> (cookie)". Without correct jsessionid the webflow app was failing down
> instantly.
>
> I check this on and my the Spider is now able to crawl complete
> spring-webflow app.
>
> However, I did not found any way to switch this on from web API, is there
> any way to make it switch on by default? Or did I miss something?
>
>
> Regards.
>
> Martin
>
>
>
> Dne čtvrtek 15. prosince 2016 13:21:53 UTC+1 martin...@jpower8.cz napsal(a):
>>
>> Hi THC,
>>
>> as you "The normal spider does submit forms but not in sequence, that

>> might explain the behaviour that you are seeing? " - *This is exactly
>> what blocks me from use spider with success.*