I have to run traditional spider like 5-10 times to get all the authenticated endpoints crawling

[Sachin Verlekar]

Hi Simon,

Its great to e-meet you again after very long time. I have been experiencing this issue with my javascript based application where all the authenticated API endpoints in order to get all of those crawling, i have to run ZAP spider like 5 to 10 times. One spider run cannot comprehensively capture all the endpoints on sitemap. I was under the impression this issue may have been addressed in the newer versions but its the same all along for the very long time.

I request you to kindly provide me with an appropriate solutions for the same.

Thanks & Regards,

Sachin

[thc202]

Hi,

You will need to provide more details, which spiders are you using with
which configurations.

Best regards.

[Sachin Verlekar]

I am running traditional spider with all your default configurations. I would suggest you to please provide us with your proven recommended configs to be used on the traditional black color spider so that all the authenticated endpoints gets populated on sitemap. Other spiders havent been so useful for me TBH

thanks & regards,

Sachin

--
ZAP by Checkmarx: https://www.zaproxy.org/
---
You received this message because you are subscribed to the Google Groups "ZAP User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/zaproxy-users/f7343362-58de-4e14-abf3-b001dd282efd%40gmail.com.

[Simon Bennetts]

Hiya Sachin,

The traditional spider will not handle modern (javascript based) apps effectively. It may still be useful, but it will not be able to explore your application effectively.

You need to use either the AJAX spider or the Client spider.

If they have not been effective then you ned to investigate why. You can run them in "non headless" mode which means you will see the browsers. Try it with just a few browsers otherwise you will not be able to see whats going on.

You may find that they are just hitting a login page, in which case you will need to configure ZAP to handle authentication.

Cheers,

Simon

[Sachin Verlekar]

Hi Simon,

It will probably be surprising to you but I have observed traditional spider performing better than Ajax spider. I have seen ajax spider giving up early without crawling essential parts of the application. 

Only thing that is happening with traditional spider is that our entire application endpoints and API accept data in multi-part/formdata POST request but even though traditional spider is crawling more Authenticated API endpoints, it's considering it as normal POST request with POST parameters in body . Ajax can do proper identifying of data. If you could help me enhance anything here that would be great.

Thanks & Regards,

Sachin

To view this discussion visit https://groups.google.com/d/msgid/zaproxy-users/effcb40c-1844-4b45-99f3-be6355742cefn%40googlegroups.com.

[Sachin Verlekar]

Also client spider doesn't work at all for me.