AJAX Spider hungs out after some browsing

227 views
Skip to first unread message

Albert

unread,
Jan 11, 2016, 5:22:49 AM1/11/16
to OWASP ZAP User Group
Hi,

I am running ZAP using the three options for automation: ZAPlugin, Ant API, ZAPClient Java API.
In all cases I am trying to run the AJAX Spider againsts a site that contains a link in the main page that would authenticated a user when clicked.

When the AJAX Spider eventually clicks that link it logs in authenitcated.

However the Spidering process ends very early, before discovering the full site.

12817461 [Thread-246] INFO com.crawljax.core.CrawlController  - Received shutdown notice. Reason is Exausted
12817461 [pool-5-thread-2] INFO com.crawljax.browser.WebDriverBackedEmbeddedBrowser  - Closing the browser...
12817466 [pool-5-thread-2] INFO com.crawljax.browser.WebDriverBackedEmbeddedBrowser  - Interrupted while waiting for the browser to close. It might not close correctly

I can see this message on the ZAP log.

Anyone had experienced it before?

When using the JAVa client API. I am just doing this call:

System.out.println("Before");
zapClientAPI.ajaxSpider.scan(API_KEY, url, "False");
System.out.println("After");
Thread.sleep(1000);

Is that enough? Or should I do any extra context, and session set up?



thc...@gmail.com

unread,
Jan 11, 2016, 11:46:27 AM1/11/16
to zaprox...@googlegroups.com
Hi.

> Anyone had experienced it before?

I saw that happening (several times actually).
And that also happens when not using authentication. Not sure what the
problem is :/

How many browsers are you using? Does that also happen when using just one?


> Is that enough? Or should I do any extra context, and session set up?

Can the AJAX spider logout itself? (e.g. by clicking a logout link)
If that can happen you need to set up the context/authentication in ZAP
to ensure that the spider is logged back in.
(or, you can also exclude the logout URL from the spider)

Best regards.
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

Albert

unread,
Jan 12, 2016, 3:58:48 AM1/12/16
to OWASP ZAP User Group
Is it possible to have AJAX spider with authentication?

I have the URL in the context and the authentication URL identified and parametrized and marked as Form Base Auth Login Request in the context. 
That works for the normal spider but what I see with the Ajax spider is that it starts a browser and randomly clicks all elements. It eventually gets logged in becouse i have a link in the landing page that logs a user. 

How can I make sure the AJAX Spider is authenticated throught all the spidering session?

thc...@gmail.com

unread,
Jan 12, 2016, 4:46:00 AM1/12/16
to zaprox...@googlegroups.com
Yes, if the normal spider is able to authenticate then AJAX Spider
should be able as well.

One (important) thing that I forgot to mention is that the AJAX Spider
does not yet allow to specify the user, so you need to enable "Forced
User" mode [1] before starting it.
("Forced User" mode can also be enabled through the ZAP API)

Does that make any difference?

[1]
https://github.com/zaproxy/zap-core-help/wiki/HelpUiTltoolbar#--force-user-mode-on--off

Best regards.
> > <mailto:zaproxy-user...@googlegroups.com>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.

Albert

unread,
Jan 12, 2016, 9:30:42 AM1/12/16
to OWASP ZAP User Group
Hi thc,

I see the Spider starting but it does not log in immediatly even I have set the "forced user".
Can you take a look at the following snippet and tell me if that's whay you mean by setting the forced user before running the AJAX Spider?

Is that what you mean?

Set user:

userId = extractUserId(zapClientAPI.users.newUser(API_KEY, contextId, "TestUser"));
System.out.println("Testing with user id: " + userId);


StringBuilder userAuthConfig = new StringBuilder();
userAuthConfig.append("username=").append(URLEncoder.encode(username, "UTF-8"));
userAuthConfig.append("&password=").append(URLEncoder.encode(password, "UTF-8"));
String authCon=userAuthConfig.toString();

zapClientAPI.users.setAuthenticationCredentials(API_KEY, contextId, userId, authCon);

System.out.println("New user added.");
zapClientAPI.users.setUserEnabled(API_KEY, contextId,userId,"True");
System.out.println("User :  is now Enabled" );


System.out.println("Start spider as enabled user");



Set forced user:

zapClientAPI.forcedUser.setForcedUserModeEnabled(API_KEY, true);
zapClientAPI.forcedUser.setForcedUser(API_KEY, contextId,userId);


System.out.println("Before");
zapClientAPI.ajaxSpider.scan(API_KEY, url, "false");
System.out.println("After");
Thread.sleep(1000);
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

Albert

unread,
Jan 12, 2016, 10:20:38 AM1/12/16
to OWASP ZAP User Group
I am basically trying to find out how to solve the following problem:

If i scan with the normal spider using an authentication URL properly set:

The scan discovers a list of sub-nodes but most of them are a 302 HTTP Found with a redirect to the Login page. Which I guess means that the authentication is lost. How can I work around that? I fixed the params JSessionID and 2 other Cookies as session cookies. But still don't manage to persist the authentication so ti works for all the site. 

If is use the AJAX Spider the browser gets started but the login does not happen even if I set forced user. Should setting the forced user login into the site as first instance?

thc...@gmail.com

unread,
Jan 12, 2016, 10:36:51 AM1/12/16
to zaprox...@googlegroups.com
Yes, the steps provided previously are correct.


OK. I suspect the logged in/out indicators might need some tweaks, if
you are seeing redirects to login page even when spidering as a user
with the normal spider.

Are you setting both logged in/out indicators? And in the logged out
indicator are you also including the redirect (e.g. using the Location
header)?


Yes, the forced user mode should log-in the user immediately (if it's
detected as logged out).

Best regards.
> > > <mailto:zaproxy-user...@googlegroups.com>.
> > > For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "OWASP ZAP User Group" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send
> > an email to zaproxy-user...@googlegroups.com
> > <mailto:zaproxy-user...@googlegroups.com>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.

Albert

unread,
Jan 12, 2016, 12:05:59 PM1/12/16
to OWASP ZAP User Group
One think I am noticing is that when using the ZAP spider earch request/response subsequen pair has a different JSessionID. 

which my be the reason why at some point I lose the session and get logged out and therefore redirected via 302 to the login page. 

However when I do a normal manual navigation and monitor using LiveHttpHeaders i see that the JSessionID is the same through all the sesssion. 

Is possible that ZAP is starting a new Session per request?
>         >     > <mailto:zaproxy-users+unsub...@googlegroups.com>.
>         >     > For more options, visit
>         https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>
>         >     <https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>>.
>         >
>         > --
>         > You received this message because you are subscribed to the
>         Google
>         > Groups "OWASP ZAP User Group" group.
>         > To unsubscribe from this group and stop receiving emails from
>         it, send
>         > an email to zaproxy-user...@googlegroups.com
>         > <mailto:zaproxy-users+unsub...@googlegroups.com>.
>         > For more options, visit https://groups.google.com/d/optout
>         <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

thc...@gmail.com

unread,
Jan 13, 2016, 3:47:35 AM1/13/16
to zaprox...@googlegroups.com
It's possible but shouldn't happen that often and not per request.
ZAP should be using the same session (cookies) until it logs out itself
or the server expired it.

You should try use the "Forced User" mode with a browser proxying
through ZAP, to see how many times does ZAP authenticate while
manually navigating the site. That would mimic the spider(s) and allows
to inspect the request/responses more easily.

Best regards.
> > > > <mailto:zaproxy-user...@googlegroups.com>.
> > > > For more options, visit
> > https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>
> > > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>>.
> > >
> > > --
> > > You received this message because you are subscribed to the
> > Google
> > > Groups "OWASP ZAP User Group" group.
> > > To unsubscribe from this group and stop receiving emails
> from
> > it, send
> > > an email to zaproxy-user...@googlegroups.com
> > > <mailto:zaproxy-user...@googlegroups.com>.
> > > For more options, visit
> https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "OWASP ZAP User Group" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to zaproxy-user...@googlegroups.com
> > <mailto:zaproxy-user...@googlegroups.com>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.

Albert

unread,
Jan 13, 2016, 7:25:39 AM1/13/16
to OWASP ZAP User Group
What I see is the following behaviour related to the session:

When running the Spider with form authentication:

The Spider authenticates and discovers the top levels of the application but fails to go deeper:

Request http://www.example.com/root  Response 200 OK
Request http://www.example.com/root/level1_A  Response 200 OK
Request http://www.example.com/root/level1_A  Response 200 OK
Request http://www.example.com/root/level1_A  Response 200 OK
Request http://www.example.com/root/level1_A  Response 200 OK
Request http://www.example.com/root/level1_B  Response 200 OK
Request http://www.example.com/root/level1_B  Response 200 OK

So after discovering part of the site seems it loses Session tracking on some levels as redirects to login page with a 302. 

I can see that the request that return the 302 have changed the JSessionID and therefore are considered as not logged in!!!

The session is lost and redirects to the logging page with a 302 Found. Is there any reason why ZAP would change the JSessionID in the middle of the spidering process?


When browsing using Firefox being Proxied through ZAP the behaviour looks fine, the JSession cookies is the same through all the navigation process and I can access all the pages. 

I have the force user enabled in ZAP and the enable session tracking checked. 
>     >         >     > <mailto:zaproxy-users+unsub...@googlegroups.com>.
>     >         >     > For more options, visit
>     >         https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >         <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>
>     >         >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >         <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>>.
>     >         >
>     >         > --
>     >         > You received this message because you are subscribed to the
>     >         Google
>     >         > Groups "OWASP ZAP User Group" group.
>     >         > To unsubscribe from this group and stop receiving emails
>     from
>     >         it, send
>     >         > an email to zaproxy-user...@googlegroups.com
>     >         > <mailto:zaproxy-users+unsub...@googlegroups.com>.
>     >         > For more options, visit
>     https://groups.google.com/d/optout <https://groups.google.com/d/optout>
>     >         <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "OWASP ZAP User Group" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to zaproxy-user...@googlegroups.com
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

thc...@gmail.com

unread,
Jan 13, 2016, 7:41:19 AM1/13/16
to zaprox...@googlegroups.com
That's weird. I'd expect that behaviour only if those pages are not
considered to be "in context" (in which case ZAP would not even try to
authenticate nor use an authenticated session).

Are those pages in context?

Also, could you check the request
http://www.example.com/root/level1_A/level2_A

to see if it's sending any cookies? If it is, are they the same that
were sent for:
http://www.example.com/root/level1_A


> I have the force user enabled in ZAP and the enable session tracking
> checked.

You can use just "forced user" mode, that's what the spider uses (more
or less).

Best regards.

On 13/01/16 12:25, Albert wrote:
> What I see is the following behaviour related to the session:
>
> _When running the Spider with form authentication:_
> *Is there any reason why ZAP would change the JSessionID in the middle
> of the spidering process?*
>
> *
> *
> _When browsing using Firefox being Proxied through ZAP_ the behaviour
> <mailto:zaproxy-user...@googlegroups.com>.
> > > > <mailto:zaproxy-user...@googlegroups.com>.
> > > > For more options, visit
> > https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>
> > > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>>.
> > >
> > > --
> > > You received this message because you are subscribed to the
> Google
> > > Groups "OWASP ZAP User Group" group.
> > > To unsubscribe from this group and stop receiving emails
> from it,
> > send
> > > an email to zaproxy-user...@googlegroups.com
> > > <mailto:zaproxy-user...@googlegroups.com>.
> > > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "OWASP ZAP User Group" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to zaproxy-user...@googlegroups.com
> > <mailto:zaproxy-user...@googlegroups.com>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.
Reply all
Reply to author
Forward
0 new messages