AJAX spider not working

[Musoke]

Hi.

I have been cracking at automatically scanning my website. The site is built using Angular 6 and is a single page application. The URL https://mysite/ takes you to a login segment https://mysite/#/login. Inspecting the page returns only calls to JavaScript files which are loaded only once. This is why I opted to use the AJAX spider.

Attached:

• Zap logs. Both with JSON auth and script: https://pastebin.com/BVbFr56F < JSON auth. https://pastebin.com/VcSH5yPh < Script
  
• auth script and httpsender script
• screenshots of the context configs.

Details:

• Zap version: 2.8.0
• System: Debian 64bit
  
• Firefox: 60.7.0
  
• python 2.7
• jython installed

When the user logs in, a POST is sent with the credentials as a JSON to an external API server. The server responds with an Auth token in JSON. The user is taken to a new segment. https://mysite/#/dashboard. This doesnt show up in the http history. Data is loaded from the API server through GET requests made by this client together with the auth token. I intend to then automate all this in Jenkins without the plugin and without Selenium.

So far: 

• I have written three scripts:
• One to authenticate by getting credentials as params then sending them in the body of a POST. It then sets the token as a global variable.
• The second is meant to be a httpsender script. Gets the token and adds it in header of subsequent requests. (not tested cause am stuck on login)
• The last is to automate all this. I will use this in Jenkins plus the other scripts. (also not tested)
• I set up the proxy and captured the login page. Started the AJAX spider at the page https://mysite/.
  
• The spider kept writing random strings in the username password field. The history tab showed authentications being made but the spider was still stuck.
• I tried using the JSON-based auth and same problem. It sends the JSON in the body but still cant proceed to dashboard.

Questions:

• How can I get past the login page?
• Can the spider be pointed to the fields and forced to write specific strings in?
• If the fields show up in "inspect code" but not "view source", does this confuse the spider?
• if it's not possible to force the spider's input, how can I intercept responses from the server during the spider? This is because I noticed I can trick the client into proceeding to the dashboard by intercepting the response (when done manually) of a login. Even if the credentials were wrong. Am thinking, I can proceed thereafter with the access token, to get resources from the API server.
  
• Should I be enabling the forced user paddlock or not. and Why?
• I chose cookie based session management. The proxy captures no cookies or tokens, so is this even necessary? If so, which mgt type should i use?

[Simon Bennetts]

Hiya,

You are right to use the Ajax Spider instead of the standard one, as your site makes a lot of use of JavaScript.

However its always much easier to configure ZAP to handle authentication manually rather than going straight into automation.

Have a look at this FAQ: https://github.com/zaproxy/zaproxy/wiki/FAQformauth

Although it focusses on form based authentication the general principals are the same.

Proxy your browser through ZAP so that you can test authentication without having to launch the ajax spider.

Follow the Diagnosing Problems section and if you cant work out whats going on then follow up on this thread.

Re the session management - how is your application managing the session? ZAP will need to understand this, and if you dont know how it works then you wont be able to configure ZAP to understand it :)

Cheers,

Simon

[Musoke]

Hi Simon. Thanks for the reply. Am confused as to what with type I should use. There is no post data sent. The application sends a JSON text in the body upon login. Should I use JSON based auth then ?

[Musoke]

Hi. So I have used JSON. I’ve proxied like you told me and carefully followed the link. I entered the correct credentials and then chose the node to use for auth from the site map, created a user and forced user. Enabled the forced user button. As far as I can tell it should be working. I started the Ajax spider, picked a context and user but same thing.

The Ajax spider still enters random characters on the login page. And more so, it enters only the username field, ignoring the password field. Then it keeps bouncing around between (failed) logging in and the “forgot password” link. Am confused now.

Ps: session mgt only has two methods so since am not sure which, I tried both. Same result

[Musoke]

Tried again and failed to get the spider to do what I want. Any help here?

[Simon Bennetts]

Follow the steps on https://github.com/zaproxy/zaproxy/wiki/FAQformauth#diagnosing-problems

You said "As far as I can tell it should be working" and then you jumped straight to the Ajax Spider.

Go back to manual proxying: you have the configuration set up an forced mode enabled, so make a request to your app from your browser.

If you are now automatically logged in then great, its working.

If not then look at the history and try to work out whats going wrong:

• is there an authentication request?
• are the right credentials being sent?
• what is the response to the authentication request?
• is the session being set up correctly?
  

Dont jump to automation until you are sure its working when you try it manually.

[Musoke]

Should I set the authentication method to manual authentication while am doing the manual proxying or should it stay as JSON-based?

[Simon Bennetts]

It should stay JSON-based.

Manual authentication means you're authenticating manually, ie not using ZAP to perform the authentication.

[Musoke]

When setting the logged-in/out indicator, does zap look at the header or the body or both?

[Musoke]

So the authentication shows up in the history tab. When I log in incorrectly, zap notices the logged out indicator which is the JSON body of the response, then zap sends its own POST with the correct auth credentials. The issue is that this happens first before the browser finishes sending its own POST. I think the order should be reversed: Zap authenticating after a failed attempt. What am i doing wrong?

On Wednesday, 19 June 2019 10:25:02 UTC+2, Simon Bennetts wrote:

[Peter Hauschulz]

I'm just guessing without seeing your setup, but it sounds like you just need to change the logged in or logged out indicator to something that only flags when you want it to (in this case, after other POST have finished)...it doesn't sound like you're doing anything wrong! It can just take a few rounds of calibration to get right

[Musoke]

Hi Peter,

I understand what you mean. But the issue is I have so few options to work with. The flow goes like this:

1. User loads page -> GET req to mysite/#/login and response with a lot of JS scripts loaded only once. The response has a page title
   
2. User enters credentials -> POST req to api.endpoint/login with JSON body
1. If correct -> response with JSON body. Then many other calls to the endpoint.
2. if false -> response from endpoint with JSON body with the word error in it.

I feel like i only have an option to use the error word as an indicator for logged out. Because nothing specific shows that you are logged in, because all info is from the endpoint and it varies.

Is there a way to send the response body of the ZAP authentication back to the browser. this would solve my problem.

[Peter Hauschulz]

Ah right!

Sometimes there are header differences that also make it easy to tell when logged in or out, but anyway it sounds like that's not your main issue. 

I could have sworn that someone else here within the last few months had a similar question, about 'passing' a response from ZAP to the browser...keep searching around, there must be something!

What if the browser is spawned by ZAP? Does that involve the browser in a better way? I know you can put client actions in Zest scripts as well and control a browser that way, but I actually haven't needed to look into that personally yet....any chance some of that helps?