Selenium WebDriver & ZAP API for security testing of JavaScript Web Client

[Zolomon]

Hello all,

I'm testing a JavaScript based web application (running on node.js accessed through http://localhost:3000) which is intentionally loaded with security problems. I'm using Selenium WebDriver (to log in and navigate through the application flow) along with ZAP Java API (Reference: https://github.com/continuumsecurity/zap-webdriver).

Here is the logical flow:

1. Start ZAP in daemon mode, listening on port 8090

2. Run Selenium WebDriver test cases (using Proxy localhost:8090) to log in through http://localhost:3000/#login and traverse through the application

3. Spider a sample url (say http://localhost:3000/#expense/1118)

4. Scan the same url (i.e. http://localhost:3000/#expense/1118)

5. Report the alerts received

The problem is I'm not getting enough security problems (just 1 or 2) using this approach. Probably I'm doing something wrong. May be because I'm not using AJAX Spider for JS client? Is it a must to have step?

Also user needs to log on in the first page and then only he can access the other pages. I'm using Selenium WebDriver script to log on and keeping the page open during the spidering and scanning but not using any setAuthenticationMethod. Could that be a reason? 

Please help me figuring out the issue.

Thanks!

[Zolomon]

After step-2, I added the following steps (e.g. creating new context, include the attack url [http://localhost:3000/] to context, set formbasedAuthentication, set loggedIn indicator, create new user, set authentication credentials, enabled userid and finally doing ajaxSpidering, spidering and active scanning), but results were not satisfactory. Any help would be highly appreciated!

 

    // Create new context
    contextId=((ApiResponseElement)(context.newContext(ZAPPROXY_APIKEY,CONTEXT_NAME))).getValue();
    System.out.println("context "+ CONTEXT_NAME +" created; Id:"+contextId);

    // Include url to context
    context.includeInContext(ZAPPROXY_APIKEY, CONTEXT_NAME, "\\Q"+attackUrl+"\\E.*");
    System.out.println("URL "+"\\Q"+attackUrl+"\\E.*"+" added to "+CONTEXT_NAME);

    // Set Form based authentication
    System.out.println("Set Form Based Authentication...");
    setFormBasedAuthentication();
    System.out.println("Form Based Authentication is set!");

    // Set logged in indicator
    clientApi.authentication.setLoggedInIndicator(ZAPPROXY_APIKEY, contextId, "%5CQ%3Clogout%3E+%5CE");

    // Create new user
    userId = ((ApiResponseElement)(clientApi.users.newUser(ZAPPROXY_APIKEY, contextId, username))).getValue();
    System.out.println("Testing with user id: " + userId);

    // Set authentication credentials
    // Prepare the configuration in a format similar to how URL parameters are formed. This
    // means that any value we add for the configuration values has to be URL encoded.
    StringBuilder userAuthConfig = new StringBuilder();
    userAuthConfig.append("username=").append(URLEncoder.encode(username, "UTF-8"));
    userAuthConfig.append("&password=").append(URLEncoder.encode(password, "UTF-8"));
    String authCon = userAuthConfig.toString();
    System.out.println("Setting user authentication configuration as: " + authCon);
    clientApi.users.setAuthenticationCredentials(ZAPPROXY_APIKEY, contextId, userId, authCon);

    // Enable userId
    clientApi.users.setUserEnabled(ZAPPROXY_APIKEY, contextId, userId, "True");
    System.out.println("User "+username+" (Id:"+userId+") is now Enabled" );

    // Check if everything is set up ok
    System.out.println("Authentication config: " + clientApi.users.getUserById(contextId, userId).toString(0));

    // Ajax Spidering
    System.out.println("Ajax spidering starts");
    clientApi.ajaxSpider.scan(ZAPPROXY_APIKEY, attackUrl, "False");
    System.out.println("Current status = "+((ApiResponseElement)clientApi.ajaxSpider.status()).getValue());
    while (((ApiResponseElement)clientApi.ajaxSpider.status()).getValue().equalsIgnoreCase("running"))
    Thread.sleep(10000);

    System.out.println("Number of results = "+((ApiResponseElement)clientApi.ajaxSpider.numberOfResults()).getValue());
    //System.out.println("Results :: \n"+ ((ApiResponseElement)clientApi.ajaxSpider.results("", "")).getValue());
    System.out.println("Ajax spidering stops");

    // Spidering
    System.out.println("Spidering starts");
    clientApi.spider.scanAsUser(ZAPPROXY_APIKEY, attackUrl, contextId, userId, "", "");

    //System.out.println(((ApiResponseElement)clientApi.spider.results("")).getValue());
    System.out.println("Spidering stops");

    // Active Scanning
    System.out.println("Scanning starts");
    clientApi.ascan.scan(ZAPPROXY_APIKEY, attackUrl, "", "", "", "", "");
    System.out.println("Current status = "+((ApiResponseElement)clientApi.ascan.status("")).getValue());
    while (((ApiResponseElement)clientApi.ascan.status("")).getValue().equalsIgnoreCase("running"))
    Thread.sleep(10000);
    System.out.println("HTML report output");
    PrintWriter outputFile = new PrintWriter("results.html");
    outputFile.write(new String(clientApi.core.htmlreport(ZAPPROXY_APIKEY)));
    System.out.println("Scanning stops");  
   
   
    private static void setFormBasedAuthentication() throws ClientApiException, UnsupportedEncodingException {
    // Setup the authentication method
    String loginRequestData = "username={%username%}&password={%password%}";

    // Prepare the configuration in a format similar to how URL parameters are formed. This
    // means that any value we add for the configuration values has to be URL encoded.
    StringBuilder formBasedConfig = new StringBuilder();
    formBasedConfig.append("loginUrl=").append(URLEncoder.encode(loginUrl, "UTF-8"));
    formBasedConfig.append("&loginRequestData=").append(URLEncoder.encode(loginRequestData, "UTF-8"));

    System.out.println("Setting form based authentication configuration as: "+ formBasedConfig.toString());
    clientApi.authentication.setAuthenticationMethod(ZAPPROXY_APIKEY, contextId, "formBasedAuthentication", formBasedConfig.toString());

    // Check if everything is set up ok
    System.out.println("Authentication config: " + clientApi.authentication.getAuthenticationMethod(contextId).toString(0));
    }

[thc...@gmail.com]

Hi.

> The problem is I'm not getting enough security problems (just 1 or 2)
> using this approach. Probably I'm doing something wrong. May be because
> I'm not using AJAX Spider for JS client? Is it a must to have step?

It shouldn't be needed, if the tests at step 2 discover all the pages
(i.e. exercises all required logic paths of your application).

> Also user needs to log on in the first page and then only he can access
> the other pages. I'm using Selenium WebDriver script to log on and
> keeping the page open during the spidering and scanning but not using
> any setAuthenticationMethod. Could that be a reason?

That might be one of the reasons, ZAP's spider will not use user's
selenium session by default.

Best regards.

> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com
> <mailto:zaproxy-user...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[thc...@gmail.com]

For the AJAX spider you need to enabled forced user mode (using
forcedUser API), it can be disabled once finished.
The active scan should be started with scanAsUser.

If that still does not help, is the application being effectively
discovered/scanned, that is, the Sites tree representation of your
application is the expected? [1]


[1]
https://github.com/zaproxy/zap-core-help/wiki/HelpStartConceptsStructmods

Best regards.

[Zolomon]

Thanks for the response!

Question:

Is it possible to pass the selenium session to spider, ajax spider and active scan? I tried the following, but while doing ajax spidering, it was trying to log on using somewhat gibberish words (e.g. QzBaihv8h). Similar behavior was noted when forced user API was used along with formBasedAuthentication (please see below):

                String site = null;
                ApiResponseList responseList = (ApiResponseList)clientApi.core.sites();
                System.out.println("Number of sites = "+responseList.getItems().size());
                //this code just tries to get the exact String in the list of sites
                for (ApiResponse response : responseList.getItems()) {
                    String potentialSite = ((ApiResponseElement)response).getValue();
                    System.out.println("potentialSite = "+potentialSite);
                    if(potentialSite.contains("http://localhost:3000")) {
                        site = potentialSite;
                    }
                }
                System.out.println("site chosen:"+site);

                // Create Empty Session
                clientApi.httpSessions.createEmptySession(ZAPPROXY_APIKEY, site, "TestSession");

                // Print active sessions
                System.out.println("Active session: "+((ApiResponseElement)clientApi.httpSessions.activeSession(site)).getValue());

                // Set Active session
                ApiResponse activeSession = clientApi.httpSessions.setActiveSession(ZAPPROXY_APIKEY, site, "TestSession");

// Set Forced user clientApi.forcedUser.setForcedUserModeEnabled(ZAPPROXY_APIKEY, true); clientApi.forcedUser.setForcedUser(ZAPPROXY_APIKEY, contextId, userId);

[thc...@gmail.com]

ZAP should automatically create the corresponding session when proxying
the messages. [1]
If the session was correctly identified by ZAP (it's shown in the view
"sessions" of "httpSessions" API and has all the necessary cookies), you
can set it as active and ZAP will force its use when running the spiders
and active scanner.

(If you use the HTTP session you do not need to "scanAsUser" nor use
Forced User mode.)

The seed for the spiders (the initial request), should be one of the
main pages after the login (to prevent the spiders from interacting with
the login form). You can also exclude the login/logout from the spiders.


[1]
https://github.com/zaproxy/zap-core-help/wiki/HelpStartConceptsHttpsessions

Best regards.

On 13/04/16 06:46, Zolomon wrote:
> Thanks for the response!
>
> Question:
> Is it possible to pass the selenium session to spider, ajax spider and
> active scan? I tried the following, but while doing ajax spidering, it
> was trying to log on using somewhat gibberish words (e.g. QzBaihv8h).
> Similar behavior was noted when forced user API was used along with
> formBasedAuthentication (please see below):
>
>
>
>
> |
>
> Stringsite=null;
>
> ApiResponseListresponseList=(ApiResponseList)clientApi.core.sites();
> System.out.println("Number of sites =
> "+responseList.getItems().size());
> //this code just tries to get the exact String in the
> list of sites
> for(ApiResponseresponse:responseList.getItems()){
>
> StringpotentialSite=((ApiResponseElement)response).getValue();
> System.out.println("potentialSite = "+potentialSite);
> if(potentialSite.contains("http://localhost:3000")){
> site=potentialSite;
> }
> }
> System.out.println("site chosen:"+site);
>
> // Create Empty Session
>

> clientApi.httpSessions.createEmptySession(ZAPPROXY_APIKEY,site,"TestSession");

>
> // Print active sessions
> System.out.println("Active session:
> "+((ApiResponseElement)clientApi.httpSessions.activeSession(site)).getValue());
>
> // Set Active session
>

> ApiResponseactiveSession=clientApi.httpSessions.setActiveSession(ZAPPROXY_APIKEY,site,"TestSession");

>
>
>
>
>
> |
> // Set Forced user
> clientApi.forcedUser.setForcedUserModeEnabled(ZAPPROXY_APIKEY,true);clientApi.forcedUser.setForcedUser(ZAPPROXY_APIKEY,contextId,userId);
> |
>
>
>
>
>
>
> |
>
>
>
>
>
> On Tuesday, April 12, 2016 at 2:47:52 PM UTC+5:30, thc202 wrote:
>
> For the AJAX spider you need to enabled forced user mode (using
> forcedUser API), it can be disabled once finished.
> The active scan should be started with scanAsUser.
>
> If that still does not help, is the application being effectively
> discovered/scanned, that is, the Sites tree representation of your
> application is the expected? [1]
>
>
> [1]
> https://github.com/zaproxy/zap-core-help/wiki/HelpStartConceptsStructmods
> <https://github.com/zaproxy/zap-core-help/wiki/HelpStartConceptsStructmods>
>
>
> Best regards.
>
> On 12/04/16 09:03, Zolomon wrote:
> > After step-2, I added the following steps (e.g. creating new context,
> > include the attack url [http://localhost:3000/] to context, set
> > formbasedAuthentication, set loggedIn indicator, create new user, set
> > authentication credentials, enabled userid and finally doing
> > ajaxSpidering, spidering and active scanning), but results were not
> > satisfactory. Any help would be highly appreciated!
> >
>

[Zolomon]

Strange thing is that for this JS web client, I'm not getting any valid session; httpSessions/view/sessions API shows the following:

{"sessions":[]}

whereas for other websites such as webgoat, I'm getting this:

{"sessions":[{"session":["Session 0",{"JSESSIONID":{"comment":"","domain":"localhost","domainAttributeSpecified":false,"expired":false,"expiryDate":null,"name":"JSESSIONID","path":"/WebGoat/","pathAttributeSpecified":false,"persistent":false,"secure":false,"value":"D077EF622E6CE1DC49D26F72E8EDEA76","version":0}},"9"]}]}

Why is ZAP not able to get the session information for JS web client?

> <mailto:zaproxy-users+unsub...@googlegroups.com>.

[thc...@gmail.com]

Was the session token added to ZAP? Or, is one of the default tokens?

If not, ZAP will not create the session.

(You can add the session token(s) with the action "addSessionToken".)

Best regards.

> > <mailto:zaproxy-user...@googlegroups.com>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

> <mailto:zaproxy-user...@googlegroups.com>.

[Zolomon]

I believe ZAP should automatically create the corresponding session when proxying the messages. Hence no session token was added programmatically. For our javascript website, session information cannot be read ( action httpSessions.sessions shows {"sessions":[]} and AJAX Spider cannot run on logged in session), but for other websites such as WebGoat or BodgeIt, session information can be read ( action httpSessions.sessions shows some valid values and AJAX Spider can run on logged in sessions ).

I tried to use addSessionToken and forcefully added one session token to the JS site, but it didn't change the output of httpSessions.sessions action  ( it still shows {"sessions":[]}).

By the way, while login, our javascript web site performs a GET request call with username, password and session information on its request header, whereas for WebGoat or BodgeIt web sites, a POST request call is performed during log-in. 

Login Request For JavaScript application

GET /login?username=<username>&password=<password>&_=1460652051476 HTTP/1.1

Host: localhost:3000

Connection: keep-alive

Accept: application/json, text/javascript, */*; q=0.01

X-Requested-With: XMLHttpRequest

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36

Referer: http://localhost:3000/

Accept-Encoding: gzip, deflate, sdch

Accept-Language: en-GB,en-US;q=0.8,en;q=0.6

Cookie: lastUsername=test1; loggingLevel=3

Login Request For WebGoat application:

POST /WebGoat/j_spring_security_check HTTP/1.1

Host: localhost:5050

Connection: keep-alive

Content-Length: 33

Cache-Control: max-age=0

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

Origin: http://localhost:5050

Upgrade-Insecure-Requests: 1

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36

Content-Type: application/x-www-form-urlencoded

Referer: http://localhost:5050/WebGoat/login.mvc

Accept-Encoding: gzip, deflate

Accept-Language: en-GB,en-US;q=0.8,en;q=0.6

Cookie: JSESSIONID=1F8FDADC134DAE0038A250769ECA8C0C; lastUsername=test1; loggingLevel=3

TextView:

username=<username>&password=<password>

>     > <mailto:zaproxy-users+unsub...@googlegroups.com>.

>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

> <mailto:zaproxy-users+unsub...@googlegroups.com>.

[thc...@gmail.com]

Hi.

The problem is that ZAP does not known (it could guess?) the name of the
session token being used, that's why it needs human intervention to tell
the names.
(ZAP has some of the common token names added by default, though.)

If you add the token name before authenticating does it show the session
after the authentication?
(You should not need to create a session manually in ZAP.)

That's fine, how the authentication is performed does not affect
detection/creation of sessions in ZAP (as long it uses cookies to
identify the session).

Best regards.

On 14/04/16 19:01, Zolomon wrote:
> I believe ZAP should automatically create the corresponding session when
> proxying the messages. Hence no session token was added
> programmatically. For our javascript website, session information cannot
> be read ( action httpSessions.sessions shows {"sessions":[]} and AJAX

> Spider *cannot *run on logged in session), but for other websites such

> as WebGoat or BodgeIt, session information can be read ( action
> httpSessions.sessions shows some valid values and AJAX Spider can run on
> logged in sessions ).
>
> I tried to use addSessionToken and forcefully added one session token to
> the JS site, but it didn't change the output of httpSessions.sessions
> action ( it still shows {"sessions":[]}).
>
> By the way, while login, our javascript web site performs a GET request
> call with username, password and session information on its request
> header, whereas for WebGoat or BodgeIt web sites, a POST request call is
> performed during log-in.
>

> _Login Request For JavaScript application_
> *GET /login?username=<username>&password=<password>&_=1460652051476
> HTTP/1.1*

> Host: localhost:3000
> Connection: keep-alive
> Accept: application/json, text/javascript, */*; q=0.01
> X-Requested-With: XMLHttpRequest
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
> Referer: http://localhost:3000/
> Accept-Encoding: gzip, deflate, sdch
> Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
> Cookie: lastUsername=test1; loggingLevel=3
>

> _Login Request For WebGoat application:_
>
> *POST /WebGoat/j_spring_security_check HTTP/1.1*

> Host: localhost:5050
> Connection: keep-alive
> Content-Length: 33
> Cache-Control: max-age=0
> Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> Origin: http://localhost:5050
> Upgrade-Insecure-Requests: 1
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
> Content-Type: application/x-www-form-urlencoded
> Referer: http://localhost:5050/WebGoat/login.mvc
> Accept-Encoding: gzip, deflate
> Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
> Cookie: JSESSIONID=1F8FDADC134DAE0038A250769ECA8C0C; lastUsername=test1;
> loggingLevel=3
>
> TextView:

> *username=<username>&password=<password>*

> > > <mailto:zaproxy-user...@googlegroups.com>.

> > > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "OWASP ZAP User Group" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to zaproxy-user...@googlegroups.com

> > <mailto:zaproxy-user...@googlegroups.com>.

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-user...@googlegroups.com

> <mailto:zaproxy-user...@googlegroups.com>.