Zap Ajax Spider not detecting all links in OWASP Juice shop when using python Client.

[Anand M R]

Hi,

I have created a python script to automate the active scan of OWASP juice shop web app. The script is working. But the problem is that the AJAX spider is not detecting some urls like password change end point. Also the active scan report is not detecting all Vulnerabilities of OWASP Juice Shop. I will attach the scripts and other details. Authentication is success. The script also sends the jwt token in all request. Please Help.

Python code -

import os
from time import sleep, time
from zapv2 import ZAPv2, spider, ascan
import json
import requests
import urllib.parse
import csv
import pandas as pd

zap = ZAPv2(apikey="dd4lir5ksmjngvq95bepb0rnmr")
target_url = 'http://192.168.1.34:3000/'
# zap = ZAPv2(apikey=apikey, proxies={'http': 'http://127.0.0.1:8090', 'https': 'http://127.0.0.1:8090'})
context_name = "context-juiceshop"
context_id = zap.context.new_context(context_name)

#json based authentication and loads the JavaScript file(which sends jwt token in all requests) into ZAP
def loadscript():
    #set_up_token()
    login_url = "http://192.168.1.34:3000/rest/user/login"
    login_request_data = '{"email":"te...@test.com","password":"testtest"}'
    json_based_config = 'loginUrl=' + urllib.parse.quote(login_url) + '&loginRequestData=' + urllib.parse.quote(login_request_data)
    print(json_based_config)
    print(zap.authentication.set_authentication_method(context_id, 'jsonBasedAuthentication', json_based_config))
    print('Configured json based authentication')
    filename = "script3.js"
    scriptname = "jwt Sender"
    zap.script.load(
        scriptname=scriptname,
        scripttype="httpsender",
        scriptengine="Graal.js",
        filename="/".join((os.getcwd(), filename)),
        scriptdescription="Adds jwt token to each request",
        charset="UTF-8",
    )
    zap.script.enable(scriptname)
    print("Loaded", filename, end="\n\n")

# includes juiceshop in new context
def set_include_in_context():
    include_url = 'http://192.168.1.34:3000.*'
    zap.context.include_in_context(context_name, include_url)
    print('Configured include and exclude regex(s) in context')

def set_user_auth_config():
    user = 'Test User'
    username = 'te...@test.com'
    password = 'testtest'
    user_id = zap.users.new_user(context_id, user)
    user_auth_config = 'username=' + urllib.parse.quote(username) + '&password=' + urllib.parse.quote(password)
    zap.users.set_authentication_credentials(context_id, user_id, user_auth_config)
    zap.users.set_user_enabled(context_id, user_id, 'true')
    zap.forcedUser.set_forced_user(context_id, user_id)
    zap.forcedUser.set_forced_user_mode_enabled('true')
    print('User Auth Configured')
    return user_id
   
# uses traditional Spider
def start_scan(user_id):
    method=[]
    url=[]
    statusCode=[]
    scanID = zap.spider.scan_as_user(context_id, user_id, target_url, recurse='true')
    sleep(5)
    #print(scanID)
    while int(zap.spider.status(scanID)) < 100:
    # Poll the status until it completes
        print('Spider progress %: {}'.format(zap.spider.status(scanID)))
        sleep(1)
    print('Spider has completed!')
    #print(zap.spider.full_results(scanID))
    for x in zap.spider.full_results(scanID)[0]['urlsInScope']:
        #print(x)
        #break
        method.append(x['method'])
        url.append(x['url'])
        statusCode.append(x['statusCode'])
    dict2 = {'Method': method, 'Url': url, 'StatusCode': statusCode}
    #print(dict2)
    df = pd.DataFrame(dict2, columns=['Method', 'Url', 'StatusCode'])
    df.to_csv('spider.csv', header=True, index=False)
    #scanId = zap.ascan.scan(url=target_url, contextId=context_id, userId=user_id ,recurse=True, inscopeonly=None, method=None, postdata=True)
    ss=zap.ascan.scan_as_user(url=target_url, contextid=context_id, userid=user_id, recurse='true', scanpolicyname=None, method=None, postdata='true')
    while int(zap.ascan.status(ss)) < 100:
    # Poll the status until it completes
        print('Active progress %: {}'.format(zap.ascan.status(ss)))
        sleep(1)
    print("Active scan completed")

# uses Ajax spider    
def start_scan2(user_id):
    result1=[]
    statusreason=[]
    method=[]
    messageId=[]
    url=[]
    statusCode=[]
    scanID = zap.ajaxSpider.scan_as_user(context_name, 'Test User', url=target_url, subtreeonly='true')
    timeout = time() + 30
    sleep(5)
    while zap.ajaxSpider.status == 'running':
        if time() > timeout:
            break
        print('Ajax Spider status : ' + zap.ajaxSpider.status)
        sleep(2)
    print('Ajax Spider has completed!')
    #print(zap.ajaxSpider.full_results)
    #result1.append(zap.ajaxSpider.full_results)
    for x in zap.ajaxSpider.full_results['inScope']:
        statusreason.append(x['statusReason'])
        method.append(x['method'])
        messageId.append(x['messageId'])
        url.append(x['url'])
        statusCode.append(x['statusCode'])
    dict1 = {'StatusReason': statusreason, 'Method': method, 'MessageId': messageId, 'Url': url, 'StatusCode': statusCode}
    print(dict1)
    df = pd.DataFrame(dict1, columns=['StatusReason','Method', 'MessageId', 'Url', 'StatusCode'])
    df.to_csv('ajaxspider.csv', header=True, index=False)
    #scanId = zap.ascan.scan(url=target_url, contextId=context_id, userId=user_id ,recurse=True, inscopeonly=None, method=None, postdata=True)
    ss=zap.ascan.scan_as_user(url=target_url, contextid=context_id, userid=user_id, recurse='true', scanpolicyname=None, method=None, postdata='true')
    while int(zap.ascan.status(ss)) < 100:
    # Poll the status until it completes
        print('Active progress %: {}'.format(zap.ascan.status(ss)))
        sleep(1)
    print("Active scan completed")

def save_report():
    with open("scan-report.html", "w") as f:
        f.write(zap.core.htmlreport())

set_include_in_context()
loadscript()
user_id_response = set_user_auth_config()
#start_scan(user_id_response)
start_scan2(user_id_response)
save_report()

-------------------------------------------------------------------------------------------------------------------------------------------

Thanks

Anand.M.R

[Simon Bennetts]

Hiya,

We do know that the AJAX spider is not able to find all links in some modern web apps.

The Client extension can help: https://www.zaproxy.org/docs/desktop/addons/client-side-integration/ajax-scan/

BTW - dont use forced user mode in automation.

For modern web apps we strongly recommend that you use browser based authentication: https://www.zaproxy.org/docs/desktop/addons/authentication-helper/browser-auth/

Also, authentication auto detection is known to work well with Juice Shop: https://www.zaproxy.org/docs/authentication/auto-detection/

Cheers,

Simon

[Anand M R]

Thanks for the reply. I have another 2 doubts.

1) Is there any python API client for authentication helper or browser based authentication?.

2) I have done authentication for owasp Juice shop web app with authentication helper and using automation framework I created a plan which includes Ajax spider, active scan and report for the Authentication Test Context. Then I exported the plan as yaml file  and executed the following command from terminal - sudo /usr/share/zaproxy/zap.sh -cmd -autorun /home/kali/Desktop/test9.yaml. But it suddenly shows automation plan succeeded but no report created. But when I open the yaml file in zap automation framework GUI and play the plan, it will run. I will attach the screenshot of the terminal with his message. what is the problem with my command line?

Thanks

Anand.M.R

[Simon Bennetts]

Dont run ZAP with sudo.

ZAP should be run as a standard user and not root.

Firefox will not run as root, which means the AJAX Spider will not work.

Cheers,

Simon

[Priyam Pratap Singh (Devil)]

Hey Simon,

I'm facing an issue in Automation Framework on Amazon EC2 kali linux server. Can you please check the issue that I've posted in thread below : 

https://groups.google.com/g/zaproxy-users/c/afQPQRx71SY/m/kA66F1uTAAAJ