ZAP AF against Owasp juice shop - not finding many vulnerabilities.

[Håkon Wiland]

Hey, I am testing ZAP automatization framework against the Owasp juice shop locally, and I am having trouble finding "enough" vulnerabilities - which makes me thing the way I have configured it, might not be optimal. 

Zap version: https://hub.docker.com/r/zaproxy/zap-stable 

Juice shop version: https://hub.docker.com/r/bkimminich/juice-shop 

zap plan configmap:

apiVersion: v1

kind: ConfigMap

metadata:

name: zap-af-plan

data:

plan.yaml: |

env:

contexts:

- name: "juice-shop"

urls:

- "http://juice-shop"

includePaths:

- "http://juice-shop.*"

authentication:

method: "browser"

parameters:

loginPageUrl: "http://juice-shop/#/login"

browserId: "firefox-headless"

loginPageWait: 5

verification:

method: "poll"

sessionManagement:

method: headers

parameters:

Authorization: "Bearer eyJ0e..."

cookie: "token=eyJ0e..."

users:

- name: "te...@test.com"

credentials:

username: "te...@test.com"

password: "testing"

parameters:

failOnError: false

failOnWarning: false

jobs:

- name: "Spider"

type: spider

parameters:

context: "juice-shop"

url: "http://juice-shop"

maxDuration: 2

- name: "AJAX Spider"

type: spiderAjax

parameters:

context: "juice-shop"

user: "te...@test.com"

browserId: firefox-headless

maxDuration: 10

excludedElements:

- description: Logout

element: span

text: Logout

- name: "Passive scan wait"

type: passiveScan-wait

parameters:

maxDuration: 5

- name: "Active scan"

type: activeScan

parameters:

context: "juice-shop"

maxScanDurationInMins: 20

maxRuleDurationInMins: 5

threadPerHost: 2

delayInMs: 50

policyDefinition:

defaultStrength: "high"

- name: "Report HTML"

type: report

parameters:

template: "traditional-html"

reportDir: "${ZAP_OUT_DIR}"

reportFile: "zap-report.html"

- name: "Report JSON"

type: report

parameters:

template: "traditional-json"

reportDir: "${ZAP_OUT_DIR}"

reportFile: "zap-report.json"

Zap job:

apiVersion: batch/v1

kind: Job

metadata:

name: zap-af-juice-shop

spec:

backoffLimit: 0

activeDeadlineSeconds: 1800 # hard stop at 30 minutes

template:

spec:

restartPolicy: Never

containers:

- name: zap

image: zaproxy/zap-stable

command:

- sh

- -c

- |

RUN_TS=$(date +%Y-%m-%d_%H-%M-%S)

export ZAP_OUT_DIR="/zap/wrk/out/$RUN_TS"

mkdir -p "$ZAP_OUT_DIR"

zap.sh -cmd -autorun /zap/wrk/plan.yaml

resources:

requests:

cpu: "500m"

memory: "2Gi"

limits:

cpu: "2"

memory: "4Gi"

volumeMounts:

- name: plan

mountPath: /zap/wrk/plan.yaml

subPath: plan.yaml

- name: out

mountPath: /zap/wrk/out

volumes:

- name: plan

configMap:

name: zap-af-plan

- name: out

hostPath:

path: /Repos/zap-dast/zap-output

type: Directory

In the report generated, it gives me: 
high: 2 (2 instances of SQLi)

medium: 5 (Http header stuff)

low: 4 (more http header stuff)

informational: 3

Based on what i read in the doc on "Zap and juiceshop": "The only significant vulnerability that we are aware of which ZAP should be able to identify but cannot is the DOM XSS vulnerability in the Search box." - I would think zap should be able to find more vulnerabilitites. 

I am not sure how i can go about improving the scanner from here, except a few ideas:

- Might need to include more jobs or active scan rules? 

- Authentication is somehow not working correctly 

Any suggestions on how to improve my scanning results? 

Also: 

From the job logs, i can see that the spiders found 112 + 351 URLs, i would like to see a full list of all the urls i have found, but i cannot seem to find such list in the report? 
Job Spider found 112 URLs
Job spider finished, time taken: 00:00:18
Job spiderAjax started
Job spiderAjax found 351 URLs
Job spiderAjax finished, time taken: 00:00:36

Any advice on how to improve my current setup is appreciated. 

[Simon Bennetts]

Hiya,

I think you miised a key sentence on https://www.zaproxy.org/docs/testapps/juiceshop/#scanning :

• "Although Juice Shop has lots of vulnerabilities, many of them can be very challenging for a DAST tool to identify."

Its been a while since I've gone through all of the JuiceShop vulnerabilities, but I didnt spot any that I thought ZAP should have caught (other than the DOM XSS).

If anyone thinks ZAP should find a specific vulnerability in this app then just say here...

Cheers,

Simon

[Håkon Wiland]

Hey Simon, thanks for answering.

I see, it might be hard for the ZAP DAST to find all vulnerabilities in Juice shop, but I am a bit supriced that it did not find more. (there is a lot of them: https://owasp.org/www-project-juice-shop/ ).

I mean if it stuggles this much to find vulnerabilities in juice shop, how could i except it to be able to find real vulnerabilities in other applications? I want to implement Zap DAST at our company, but I need to somehow verify that it is capable of finding stuff. 

Is it possible to add additional rules to the active scan? Now I am just using the default config with defaultStrength: "high". 

Best regards,

Haakon

[Simon Bennetts]

Hi Haakon,

I'm a big fan of OWASP Juice Shop, but I think of it more as a training platform for manual testing.

There are a lot of vulnerabilities, but most of them are "logical" vulnerabilities that require context to understand.

Automated tools can find some of the underlying problems, but they will not typically be able to find logical issues.

For those you either need pentesters, or potentially AI based testing.

I'm not aware of any other DAST tools which claim to find most of the Juice Shop vulnerabilities.

You may notice that some vendors implement their own deliberately vulnerable apps in order to showcase their capabilities, e.g.

• https://ginandjuice.shop/ (Burp)
• https://brokencrystals.com/ (Bright)

Security is hard, and no one type of tool will find all vulnerabilities.

Ideally you should have a layered approach including DAST, SAST, SCA and pentesting. (In addition to things like a secure SDLC etc).

Pentesting (by experienced practitioners) is the most effective way to find the most vulnerabilities, but it is also the most expensive in both time and money. AI based pentesting is starting to encroach on that, but it is also expensive.

DAST is one of the most cost effective ways to find vulnerabilities that are exploitable.

Attackers will be running DAST scans against your apps. If you dont do the same then you wont know what they are finding.

Yes, you can add additional scan rules to ZAP.

The main release just has the "release" status scan rules, but you can install the beta and alpha ones from the ZAP Marketplace as well.

ZAP also supports script based rules - we have examples in the community scripts repo, and you can of course implement your own.

https://github.com/zaproxy/community-scripts

Cheers,

Simon