Inconsistence Scanning result and problem after implement OWASP ZAP in CI/CD

[Jun Fulcrum]

Hi everyone,

 

First of all, I am sorry that this will be a bit long as I am trying to explain my situation in detail.

I need some help on increasing the consistency and accuracy of the reports. I have been using OWASP ZAP automation framework to scan my webapp. However, the report that generate each time will have a different in numbers of instances for certain alerts.

For example, the images below are the comparison of the two latest scan report. The number of instances of each alert will have a large different like the CSP header and also the SQL Injection.

The yaml file that used on both scan is the same. Then, the webapp that scan by the OWASP ZAP is also automated to reset or rebuilt each time before the scan. The automation operation is run using Jenkin.

I have set the Jenkin to first clear the data in the database -> Build up the webapp WAR -> deploy WAR on Weblogic app server -> Run OWASP ZAP scan on it.

So, the webapp status in every scan would be the same.

 

The environment for the scan:

The webapp is deploy on the same machine using a Docker that running the OWASP ZAP. In simple word, the webapp is running on the Window and the OWASP ZAP is running in a Docker container.

After that, I have also try to deploy the webapp in the same docker container with the OWASP ZAP and run for a scan. The result is more consistent however it getting less instances than the previous one. The below two reports is in the same environment (webapp deployment and OWASP ZAP using same docker container).

I am aware that the result might affect by a little small changes from https://www.zaproxy.org/faq/why-can-zap-scans-be-inconsistent/. But I would like to know more about the factors that might affect the scan result other than the network speed.

So, below are a few questions that I would like to ask:

1.       What are the possible factors that that might affect the scan result other than the network speed?

·       Are the factors something that we cannot control (like the network speed)?

2.       Is there any suggestion on overcome those small changes of the environment in the yaml configuration? (For example: increase the scanning time/attack attempt)

 

I have attached the yaml file that used in each test below. (In the yaml file, those sensitive information has been redacted to ‘xxx’)

The reason I am trying to get a consistent report is because this OWASP ZAP scan is planned to implement in the GitLab CI/CD environment on daily basis. However, if the report that generated is different on each time, despite using the same configuration and environment, that mean my team would need someone to validate the new instances of the alerts. Then, in the case it is a false positive, they will need to configure the yaml file to set the new instances as false positive which happened most of the time. This repetitive process is taking a lot of time and I would avoid this.

If anyone have the experience with the workflow after implementing the OWASP ZAP to the CI/CD, please share with me as well.

Thank you.

Regards,

Jun

[Simon Bennetts]

Have you seen this FAQ?

https://www.zaproxy.org/faq/why-can-zap-scans-be-inconsistent/

Cheers,

Simon

[Jun Fulcrum]

Hi Simon, 

Thanks for the fast reply.

Yes, I have check on the FAQ and mention in the post above. 

However, I need more detail on other factors that may also causing the inconsistent. So perhaps I can overcome those environment issue.

Or is there any configuration in OWASP ZAP that may avoid/at least reduce the inconsistency in the scanning result?

Regards,

Jun

[Simon Bennetts]

Hi Jun,

The first thing to check is if the spiders find the same URLs each time, in particular the URLs you are getting different alerts on like SQL injection.

If the spiders sometimes dont find a URL then it wont get scanned those times, its as simple as that.

If that is the case then try playing with the relevant spider parameters.

Cheers,

Simon

[Jun Fulcrum]

Hi Simon, 

The URL found by the spider is always consistent. However, it is not for the spiderAjax. 

So, I have tried some configuration for spiderAjax before, which I have adjust the maxDuration to 0 (unlimited) and eventWait and reloadWait to become longer. However, the result for ajaxSpider is still not the same but it do increase the URLs that crawled by it. 

For example, the below image has show some logs for each scan, they are using the same config file. But the first one get 2679 URLs by spiderAjax and the second one get a 3190 URLs. 

May I get some advice on what to configure so that the spiderAjax is able to crawl a full URLs in my webapp?

Regards,

Jun 

[Simon Bennetts]

Er, sorry, I dont have any :/

The Ajax Spider is a complex beast, and based on a third party project - Crawljax.

It has it limitations I'm afraid.

Anyone else got any advice and guidance?

Cheers,

Simon

[Abir Ayed]

Hello , I have the same problem :) 

I run OWASP ZAP in my pipeline CI/CD in  Automation Framework mode  with a docker image  , the number of URL find in Ajax Spider is not the same so I have Inconsistence Scanning result  , you can find here my .yaml file ,THANK YOU :) 

---

env:

  contexts:

  - name: "dev"

    urls:

    - "${ENVIRONMENT}"

    excludePaths:

    - "URL"

    authentication:

      method: "json"

      parameters:

        loginPageUrl: "URL"

        loginRequestUrl: "URL"

        loginRequestBody: "{\"Login\":\"{%username%}\",\"Password\":\"{%password%}\"}"

      verification:

        method: "response"

        pollFrequency: 60

        pollUnits: "requests"

        pollUrl: ""

        pollPostData: ""

    sessionManagement:

      method: "script"

      parameters:

        script: /zap/wrk/management_session.js

        scriptEngine: "Graal.js"

    technology:

      exclude: []

    users:

        - name: "tester"

          credentials:

            password: "${LOGIN}"

            username: "${PASSWORD}"

  parameters:

    failOnError: false

    failOnWarning: false

    progressToStdout: true

  vars:

    LOGIN: ${LOGIN}      # Can be used as ${myVarOne} anywhere throughout the config

    PASSWORD: ${PASSWORD}

    ENVIRONMENT: ${ENVIRONMENT}

jobs:

- parameters:

    action: "add"

    type: "session"

    engine: "ECMAScript : Graal.js"

    name: "test"

    file: management_session.js

    target: ""

    inline: ""

  name: "script"

  type: "script"

- parameters:

    context: "dev"

    user: ""

    url: "${ENVIRONMENT}"

  name: "spider"

  type: "spider"

  tests:

  - onFail: "INFO"

    statistic: "automation.spider.urls.added"

    site: ""

    operator: ">="

    value: 40

    name: "At least 100 URLs found"

    type: "stats"

- parameters: {}

  name: "passiveScan-wait"

  type: "passiveScan-wait"

- parameters:

    scanOnlyInScope: true

    enableTags: false

    disableAllRules: false

  rules: []

  name: "passiveScan-config"

  type: "passiveScan-config"

- parameters:

    context: "dev"

    user: ""

    url: ""

  name: "spiderAjax"

  type: "spiderAjax"

  tests:

  - onFail: "INFO"

    statistic: "spiderAjax.urls.added"

    site: ""

    operator: ">="

    value: 100

    name: "At least 100 URLs found"

    type: "stats"

- parameters:

    context: "dev"

    user: ""

    policy: ""

  policyDefinition:

    defaultStrength: "medium"

    defaultThreshold: "medium"

    rules: []

  name: "activeScan"

  type: "activeScan"

- parameters:

    template: "risk-confidence-html"

    reportFile: "zap_report"

    reportDir: /zap/wrk

    reportTitle: "ZAP Scanning Report"

  sites:

  - ${ENVIRONMENT}

  risks:                            

      - high

      - medium

  name: "report"

  type: "report"

- parameters:

    template: "traditional-json"

    reportFile: "zap_report"

    reportDir: /zap/wrk

    reportTitle: "ZAP Scanning Report"

  sites:

  - '${ENVIRONMENT}'

  risks:                            

      - high

      - medium

  name: "report"

  type: "report"

[psiinon]

And I have the same answer: https://www.zaproxy.org/faq/why-can-zap-scans-be-inconsistent/

Cheers,

Simon

Les informations contenues dans ce courriel sont strictement confidentielles et réservées à l'usage de la ou les personne(s) identifiée(s) comme destinataire(s). Si vous n'êtes pas le destinataire visé, veuillez en aviser l'expéditeur immédiatement et le supprimer. Par ailleurs, toute copie ou divulgation des informations communiquées et/ou des documents attachés à l'ensemble des courriels est interdite.

This message is intended for the use of the addressee and may contain information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in this message is strictly unauthorized and prohibited.

--
You received this message because you are subscribed to the Google Groups "ZAP User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zaproxy-users/703aafea-89df-4ee1-8b0c-6c8d10e9f3fan%40googlegroups.com.

--

ZAP Project leader