Job spider error accessing URL https://www.example.com/abv status code returned : 403 expected 200

[nagender singh]

Hi 

I am able to run active scan successfully on GUI with script for authentication and HTTP sender script.

I recoded the same in 'Automation' tab in GUI and tried executing the same.

My YAML file looks like following:

---
env:
  contexts:
  - name: "Default Context"
    urls:
    - "https://example.com"
    - "https://example.com/dev/api"
    - "https://example.com/dev/api/test"
    - "https://example.com/*"
    includePaths:
    - "https://example.com"
    - "https://example.com/dev/api"
    - "https://example.com/dev/api/test"
    - "https://example.com/*"
    excludePaths: []
    authentication:
      method: "script"
      parameters:
        script: "/zap/wrk/scripts/authScript.py"
        scriptEngine: "jython"
        Extra_POST_data: ""
        Auth_URL: "https://example.com/fetchToken"
      verification:
        method: "response"
        loggedOutRegex: "Unauthorise|Forbidden|invalid|expired|Missing|UnauthorizedException|Unauthorized|IncompleteSignatureException"
        pollFrequency: 60
        pollUnits: "requests"
        pollUrl: ""
        pollPostData: ""
    sessionManagement:
      method: "cookie"
      parameters: {}
    technology:
      exclude: []
    users:
    - name: "user"
      credentials:
        Username: "us...@email.com"
        Password: "XyZ@2023"
  parameters:
    failOnError: false
    failOnWarning: false
    progressToStdout: true
  vars: {}
jobs:
- parameters:
    action: "add"
    type: "httpsender"
    engine: "jython"
    name: "applyToken.py"
    file: "/zap/wrk/scripts/applyToken.py"
  name: "script"
  type: "script"
- parameters:
    scanOnlyInScope: true
    enableTags: false
  rules: []
  name: "passiveScan-config"
  type: "passiveScan-config"
- parameters: {}
  name: "spider"
  type: "spider"
  parameters:
    user: "user"
  tests:
  - onFail: "INFO"
    statistic: "automation.spider.urls.added"
    site: ""
    operator: ">="
    value: 100
    name: "At least 100 URLs found"
    type: "stats"
- parameters:
    maxDuration: 600
    maxCrawlDepth: 10
    numberOfBrowsers: 1
  name: "spiderAjax"
  type: "spiderAjax"
  parameters:
    user: "user"
  tests:
  - onFail: "INFO"
    statistic: "spiderAjax.urls.added"
    site: ""
    operator: ">="
    value: 100
    name: "At least 100 URLs found"
    type: "stats"
- parameters: {}
  name: "passiveScan-wait"
  type: "passiveScan-wait"
- parameters:
    template: "risk-confidence-html"
    reportDir: "/zap/wrk/scripts/"
    reportTitle: "ZAP Scanning Report"
    reportDescription: ""
  name: "report"
  type: "report"

I am executing it using below command:

 docker container run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-weekly zap.sh -cmd -autorun /zap/wrk/active-v5.yaml -addoninstall jython

On GUI, the Active scan runs fine the, the spider proceeds when an API return 404.

However, with above command, after crawling through some of the URLs correctly and publishing the final report, I am getting below output in the end of scan:

Job spiderAjax found 50 URLs
Job spiderAjax test of type stats failed: At least 100 URLs found [50 < 100]
Job spiderAjax finished, time taken: 00:00:33
Job passiveScan-wait started
Job passiveScan-wait finished, time taken: 00:00:00
Job report started
Job report generated report /zap/wrk/scripts/2023-08-02-ZAP-Report-example.com.html
Job report finished, time taken: 00:00:00
Automation plan warnings:
        Job spider error accessing URL https://example.com status code returned : 403 expected 200
        Job spider error accessing URL https://example.com/dev status code returned : 403 expected 200
        Job spider error accessing URL https://example.com/dev/api status code returned : 404 expected 200
        Job spider error accessing URL https://example.com/dev/api/test status code returned : 403 expected 200

It would be really helpful if someone can help me to understand how can I achieve same behavior as GUI

Thank you!

Best Regards

Nagender