ajaxspider issue

[Joe G]

I believe the ajaxspider isn't properly crawling the modern site I'm trying to scan.  I've tried a few things to troubleshoot like adding all the URLs to the automation job and changing the include paths.  I'm brand new to zap, so feel free to respond like I'm 6. The command is being executed from a shell script on a linux vm but I see similar issue from the desktop client. 

Job output log failing with <100
./run_zap.sh
Starting ZAP Automation Plan...
Job spider/stats/null set name = Traditional Spider
Job spider/stats/null set type = stats
Job spider/stats/null set onFail = Info
Job spider/stats/null set statistic = automation.spider.urls.added
Job spider/stats/null set operator = >=
Job spider/stats/null set value = 100
Job spider adding test of type stats : Traditional Spider
Job spiderAjax set maxDuration = 5
Job spiderAjax set numberOfBrowsers = 2
Job spiderAjax set clickDefaultElems = true
Job spiderAjax set maxCrawlStates = 1,000
Job spiderAjax/stats/null set name = AJAX Spider
Job spiderAjax/stats/null set type = stats
Job spiderAjax/stats/null set onFail = Info
Job spiderAjax/stats/null set statistic = spiderAjax.urls.added
Job spiderAjax/stats/null set operator = >=
Job spiderAjax/stats/null set value = 100
Job spiderAjax adding test of type stats : AJAX Spider
Job report set template = traditional-pdf
Job report set reportDir = /opt/zap/ZAP_2.16.1/reports
Job report set reportTitle = Test Env Scan Report
Job passiveScan-config started
Job passiveScan-config finished, time taken: 00:00:00
Job spider started
Job spider requesting URL https://test-login.acme.com
Job spider requesting URL https://test-conn.acme.com
Job spider requesting URL https://test-buy.acme.com
Job spider requesting URL https://test-link.acme.com
Job spider requesting URL https://test-ins.acme.com
Job spider requesting URL https://test-ask.acme.com
Job spider requesting URL https://test-admin.acme.com
Job spider found 100 URLs
Job spider test of type stats passed: Traditional Spider [100 >= 100]
Job spider finished, time taken: 00:00:08
Job spiderAjax started
Job spiderAjax found 20 URLs
Job spiderAjax test of type stats failed: AJAX Spider [20 < 100]
Job spiderAjax finished, time taken: 00:00:49
Job passiveScan-wait started
Job passiveScan-wait finished, time taken: 00:01:17
Job report started
Job report generated report /opt/zap/ZAP_2.16.1/reports/2025-11-03-ZAP-Report-test-login.acme.com.pdf
Job report finished, time taken: 00:00:12
Automation plan succeeded!

my yaml:
env:
  contexts:
  - name: test environment
    urls:
    - https://test-login.acme.com
    - https://test-conn.acme.com
    - https://test-buy.acme.com
    - https://test-link.acme.com
    - https://test-inst.acme.com
    - https://test-ask.acme.com
    - https://test-admin.acme.com
    includePaths:
    - https://test-.*\.acme.com.*
    authentication:
      method: browser
      parameters:
        loginPageUrl: https://test-login.acme.com/
        loginPageWait: 10
        browserId: firefox-headless
        stepDelay: 0
        diagnostics: false
        steps: []
      verification:
        method: autodetect
        loggedInRegex: \Q 200 OK\E
        loggedOutRegex: \Q 401 Unauthorized\E
        pollFrequency: 60
        pollUnits: requests
        pollUrl: https://test-login.acme.com/apps/true
        pollPostData: ""
        pollAdditionalHeaders:
        - header: content-type
          value: application/json
    sessionManagement:
      method: headers
      parameters:
        Authorization: "Bearer {%header:authorization%}"
    technology: {}
    structure: {}
    users:
    - name: te...@yopmail.com
      credentials:
        password: 'redacted'
        username: te...@yopmail.com
  parameters: {}
jobs:
- type: passiveScan-config
  parameters: {}
- type: spider
  parameters: {}
  tests:
  - name: Traditional Spider
    type: stats
    onFail: INFO
    statistic: automation.spider.urls.added
    operator: '>='
    value: 100
- type: spiderAjax
  parameters:
    maxDuration: 5
    numberOfBrowsers: 2
    clickDefaultElems: true
    maxCrawlStates: 1000
  tests:
  - name: AJAX Spider
    type: stats
    onFail: INFO
    statistic: spiderAjax.urls.added
    operator: '>='
    value: 100
- type: passiveScan-wait
  parameters: {}
- type: report
  parameters:
    template: traditional-pdf
    reportDir: /opt/zap/ZAP_2.16.1/reports
    # reportFile: "{{env ZAP_REPORT_FILENAME}}" (set via shell script using -config)
    reportTitle: Test Env Scan Report

[kingthorin+zap]

If the 100 URLs test isn't relevant for you then remove it.

[Joe G]

Thanks for the response. As I said I'm new to zap and don't know if it's relevant. I assume it's part of the process of using the ajax spider which ideally I would like use, if I can ensure that it's crawling the site properly. Based on it not finding many urls I would think it is not. I was hoping for some ideas on how to get it working properly on my site, but maybe all is well with the traditional spider and I can remove the ajax scan completely as you suggested.

[Paul H]

Joe,

Can you give a brief description of what you mean by a modern website. i have also had problems with scanning my websites. In my case I have a c# backend and an angular frontend. I couldn't get the frontend urls to scan.

--
ZAP by Checkmarx: https://www.zaproxy.org/
---
You received this message because you are subscribed to the Google Groups "ZAP User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/zaproxy-users/44fb0368-a722-4839-87ab-55a9a3547422n%40googlegroups.com.

[Joe G]

My mention of modern website maybe isn't the best description, but the site is java based which is why I wanted to get the ajax spider working. I'm not a developer and can't really give a better explanation than that. I suspected that my issues might be token based as I'm trying to do an authenticated scan, and maybe there is some issue with the scanner not persisting the session. Once authenticated via the login page all the other URLs should have been discoverable, but I had to manually add them for the spider to scan them.  From what I gather, neither the traditional spider nor the ajax spider doesn't seem to be able to even do that.