Zap Automation Framework - See all URLs found by spiders

[Anthony Robinson]

Hi,

When using the automation framework I was wanting to know how do you see the URLs that the spider and ajaxSpider find? I'm not using the desktop ZAP for my task so it would be good if there was a way to get the list of URLs displayed in my report. If this is possible, how do I do it? 

Thanks

[Alex K]

Hi,

I'd love to know the answer as well. Please, advise

Alex

[Simon Bennetts]

You can do this using a standalone script, eg https://github.com/zaproxy/community-scripts/blob/main/standalone/Traverse%20sites%20tree.js

You can configure scripts in the AF using the script job: https://www.zaproxy.org/docs/desktop/addons/script-console/automation/

Cheers,

Simon

[Alex K]

Wow! That was really fast! :) Thank you, Simon! I'll give it a try.

I must say I'm studying ZAP for last couple weeks and watched a lot of your videos and read a huge amount of your answers in this group and SoF. Thank you very much for your efforts!

[Simon Bennetts]

Thanks for the appreciation!

I cant promise to answer other questions that quickly, but we do try to answer them as quickly as we can...

[Alex K]

Simon,

While the standalone script works perfectly in automation under the desktop, I can't get it to work under docker :(

$ docker exec -ti zap-live bash

zap@e8e673c4f4c1:/zap$ zap.sh -autorun wrk/auto-script.yaml -cmd
Found Java version 11.0.16
Available memory: 3426 MB
Using JVM args: -Xmx856m
Job sessionManagement set method = cookie
Job sessionManagement set parameters = {}
Job Traverse_sites_tree set action = run
Job Traverse_sites_tree set type = standalone
Job Traverse_sites_tree set engine =
Job Traverse_sites_tree set name = Traverse_sites_tree.js
Job script started
Job: Traverse_sites_tree Start action: run
Job: Traverse_sites_tree Script with name: Traverse_sites_tree.js not found
Job script finished
Automation plan failures:
        Job: Traverse_sites_tree Script with name: Traverse_sites_tree.js not found

zap@e8e673c4f4c1:/zap$ ls -l wrk/Traverse_sites_tree.js

-rwxrwxr-x+ 1 556 10003 575 Sep  8 12:33 wrk/Traverse_sites_tree.js

zap@e8e673c4f4c1:/zap$ cat wrk/auto-script.yaml
---
env:
  contexts:
  - name: "example.com"
    urls:
    - "https://example.com"
    excludePaths: []
    sessionManagement:
      method: "cookie"
      parameters: {}
  parameters:
    failOnError: true
    failOnWarning: false
    progressToStdout: true
  vars: {}
jobs:
- parameters:
    action: "run"
    type: "standalone"
    engine: ""
    name: "Traverse_sites_tree.js"
  name: "Traverse_sites_tree"
  type: "script"

I've tried to add "userDir=/zap/wrk" config entry and tried to put the script under ~/.ZAP_D/scripts/scripts/standalone (within the container) - no luck.

I've even tried to run things under strace - java does not make any attempts to look up the script at all.

Any ideas?

[Simon Bennetts]

Hi Alex,

Thats because the script probably isnt available in docker with the command you've run.

You need to map a local drive as per https://www.zaproxy.org/docs/docker/about/#automation-framework

If you use the "$(pwd):/zap/wrk/:rw" option then your CWD will be mapped to /zap/wrk

You will need to change you plan to reference any scripts to their location under this directory.

Cheers,

Simon

[Alex K]

This is how I started docker:

docker run --name zap-live -v $(pwd):/zap/wrk:rw -t owasp/zap2docker-live bash

and here is the contents of /zap/wrk in the container:

zap@e8e673c4f4c1:/zap$ ls -al /zap/wrk
total 13552
drwxrwxrwx  12 556   500    4096 Sep  8 14:00 .
drwxr-xr-x   1 zap zap      4096 Sep  8 14:10 ..
-rwxrwxr-x+  1 556 10003     575 Sep  8 12:33 Traverse_sites_tree.js
-rwxr-xr-x   1 556   500     429 Sep  8 14:22 auto-script.yaml

As you can see, ZAP does see auto-script.yaml, so the mapping is correct, I guess.

[Simon Bennetts]

Ah ok.

You need to "add" a script before you can "run" it :)

[Alex K]

Just tried one more time. The script and yaml files are in the folder ($(pwd)) before running docker. But this didn't help:

$ ls -l
-rwxr-xr-x  1 alexkol spbgroup         429 Sep  8 17:22 auto-script.yaml
-rwxrwxr-x+ 1 alexkol Domain Users     575 Sep  8 15:33 Traverse_sites_tree.js

$ docker run --name zap-live -v $(pwd):/zap/wrk:rw -ti owasp/zap2docker-live bash

zap@fd0651222b54:/zap$ zap.sh -autorun wrk/auto-script.yaml -cmd

Found Java version 11.0.16
Available memory: 3426 MB
Using JVM args: -Xmx856m

1762 [main] INFO  org.parosproxy.paros.Constant - Copying default configuration to /home/zap/.ZAP_D/config.xml
2157 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/session
2158 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/dirbuster
2158 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/fuzzers
2159 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/plugin
Sep 08, 2022 2:55:08 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.

Job sessionManagement set method = cookie
Job sessionManagement set parameters = {}
Job Traverse_sites_tree set action = run
Job Traverse_sites_tree set type = standalone
Job Traverse_sites_tree set engine =
Job Traverse_sites_tree set name = Traverse_sites_tree.js
Job script started
Job: Traverse_sites_tree Start action: run
Job: Traverse_sites_tree Script with name: Traverse_sites_tree.js not found
Job script finished
Automation plan failures:
        Job: Traverse_sites_tree Script with name: Traverse_sites_tree.js not found

[Simon Bennetts]

Can you share your new plan?

[Alex K]

It didn't change:
zap@fd0651222b54:/zap$ cat wrk/auto-script.yaml

---
env:
  contexts:
  - name: "example.com"
    urls:
    - "https://example.com"
    excludePaths: []
    sessionManagement:
      method: "cookie"
      parameters: {}
  parameters:
    failOnError: true
    failOnWarning: false
    progressToStdout: true
  vars: {}
jobs:
- parameters:
    action: "run"
    type: "standalone"
    engine: ""
    name: "Traverse_sites_tree.js"
  name: "Traverse_sites_tree"
  type: "script"

[Simon Bennetts]

Then it wont work :P

You need to have 2 script jobs, one with an action of "add" and another with an action of "run" :)

The "add" job gives ZAP much more info, eg what type of job it is, where the file is etc.

The "run" job can then be called multiple times if you like with much less info.

[Alex K]

Oh boy.. Such an obvious solution.. Thank you so much!!

[Alex K]

Simon, I'm sorry to bother you - another weirdness which is (most chances) just my miss: under docker the traversing script is called two times for some reason. Here are the details:

$ docker run --rm --name zap-live2 -v $(pwd):/zap/wrk:rw -t owasp/zap2docker-live bash -c "zap.sh -autorun wrk/auto-test2.yaml -cmd; cat wrk/auto-test2.yaml; cat wrk/test.js"

Found Java version 11.0.16
Available memory: 3426 MB
Using JVM args: -Xmx856m

1702 [main] INFO  org.parosproxy.paros.Constant - Copying default configuration to /home/zap/.ZAP_D/config.xml
2072 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/session
2074 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/dirbuster
2074 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/fuzzers
2075 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/plugin
Sep 09, 2022 8:44:44 AM java.util.prefs.FileSystemPreferences$1 run

INFO: Created user preferences directory.

Job script set action = add
Job script set type = active
Job script set engine = ECMAScript : Oracle Nashorn
Job script set name = test.js
Job script set file = wrk/test.js
Job test set action = run
Job test set type = standalone
Job test set engine =
Job test set name = test.js
Job script started
Job: script Start action: add
Job script finished
Job script started
Job: test Start action: run
test
test
Job script finished

Automation plan succeeded!

---
env:
  contexts:
  - name: "test"
    urls:
    - "example.com"
    includePaths:
    excludePaths: []

  parameters:
    failOnError: true
    failOnWarning: false
    progressToStdout: true
  vars: {}
jobs:
- parameters:

    action: "add"
    type: "active"
    engine: "ECMAScript : Oracle Nashorn"
    name: "test.js"
    file: "wrk/test.js"
  name: "script"
  type: "script"

- parameters:
    action: "run"
    type: "standalone"
    engine: ""

    name: "test.js"
  name: "test"

  type: "script"

print("test");

[sonawan...@gmail.com]

Hello

I also worked on one HTTPSender script - which logs all the URLs

Please check post - https://groups.google.com/g/zaproxy-users/c/LoWFDfhIzh8/m/cVMaZHdXAwAJ

[Alex K]

Thank you! I've ended up with this simple function in HTTP Sender script:
function responseReceived(msg, initiator, helper) {
    print("*** " + initiator + " " + msg.getRequestHeader().getMethod().toString() + " " + msg.getRequestHeader().getURI().toString()+ " " + msg.getResponseHeader().getStatusCode());
}

Still wondering why the standalone function was called twice, but that doesn't really matter now.

[thc...@gmail.com]

It was not called twice, it was printed twice. (A mishandling of the
print statements.)

If you change the script to:
java.lang.System.out.println("test");

it will just be printed once.

Best regards.

[Alex K]

Mishandling means I used it in a wrong way or ZAP? If the former, could you pls explain?

[thc...@gmail.com]

A ZAP issue.

Best regards.

[thc...@gmail.com]

Raised as:
https://github.com/zaproxy/zaproxy/issues/7455

Best regards.