HTTP Sender script and Authentication script in yaml file

[Richard DAmelio]

HTTP Sender script and Authentication script don't show up in the yaml file created when doing an automation plan.
How do you add authorization and scripts to the yaml?

[kingthorin+owaspzap]

https://www.zaproxy.org/docs/desktop/addons/script-console/automation/

[Richard DAmelio]

I downloaded an OpenAPI definition/Spec then imported it into the gui.
I configured the default context for Authentication.
For the scripts we're using a Jython template - Simple form.
We have the Python Scripting Add-on installed.
We have an Authentication script and a HTTP Sender script.
The scripts are loaded/enabled and using the Default context
We have a user/pass for Authentication
Everything works fine in the GUI when doing a manual scan and a automated scan.
We can also run the yaml that was created on the command line and it runs fine too.

The issue we have no is we download the stable image and are running that now with the same yaml but it doesn't seem to pick up the scripts.
We've moved the scripts and contexts directory into our /zap/wrk/ along with the config.xml and the yaml file.

We set the report dir:
reportDir: "/zap/wrk/"

We added the jython addon, which I see installing when we run.

Job addOns started
The updateAddons option has been disabled due to problems updating the framework and jobs while they are running
Downloading add-on from: https://github.com/zaproxy/zap-extensions/releases/download/jython-v12/jython-beta-12.zap
Add-on downloaded to: /home/zap/.ZAP/plugin/jython-beta-12.zap
Job addOns finished

jobs:
- parameters:
    updateAddOns: true      
  install:                  
  - jython                   
  uninstall: []              
  type: "addOns"             

Run the image:
docker container run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap.sh -cmd -autorun /zap/wrk/ssc-full-scan.yaml

But I can't get the script loaded.

If I add it to the yaml it complains about the engine?

I get
Unexpected error accessing file /zap/wrk/ssc-full-scan.yaml : No such engine: python : jython - see log for details

Do we even need the config.xml from our ZAP install, Library/Application Support/ZAP/
Also do I need everything under scripts.
I can't seem to figure out how to get these scripts and authentication which is set for my default context into my conatiner.

[Richard DAmelio]

if I add the script like so it doesn't like "add" or Authentication,

- parameters:
  action: add                                                         # String: The executed action - available actions: add, remove, run
  #type: "Authentication"                                              # String: The type of the script
  type: script                                              # String: The type of the script
  engine: "python : jython"                                           # String: The script engine to use - can be used to override the default engine for the file extension
  name: "sscauthpy"                                                   # String: The name of the script, defaults to the file name
  file: "/zap/wrk/scripts/scripts/authentication/sscauthpy.py"

Automation plan failures:
    Action is required, but not specified. Following actions are valid: add,run,remove

Automation plan failures:
    Unrecognised job type: Authentication

[Simon Bennetts]

The stable docker image does not contain the latest versions of the add-ons, and so I suspect the automation framework add-on it uses does not currently support the features you need.

You can either try with the weekly or live images - if they work then that is likely to be the problem.

Or you can use the command given on https://www.zaproxy.org/docs/docker/about/#automation-framework :

• docker run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable bash -c "zap.sh -cmd -addonupdate; zap.sh -cmd -autorun /zap/wrk/zap.yaml"

Cheers,

Simon

[Richard DAmelio]

Thanks I will give it a try and get back to you.

[Richard DAmelio]

I had used the addonupdate on the stable and got the same results.
So i tried the weekly image and got an issue creating the plugin directory?

626 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/session
Failed to create directory /home/zap/.ZAP_D/session
628 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/dirbuster
Failed to create directory /home/zap/.ZAP_D/dirbuster
628 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/fuzzers
Failed to create directory /home/zap/.ZAP_D/fuzzers
629 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/plugin
Failed to create directory /home/zap/.ZAP_D/plugin

[Simon Bennetts]

Can you share the command you are using?

Obfuscating any sensitive information of course.

Cheers,

Simon

[Richard DAmelio]

sure, also the live image failed to download.
failed to register layer: Error processing tar file(exit status 1): write /home/zap/.gradle/caches/jars-9/c0363f4ef5a2cd8f84fa200edc5a72cf/groovy-ant-3.0.9.jar: no space left on device

docker container run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-weekly bash -c "zap.sh -cmd -addonupdate; zap.sh -cmd -autorun /zap/wrk/ssc-full-scan.yaml"

[Richard DAmelio]

Actually it's failing on creating all the dir

docker container run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-weekly bash -c "zap.sh -cmd -addonupdate; zap.sh -cmd -autorun /zap/wrk/ssc-full-scan.yaml"

Found Java version 11.0.14.1
Available memory: 1985 MB
Using JVM args: -Xmx496m
854 [main] INFO  org.parosproxy.paros.Constant - Copying default configuration to /home/zap/.ZAP_D/config.xml
1075 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/session

Failed to create directory /home/zap/.ZAP_D/session

1076 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/dirbuster

Failed to create directory /home/zap/.ZAP_D/dirbuster

1076 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/fuzzers

Failed to create directory /home/zap/.ZAP_D/fuzzers

1077 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/plugin

Failed to create directory /home/zap/.ZAP_D/plugin

Found Java version 11.0.14.1
Available memory: 1985 MB
Using JVM args: -Xmx496m

626 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/session
Failed to create directory /home/zap/.ZAP_D/session
628 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/dirbuster
Failed to create directory /home/zap/.ZAP_D/dirbuster
628 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/fuzzers
Failed to create directory /home/zap/.ZAP_D/fuzzers
629 [main] INFO  org.parosproxy.paros.Constant - Creating directory /home/zap/.ZAP_D/plugin
Failed to create directory /home/zap/.ZAP_D/plugin

[Simon Bennetts]

Have you looked at the error message?

"no space left on device" :)

Think thats one for you to sort out...

[Richard DAmelio]

I have 200GB available on my system,

[Simon Bennetts]

Is it a Mac?

There are fairly signifcant limits on Macs, I know I had to increase the space available for docker images on my MacBook.

https://docs.docker.com/desktop/mac/space/

[Richard DAmelio]

yes it is and I just freed up some space by getting rid of some older images.
I was able to download the live image and run the following command.

docker container run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-live bash -c "zap.sh -cmd -addonupdate; zap.sh -cmd -autorun /zap/wrk/ssc-full-scan.yaml"

Job report set reportDescription =
Automation plan failures:
    Action is required, but not specified. Following actions are valid: add,enable,disable,run,remove

[thc...@gmail.com]

The parameters of the script job are not properly indented, you need to
shift them to the right, e.g.:

- parameters:
action: "add"
type: "authentication"
engine: "python : jython"
name: "sscauthpy"
file: "/zap/wrk/scripts/scripts/authentication/sscauthpy.py"
name: "script"
type: "script"


Best regards.

>>>>>>>>> - docker run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable

[thc...@gmail.com]

Just noticed that the example is not shown correctly in the website…

[Richard DAmelio]

OK all cleaned up and it's running better now with one minor issue with
    Job script Script Engine: python : jython not found

I will start a new conversation for that.

[Richard DAmelio]

# Add jython addon
  - type: addOns            # Add-on management
    parameters:
      updateAddOns: true    # Currently disabled due to problems updating the framework and jobs while they are running
    install:                # A list of non standard add-ons to install from the ZAP Marketplace
    - jython                # Id for Python Scripting add-on

So why wasn't this installed?

[Richard DAmelio]

OK got jython installed using the following command

docker container run -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-live bash -c "zap.sh -cmd -addonupdate -addoninstall jython; zap.sh -cmd -autorun /zap/wrk/ssc-full-scan.yaml"

Add-on update check complete

Downloading add-on from: https://github.com/zaproxy/zap-extensions/releases/download/jython-v12/jython-beta-12.zap

Add-on downloaded to: /home/zap/.ZAP_D/plugin/jython-beta-12.zap

[Richard DAmelio]

So now that I have my scan running it does not seem to be scanning everything.
In the GUI I have the default context and under Authentication I have Script-based Authentication
I have a targetURL set and a Username and Password.

This targetURL is different then the url included in my context

I have a targetURL for authentication similar to https://0.000.000.000:443/api/com.###.####.######/api-tokens

The Target URL that I used when importing my openAPI spec was similar to https://0.000.000.000:443

I also have HTTP Authentication Session Management set for Session Management

I believe I have the sessionManagement set right in my yaml but not the authentication.
Also I have the url included in my context in my yaml but not the targetURL from the authentication

is there an example somewhere on how to add this to my yaml?
Can someone please help get this working?

    authentication:
      parameters: {}
      verification:
        method: "response"
        pollFrequency: 60
        pollUnits: "requests"
        pollUrl: ""
        pollPostData: ""

    sessionManagement:
      method: "http"              '
    users:                       
      - name: user1         
        credentials:             
          username: my-username
          password: my-password

[Simon Bennetts]

Getting ZAP (or any other tool) to work with authentication can be hard.

We can give you general advice and guidance but we cant give you specific help as we dont have access to your app.

We do have a load of videos which show authenticated scans working with ZAP in the desktop and in automation - search for "auth" in the Tags field on https://www.zaproxy.org/videos-list/

Also see:

• https://www.zaproxy.org/docs/authentication/ (Work In Progress)
• https://www.zaproxy.org/docs/docker/diagnosing-problems/#investigating-non-trivial-issues
• https://www.zaproxy.org/faq/how-can-zap-automatically-authenticate-via-forms/#diagnosing-problems

Basically you need to completely understand how you app handles authentication and session management.

You need to configure ZAP to drive your app in the right way.

If it doesnt work then you will need to work out _exactly_ whats going wrong - you'll find this much easier in the desktop if thats an option for you.

If you can tell us whats going wrong in enough detail then we'll hopefully be able to help further.

Cheers,

Simon

[Richard DAmelio]

OK thanks I'll check out the links you sent.
It does work fine in the desktop.
We want to duplicate what we did in the desktop, but now with the docker image and passing in the yaml so we can integrate it into our CI/CD pipeline.

[Simon Bennetts]

In that case see this section: https://www.zaproxy.org/docs/docker/diagnosing-problems/#investigating-non-trivial-issues :)

Cheers,

Simon

[Richard DAmelio]

Thank you.

Another question real quick,

I have a TargetURL set in my default context under authentication that points to a token api which we use in all our request headers and this works fine in the desktop.
https://9.###.###.###:443/api/com.###.####.system/api-tokens
Then we have an Endpoint for our OpenAPI spec when we import it.
This all works great in the desktop.

Now in my yaml I have the Endpoint for our OpenAPI spec, but not the TargetURL for the token.
Does this need to be in the yaml and if so were would it go, maybe under authentication:?

env:
  contexts:                      
  - name: "Default Context"      
    urls:
    - "https://9.###.###.###"    
    includePaths:                
    - "https://9.###.###.###.*"
    excludePaths: []

    authentication:
        parameters: {}
        verification:
           method: "response"
           pollFrequency: 60
           pollUnits: "requests"
           pollUrl: ""
           pollPostData: ""            

[Richard DAmelio]

OK everything is working now, thanks.