ZAP docker run command example that uses both authentication and session scripts

[Federico Damián]

Hello ZAP community,

I have been 4 hours trying to get a docker run command that works going over the answers to different related questions in this group with no luck and was wondering if I could maybe get some help.

Context: I was able to get authentication and session management working in ZAP UI with an authentication script (ExampleAuthentication.js) and a session script (ExampleSession.js), together with a selenium script that I believe wouldn't be required in the docker mode. I successfully launched the browser with force user mode in the ZAP UI and it does everything to get the user logged in and shows the logged-in UI. I am now trying to get those configs working in the dockerized version.

The closest command that I used to get it working, which I believe successfully parses the authentication script is the following:

docker run -p 8080:8080                                \

           -v $(pwd):/zap/wrk/:rw                      \

           -t owasp/zap2docker-stable                  \

           zap-full-scan.py                            \

           -t https://domain.example.com/ \

           -z "-config script.scripts.name=ExampleAuthentication.js -config script.scripts.engine=Graal.js -config script.scripts.type=authentication -config script.scripts.enabled=true -config script.scripts.file=/zap/wrk/scripts/authentication/ExampleAuthentication.js" \

           -P 8080                                 \

           -c /zap/wrk/zap-casa-config.conf        \

           -x results-full.xml                     \

           -n /zap/wrk/contexts/Example.context \

           -U Fede

In this one, Authentication seems successful since I got 17036 [ZAP-IO-Server-1-1] INFO  org.zaproxy.zap.authentication.ScriptBasedAuthenticationMethodType - Loaded script:ExampleAuthentication.js.

However, I got afterwards session-related errors (which are expected since I didn't configure anything for the required session script):

18151 [ZAP-IO-Server-1-1] ERROR org.zaproxy.zap.extension.api.ContextAPI - null

java.util.ConcurrentModificationException: null

at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) ~[?:?]

at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]

at org.parosproxy.paros.model.Model.importContext(Model.java:575) ~[zap-2.14.0.jar:2.14.0]

at org.parosproxy.paros.model.Session.importContext(Session.java:1632) ~[zap-2.14.0.jar:2.14.0]

at org.parosproxy.paros.model.Session.importContext(Session.java:1537) ~[zap-2.14.0.jar:2.14.0]

Now I am trying to use the same pattern to add the session script, for which I tried just duplicating all those script.script.name, script.scripts.engine, etc, entries, but not only it doesn't work, but it also breaks the authentication parsing:

docker run -p 8080:8080                                \

           -v $(pwd):/zap/wrk/:rw                      \

           -t owasp/zap2docker-stable                  \

           zap-full-scan.py                            \

           -t https://domain.example.com/              \

           -z "-config script.scripts.name=ExampleAuthentication.js -config script.scripts.engine=Graal.js -config script.scripts.type=authentication -config script.scripts.enabled=true -config script.scripts.file=/zap/wrk/scripts/authentication/ExampleAuthentication.js -config script.scripts.name=ExampleSession.js -config script.scripts.engine=Graal.js -config script.scripts.type=session -config script.scripts.enabled=true -config script.scripts.file=/zap/wrk/scripts/authentication/ExampleSession.js"       \

           -P 8080                                 \

           -c /zap/wrk/zap-casa-config.conf        \

           -x results-full.xml                     \

           -n /zap/wrk/contexts/Example.context    \

           -U Fede

This last command returns authentication errors, so I think it messes up all the config. I was wondering if I could get a functional / operational command example showing how both an auth and a session script can be included as configs to see how the tool expects them to be provided.

Regards,

Fede

[Simon Bennetts]

Hi Fede,

Lets step back for a minute.

Have you tried Authentication Auto detection?

https://www.zaproxy.org/blog/2023-05-23-authentication-tester/

If we can get that to work then your life will be _much_ easier :)

Cheers,

Simon

[Federico Damián]

Hello Simon, thanks a lot for jumping in, it is much appreciated.

I haven't tried that feature before. I followed along with all the ADDO video tutorials that you created, which you surely know entail the other methods I mentioned.

I just tried that feature. If I leave the default Time to Wait, I get a PASSED status with a green Identified sign in all fields except for the Verification URL, which is left with no sign and the grey circle.

If I increase the Time to Wait number to, say, 25 seconds, which is a bit more time than what it takes to get into a steady state in which the Browser is fully logged in (it takes quite a bit to log into this app), I can see the browser logged in but the Status now says FAILED, and the verification URL is still empty.

Do you think it is a good idea to try this different feature to try to make the docker scan work? I really liked your willingness to make my life _much_ easier, but I spent all last week figuring out all those inner workings to configure a functional context and code the 3 scripts (auth, session, selenium), which I am totally fine discarding if there is a magic button that can do the thing, but I just want to avoid spending a full new week having to learn a new feature if I am close to making it work with what I got.

The Diagnostics has a lot of data about the app, so I am a bit hesitant just attaching it here.

Thanks,

Fede

[Simon Bennetts]

Hi Fede,

The ADDO videos were recorded in 2020, things have moved on significantly since then ;)

I strongly recommend having a go with auto-detection, especially as it already seems to work fairly well for you.

The fact that it failed on the Verification URL is actually good, thats the one part that should be easy for you to fix.

Presumably you worked out a verification URL before - just use that!

If you have time it would be interesting to see what ZAP didnt find it, but I understand if you have other pressures on your time.

So .. try specifying Browser Based auth, auto detect Session Management and the verification URL you used before.

Does that work reliably for you?

Cheers,

Simon

[Federico Damián]

Hi Simon,

It's pretty cool there have been improvements done since then. In full transparency, the logic behind the scanner auth, although pretty granular and adaptative, overcomplicated things quite much. Don't misunderstand me, I believe it's quite an achievement to get an OSS project as ZAP to what it became, but if I were the creator of it I believe I would have wanted people to tell me where things are not working very well. But it looks like I am speaking about the past, so I am more than happy to know the current approach is to automate the process.

We are kind of running against the clock here to get Google's approval on one of our products by getting clean scan results as specified by https://appdefensealliance.dev/casa/tier-2/ast-guide/dynamic-scan. This page even covers the usage of the traditional authentication features, and I believe from there, I followed the rabbit hole that got me into the ADDO videos..

I get your recommendation on the auto-detection, so I'll focus on that. I am more than happy going into a different channel to debug this if it helps the project. In my previous excersise, I didn't specify somithing such thing as a Verification URL. In my previous context I used the script-based authentication and that field appears grayed out. I configured the request and parsed the response in my script using JS, and then I configured the regexp to detect the scanner logged out. What is it that the scanner expects to have there configured (in the Verif. URL)?

Finally, if I get this context to work with browser-based auth configured, do you think I can export it and apply it in docker as it is?

Thanks,

Fede 

[Simon Bennetts]

Hi Fede,

Replies inline

 but if I were the creator of it I believe I would have wanted people to tell me where things are not working very well.

Oh believe me, we ask all of the time!

Sometimes people even give us some feedback ;)

The most recent time was in a dev focus questionnaire: https://www.zaproxy.org/blog/2023-12-04-development-focus-results/

You'll see authentication still came 2nd after all of the improvements we have made :)

 

We are kind of running against the clock here to get Google's approval on one of our products by getting clean scan results as specified by https://appdefensealliance.dev/casa/tier-2/ast-guide/dynamic-scan. This page even covers the usage of the traditional authentication features, and I believe from there, I followed the rabbit hole that got me into the ADDO videos..

I openned a simple doc issue on their repo in March: https://github.com/appdefensealliance/ASA/issues/9

That still hasnt been addressed so the chance of getting more significant changes made seem to be low :/

 

I get your recommendation on the auto-detection, so I'll focus on that. I am more than happy going into a different channel to debug this if it helps the project. In my previous excersise, I didn't specify somithing such thing as a Verification URL. In my previous context I used the script-based authentication and that field appears grayed out. I configured the request and parsed the response in my script using JS, and then I configured the regexp to detect the scanner logged out. What is it that the scanner expects to have there configured (in the Verif. URL)?

You went down the "handle it yourself" route: https://www.zaproxy.org/docs/authentication/handling-auth-yourself/

As you probably now know thats non trivial.

Re verification URLs see https://www.zaproxy.org/docs/authentication/finding-a-verification-url/

This thread is fine for going into more debugging details, unless you need to share anything which you dont want to make public, in which case you can email those bits directly to me.

Finally, if I get this context to work with browser-based auth configured, do you think I can export it and apply it in docker as it is?

One way or another, yes :)

Cheers,

Simon

[Federico Damián]

The autodetected verification URL, logged in pattern, and logged out pattern, should work fine. What seems to be wrong is the session management detection. I'll explain how it works in this app.

User logs in with an application/x-www-form-urlencoded AJAX request, and gets a JSON response with an authentication token in a certain key, let's call it chef. Now, to be authenticated, you user needs to have a session cookie called chef set with its value, which the app does using JS, not with a Set-Cookie header in the response. There is also a JSESSIONID cookie, but the app doesn't use it for authentication, as long as you send chef, with or without JSESSIONID, the server will authenticate you. In the previous example I had to set his chef cookie manually in the session management script because it is not set by the server through a Set-Cookie header.

With the authentication tester, what ZAP did was to configure a Header-based session management with the Authorization header, and the value of this chef JSON parameter. But the app doesn't use an authorization header, it uses a cookie.

What do you think would be the best to sort this out? I think I'll still have to use the script approach for session management based on this behaviour but maybe I'm wrong.

Regards,

Fede

[Federico Damián]

I am trying to use the Header-Based session management and set the header to Cookie, and value to chef={%json:data.chef%}

This seems to do what I need, but yet another problem I am facing is that this app does a check between an encoded user agent string inside that chef value and what the verification URL request (or any request) has in the user-agent.

For some reason, the browser authentication has:

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/119.0

But the request to the verification URL has:

user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 OPR/102.0.0.0

So this difference makes the check on the response of the verification URL fail because it doesn't match what the chef value has. I remember fixing this in the authentication script by using

post.getRequestHeader().setHeader(HttpHeader.USER_AGENT,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/119.0");

So it's quite a tricky app with a lot of customization. Does it still make sense to try getting autodetection work?

Regards,

Fede

[Federico Damián]

After spending some more time on this, I got the headers I needed (in the end I also needed an Authorization: header, so it's that one, the cookie and the user-agent) through the header-based session management screen configuration.

The last bit I am missing is that, my app not only sends those cookies / headers to the main domain to work, but also to a different API domain. I tried adding that other domain into the Context -> Include in Context screen to get the forced user mode inject them there too, but that doesn't happen. The app sends those requests with those fields empty.

Any hint on how I can get those there too?

Thanks,

Fede

[Simon Bennetts]

Hi Fede,

You should not be using forced user mode.

Thats only for manual testing and we strongly recommend you do not use it in automation.

You either need to handle authentication yourself or get ZAP to do it, without forced user mode!

Cheers,

Simon

[Federico Damián]

Hi Simon,

If I don't have to use the forced user mode, implying that if I open the browser with the toolbar button or manual explore nothing happens, then how should I continue to get the scan running either in the UI or docker?

I made configuration changes in the session management to try to make authentication and session management work (which I was testing with the forced user mode), how do I test it if I don't have to use it?

Regards,

Fede

[Simon Bennetts]

Hi Fede,

ZAP Authentication handling is currently designed for tools like the spiders and active scanner. There is no integration for manual testing.

I think the easiest way to test it is via the Automation Framework.

Create a context with the correct auth settings and then make a request via the requestor to an in-context URL specifying a valid user.

ZAP should make all of the necessary authentication requests which you can then examine.

If you have specified browser-based auth then it will launch a browser to do that. You can launch browsers with a UI if you want to see them in action.

Cheers,

Simon

[Federico Damián]

Well, we are forced to use the Docker packaged scan to get the Cloud Application Security Assessment (CASA) certification by providing the xml output.

I tried in the UI the spider and automated active scan with the context generated by the authentication tester and they seem to work fine. But when I tried to use the same context file in the docker-packaged scan I got an error (I am redacting the username and password): 

20073 [ZAP-IO-Server-1-1] ERROR org.zaproxy.zap.users.User - An error occured while decoding user from: 2059;true;ZmVkZQ==;6;USER_IN_BASE64~PASS_IN_BASE64~

Do you know what could be going on? The context was based on what was generated by the Authentication Tester.

Regards,

Fede

[Simon Bennetts]

Thats surprising :/

Can you detail the steps you took to generate the context file?

Cheers,

simon

[Federico Damián]

Hi Simon,

I am giving details for the sake of completeness that may not be relevant to reproduce the error but this is how I got the contexts file. Although this is a testing environment and a read-only user I am still hesitant to share creds in a public forum so I'll send them to you by email Simon.

Steps to Reproduce

- Run the Authentication Tester on ZAP 2.14.0 on MacOS X Ventura 13.6.

- Use the following configs:

  - Login URL: https://sdpen.strikedeck.com/login

  - Context: Authentication Test

  - Username: shared-on-separate-email

  - Password: shared-on-separate-email

  - Browser: Firefox Headless

- Test and save the context into a file named test.context inside a folder named contexts.

- In a terminal, change into the directory that has the contexts folder.

- Download the file https://appdefensealliance.dev/static/casa/tier-2/files/zap-casa-config.zip in that folder and decompress the zip file to get zap-casa-config.conf for the scan configuration.

- Use the casa config and the contexts folder with the context file in the following docker command:

docker run -p 8080:8080                      \

           -v $(pwd):/zap/wrk/:rw            \

           -t owasp/zap2docker-stable        \

           zap-full-scan.py                  \

           -t https://sdpen.strikedeck.com/  \

           -P 8080                           \

           -c /zap/wrk/zap-casa-config.conf  \

           -x results-full.xml               \

           -n /zap/wrk/contexts/test.context \

           -U "shared-on-separate-email"

The error returned will be:

26056 [ZAP-IO-Server-1-1] WARN  org.zaproxy.zap.extension.authentication.ExtensionAuthentication - No authentication method type found for ID: 6

26061 [ZAP-IO-Server-1-1] ERROR org.zaproxy.zap.users.User - An error occured while decoding user from: 0;true;REDACTED;6;REDACTED~REDACTED~

Regards,

Fede

[Simon Bennetts]

Hi Fede,

Sorry, I usually say something like "obfuscating anything sensitive" but I failed to this time.

I've tried those step locally against a test app and they appear to work fine for me.

I only tested that the auth specific arguments were all accepted and that the scan ran to completion rather than checking that all of the requests were authenticated.

One change I made was to use "-n test.context" instead of the absolute path - context files are expected to be in the CWD mapped to "/zap/wrk", specifying the full path didnt work for me.

However I've just noticed the error "No authentication method type found for ID: 6"

Thats surprising - Auth mthod 6 is for browser based authentication, and that should be present in the ZAP stable docker image.

When did you last pull that image?

I suggest pulling it and trying again.

Cheers,

Simon

[Federico Damián]

Hello Simon,

Thanks for the suggestions. I got some points to go over:

• It looks like I wasn't using the latest version of the docker image. For some reason, doing a docker pull without specifying "latest", didn't perform what I expected to be the default behavior of choosing the latest. This is what I see in my local registry, the one that says none is the one that was pulled without specifying the tag. Could it be the case there is an uploaded image that doesn't have a tag and I was getting that?

owasp/zap2docker-stable   latest        37613a7e2865   8 days ago      1.99GB

owasp/zap2docker-stable   <none>        547a4a270adb   5 weeks ago     1.97GB

• Using the latest image and the context file, the execution didn't show any message and kept running for several hours. Is it expected not to see any message until execution is completed?

• Checking the next day, the execution ended, starting with a list of all the checks that passed, and finishing with some error messages. I am attaching the console log here to see if I can get your help.

I see errors related to timeouts. I ran this on my computer, and I believe it may have happened that the internet connection might go down at some point during all those hours. So the next thing I am going to try is running this in an AWS EC2 and see how it goes, but any other suggestion is welcome.

Thanks,

Fede

[Simon Bennetts]

Theres this in the log:

• ERROR StatusConsoleListener Unable to write to stream /home/zap/.ZAP//zap.log for appender RollingFile

That could be because it ran out of disk space? Definitely worth checking that.

There doesnt appear to be an untagged image on https://hub.docker.com/r/owasp/zap2docker-stable/tags

FYI we recommend you stop using the owasp docker org, the supported options are listed on https://www.zaproxy.org/download/#docker

Cheers,

Simon

[Federico Damián]

My computer has 616 GB available of free space, so unless there is some restriction I may be ignoring for the container, it is unlikely it ran out of space.

I'll try in a VM with the image you recommended and let you know how it goes.

Is there a way to see what's going on during the scan without having to wait until it ends?

Regards,

Fede

[Federico Damián]

I'll correct myself. There was a space limitation in the docker volume of my computer, so I am cleaning it up. Nevertheless, the second run I am doing is in an EC2 instance as I mentioned, so it won't have that problem.

[Federico Damián]

Running this in a container leads to the same result, but after assigning it 1TB of storage and leaving it for 4 days, it is still scanning (with the docker command in the prompt and a blank output). The container size keeps growing, now the output of the docker system df command gives:

Containers      1         1         160.7GB   0B (0%)

And this is the only workload running on the server. Is it expected to have this container write so much data and take this long to finish for the full scan script?

Thanks,

Fede

[Simon Bennetts]

Hi Fede,

Its not possible to be able to tell in advance how long a ZAP scan will take.

If the target just has one page then it probably will take a few seconds.

If the target is a huge website then it will take much, much longer.

However 4 days is way longer than I would expect an active scan to take.

See https://www.zaproxy.org/faq/how-can-you-speed-up-scans/

Also note the link to the Mozilla blog post at the end of that page.

My first scan of the service mentioned in that post took 13 hours, and I thought that was way too long.

With a bit of configuration I got it down to 40 mins!

The first thing I recommend you do is to work out why the scan is taking so long.

That will be _much_ easier to do using the ZAP desktop rather than the docker image as you'll be able to see what is going on.

How many URLs is ZAP finding?

How fast is ZAP making requesting?

Lets try to narrow this down because right now we are in the dark..

Cheers,

Simon