How to use requests collected via proxy as a base for ZAP scans

[Maciej Bała]

Hi there,

What I'm trying to do 

1. Run Application under test (AUT) in docker

2. Run OWASP ZAP in docker 

3. Run E2E tests of AUT with proxy

4. Collect requests sent and received during E2E tests in ZAP via proxy

5. Run ZAP scans, using E2E traffic as a base.

All is set, both AUT and ZAP up and running in the same Docker network.

Proxy seem to be working fine as well, running E2E on AUT initialises passive scans in ZAP log (I run it headless, so log is my only output)

When I run baseline or full scans later on, I don't see any URLs with AUT in tests report...and I know there should be some :) So it makes me think that active scanners are not using E2E gathered requests as a base. 
What am I doing wrong? 
I run scans e.g. like so:
docker exec -it owasp_zap zap-full-scan.py -t http://172.26.0.8:8082 -r testreport.html

Is is possible that reqests are not saved in ZAP DB?
One more thing - AUT has json based authentication, that's why it is important to me to use E2E requests that have fresh and valid authentication tokens :) I guess it should work, right? It does work when I try it with desktop ZAP.

If there's anyone that is willing to discuss it via slack/skype/teams please let me know :)

[Simon Bennetts]

Hiya,

Everytime you run ZAP in docker a new docker image gets created, which in turn will have its own ZAP db.

So if you proxy through ZAP in one image and then run a scan in another image then you'll have lost all of the trafgfic from the first image.

The packaged scans have a delay option - "-D secs" - the idea is you run the relevant scan with this option supplying a number of seconds greater than the time it takes your tests to run.

You then run your tests proxying them through ZAP while the scan has paused. The scan will then have access to all of your requests.

Cheers,

Simon

[Maciej Bała]

I was convinced that it was all in one docker; I'm using  docker exec to run active scan, I really don't think I'm initialising another one...there is no other ZAP docker instance when scan is running and I list images with docker ps.
Anyway, I'l try to implement it the way you suggested tommorow morning. Thank you for your help :) PS really liked ZAP in 10 series :)

[Simon Bennetts]

Can you paste in the exact docker commands you are using? Obfuscating anything sensitive of course.

And glad you liked the ZAP in 10 series :) Have you seen the Deep Dive one? https://www.zaproxy.org/zap-deep-dive/

Cheers,

Simon

[Maciej Bała]

I use yaml config file to run ZAP (it is correctly formated in .yaml)

owasp-zap:

image: owasp/zap2docker-stable
ports:
- 8081:8081
volumes:
- ./test:/zap/wrk
container_name: owasp_zap
networks:
- ion-processor
command: ["zap.sh", "-daemon", "-host", "0.0.0.0", "-port", "8081"]

and then I try to run scans with

docker exec -it owasp_zap zap-full-scan.py -t http://172.26.0.8:8082 -r testreport.html

[Maciej Bała]

Hi Simon,

I've tried to do it the way you suggested, but I'm having a hard time putting config to command line.  
docker run --net ion-processor -t owasp/zap2docker-stable zap-baseline.py -host 0.0.0.0 -port 8081 -t http://172.26.0.6:8082 -D 100

does not work, how should I pass proxy parameters? Within config file? 

[Simon Bennetts]

Ah ok. Yes, if you are using "docker exec" then it should work.

But you have an issue with your command line.

The -host and -port parameters are zap.sh parameters rather than zap-baseline.py parameters (confusing I know;)

The baseline parameters are shown if you use "-h" or here: https://www.zaproxy.org/docs/docker/baseline-scan/

So use "-P" for the port.

I dont think you need to change the host, but if you find you do then you would need to set it via " -z '-host 0.0.0.0' "

Cheers,

Simon

[Maciej Bała]

So you say my 1st solution should work...no idea why it does not :/

I'll try your suggestion once again, setting proxy port with -P.

One more thing - you suggested using -D to delay scans...but zap documentation says it's meant to delay passive scan, and I need it to delay active scan so zap DB could fill with input from E2E + proxy :/ Is it going to delay active scan as well? 

[Simon Bennetts]

Yes, everything is sequencial in the packaged scans.

The spiders run, we wait for any delay specified then we wait for the passive scan to finish.

The active scanner runs after that.

Cheers,

Simon

[Maciej Bała]

Ok, I think I got this, here's the command
docker run  -v *myLocalPath*:/zap/wrk/:rw --net ion-processor -p 8081:8081 -t owasp/zap2docker-stable zap-full-scan.py -P 8081 -t http://172.26.0.6:8082 -D 200 -d -z '-host 0.0.0.0' 

delay works and I think ZAP has collected requests from simple E2E, but I think my config is still mising something. Could you please take a look at debug log? It seems that active scan could not connect propertly.
I'm planing on adding context file later on and fine tune the command itself, just want to make sure the basic config  and connection is ok.

 Cheers

[Simon Bennetts]

TBH I'm not really sure whats going on here - the zap.log only tells part of the story.

What are you seeing going on?

Are your E2E tests successfully proxying through ZAP?

How long to they usually take, and are you sure they have completed in the 100 seconds delay you've set?

Cheers,

Simon

[Maciej Bała]

When I run the command listed above, but I don't run E2E I can see that ZAP script logs only few URLs ("Total of XX URLs")

When I run the command listed above, and I  do run E2E in that 100 seconds window, then  once scan is initialized I can see that ZAP script logs significantly more URLs. That makes me think, that proxy acually does work, and requests from E2E are saved in ZAP DB. If there is a way to validate it e.g. by logging requests sent by Active Scanner, please let me know :) 

Oh - E2E tests of AUT are much bigger, I'm only running a single feature file for now  to test the connection :) 

I would take it as a success and continue e.g. with adding context file, but that log does not look good at all :/ 
Anyway, I think we can end the thread here - as I said, running E2E makes URL count go up, so at least that part seem to be configured properly.

Thank you Simon, I really admire your job and what you do for all the newbies out here. Keep up the great work, all the best to you.

I don't want to take advantage of your kindness, nontheless, if you have any other hints regarding that matter please hit me up :) Cheers

[Maciej Bała]

on that matter Simon - running docker exec -it owasp_zap zap-full-scan.py - myParams on existing docker based ZAP instance logs Params: ['zap-x.sh' so  it is ZAP Headless with xvfb.
 That's why it looked to me like it is running another ZAP instance within existing one...because it does :)

czwartek, 23 września 2021 o 10:41:05 UTC+2 psi...@gmail.com napisał(a):