Testing ZAP against benchmarks

[psiinon]

We need to regularly test (and score) ZAP against the growing number of benchmark apps, such as:

• Wavsep: http://sourceforge.net/projects/wavsep/
• OWASP Benchmark: https://www.owasp.org/index.php/Benchmark
• Webseclab: https://github.com/yahoo/webseclab
  
• Google Firing Range: https://github.com/google/firing-range
• WIVET: https://github.com/bedirhan/wivet
• Watcher test cases: http://www.testcases.org/watcher/
• .. add your suggestions here;)
  

What I'd _love_ to have is a CI server set up that regularly runs ZAP against these apps with various scan profiles, produces an easy to consume summary and alerts if ZAP fails to find things it should or finds new things that it didnt.

The Wavsep testing page on the Wiki did have some examples of reports, but it looks like these havnt migrated so well from Google Code :/ https://github.com/zaproxy/zaproxy/wiki/TestingWavsep

Getting a CI server is not a problem - I can use Mozilla's AWS account for that ;)

Setting them up and getting the scoring working is another matter.

One option that I think could work well is to use Docker images of the benchmarks (and ZAP?).

I know other projects like w3af are already doing this: Andres has even created a Docker image for OWASP Benchmark: https://github.com/OWASP/Benchmark/pull/6

So we'd need to do things like:

• Identify and/or create docker images for selected benchmarks
• Document/script the easiest possible way to run ZAP against them in an automated way
• Document/script how to run ZAP against them with different scan profiles
  
• Generate scores for each benchmark
• Generate detailed reports for each benchmark
• Work out when results change (for better or worse) and generate alerts
  
• Document how to set this all up in a CI server (ideally with a script)
• Set is all up on AWS with alerts raised for result changes (I can do that, if eveything else is in place;)
  

so ... lots of work, and much of it can be worked on in parallel by different people.

Who would like to get involved in this??

You dont have to be an expert in security, ZAP, docker etc etc, just willing to get stuck in and learn!

Many thanks,

Simon

[Mário Areias]

Hi Simon,


I want to ask some questions, so the idea is to have a CI server running ZAP against difference servers or VMs with different scan/profiles? If that's the case, how we measure if an alert reported by ZAP is a Test Positive? Also, do you have any idea in how to present this information?

I did some work in the past to put ZAP running in CI, in an automated fashion. I used grunt-zaproxy plugin to do that, although this plugin doesn't have yet many options, it should be easy to improve it. If that helps, I can contribute with that.


Cheers,


Mário

[psiinon]

Yes, exactly that.

For wavsep I created a script which did the scoring: https://github.com/zaproxy/zaproxy/blob/develop/python/scripts/wavsep/wavsep.py
That also generates the 'pretty' reports, which arent currently displaying well in the wiki, linked off https://github.com/zaproxy/zaproxy/wiki/TestingWavsep :/

I cant see any real option apart from manually setting up the scoring.

However there is vaguely related issue relating to allowing specified alerts to be ignored: https://github.com/zaproxy/zaproxy/issues/1843
I'm looking at that now, and thinking of implementing something more generic that allows you to also specify alerts that are expected.
We could then use that for testing ZAP against benchmarks.
Then instead of writing a script you could run ZAP against the test app, manually flag the alerts as 'true' positives and false positives, then add the false negatives.
It then should be easy to write a simple addon that reruns ZAP against the app with the given configuration and generates a total score and more detailed reports.

I'll start an email thread off list regarding this - anyone else fancy getting involved??

Cheers,

Simon

[Wei Ma]

Hi Simon, please count me in.

Thanks.

[psiinon]

Great :)

I've created a Google Doc for this: https://docs.google.com/document/d/1zBg89q8eHyxZ_7GqTfb7v1-4L11RqSp2V7mH6YDgdd8/edit?usp=sharing

I've given write access to those who've expressed an interest - please go ahead and update it.
Everyone elose should be able to comment on it, and I'll give write access to anyone who asks :)

Many thanks,

Simon

[Zoltán Kornél Török]

Hi Simon!

I think we talked about this thread 2 or 3 weaks ago via IRC? Sorry to not connect earlier I was busy :(
So I got some knowledge about docker and wawsep and ZAP.

I would be happy to work on the script which compare the results. (Are we talking here a phython script or orther script language?).

I am looking forward your response.

Zoli

[psiinon]

Hi Zoli,

I wrote a python script some time ago for scoring wavsep, and I also started an add-on - both are linked from the Google doc.
To be honest I'm not sure what the best approach is.
Ideally whoever maintains the benchmark apps would also maintain the scoring scripts - thats what the OWASP Benchmark guys have done. But I also understand that its a lot of work :/

Another option would be to enhance the Context Alert Filters add-on (https://github.com/zaproxy/zap-extensions/wiki/HelpAddonsAlertFiltersAlertFilter) to allow the user to specify 'expected' alerts, and then report if they werent present. However adding a load of 'expected' alerts for a large benchmark app could be very time-consuming.

So maybe the best option would be to handle each app differently - OWASP Benchmark already has a ZAP scoring script, so we can help maintain that. The wavsep scoring script I wrote seemed to work well, so we can update that, and maybe encourage Shay (the wavsep author) to help keep it up to date?

Any thoughts?

Simon

[Mário Areias]

Simon,

I've updated the google docs with Travis CI answers to my questions. Could you please (and anyone else interested) review it? So we can make a decision whether use Travis CI or not :)

Cheers,

Mário

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[taz]

Hi Simon!

I think you are right, the last option is the best. I will checkout the scoring codes until Saturday, after that we could talk, how to proceed.

Cheers,
Zoli

You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/SNWxnOtsCo4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

[Mário Areias]

Hello everyone,

How is that going? In Google docs there is no update for almost one month :/. Are we blocked by something? Is there anything I could do to help to unblock?

Cheers,

Mário

[Zoltan Kornel Torok]

Hi everybody!

Last week I tried to run the existing python script, but I got some error message. I haven't had enough time to find what the problem is.

However I think of this related to the scoring script:
- modify the python script to become fully automatic. I read in the script documentation, that right now we have to manually browse and attack the test page, before extract the result. I am looking for an automation solution if that okay.
- I guess, after that we should configure the build to call this script
- and after that create other script, which compare the result with the expected results.

What do you think of that approach?

[psiinon]

Quick update on this project.

We've now got ZAP running against Wavsep using Docker on AWS :)
Its nearly completely automated, just need to do a few more tweaks, and a load of tidying up ;)
FYI the scripts are all under: https://github.com/zapbot/zap-mgmt-scripts a good example is https://github.com/zapbot/zap-mgmt-scripts/blob/master/wavsep/zap-weekly-vs-wavsep-1.5-RB-H-M.sh

You can see the results here: http://zapbot.github.io/zap-mgmt-scripts/scans.html
Click on the scores to see the full reports.
And if you scroll down a recent report (like http://zapbot.github.io/zap-mgmt-scripts/reports/wavsep-1.5-weekly-RB-H-M.html) then you'll see a chart and full details of how long each rule took :)

We also show the number of errors logged.
Errors should really only be logged for unexpected conditions, so if we keep seeing errors then we should change the code to handle them (which could mean just changing the errors to debug messages, if they are not really important).

The plan is for scans against wavsep using different configurations to be run using cron jobs.

I'd also still like to automate the scanning of other apps.
For that we need Docker images (or online versions we can use) of the apps and scoring scripts.

The OWASP Benchmark has both (I created the Docker image;) but its still a pain to run so it might be a while before that can be completely automated.

Anyone interested in helping out with other apps, like Webseclab, Google Firing Range, WIVET or the Watcher test cases?

Cheers,

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/SNWxnOtsCo4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/SNWxnOtsCo4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.