Very large Web application: Mixing manual browsing, active scanning, and fuzzing for exhaustive security testing?

[Frank Bergmann]

Hi!

Simon Bennets suggested on StackOverflow that I'd re-post my question here.

Quickly about myself, because I just joined the group: I'm the founder of ]project-open[, an open-source project management software for "project businesses" (SME service companies, IT departments, PMO (Project Management Office) of large enterprises, ...). We've got some 6000 productive customers running their business on ]po[, including 5 banks and ~100 companies with >1000 users. 

]project-open[ is really, really big. It consists of about 4.5 million LoC and has about 1000 HTML pages (distinct base URLs). You can get ]po[ running in 10 minutes by downloading a CentOS Linux based virtual machine appliance for VMware ESXi / VMware Player. There is also a public demo server in case you want to have a quick look (but please don't do heavy testing on it). I guess this could be an interested test case for ZAP as a product. Using ]po[ as a target for (local!) testing is fine, just please send us vulnerability reports. There are several published issues (2x SQL injection, 1x persistent XSS, multiple reflections) for version 5.0.3.0.0.

We are now going to add additional parameter type checks for 1000 pages: Each page can take multiple parameters (object_id, filters, column name for sorting, ...). Normally, parameters are checked for a given type by a "procedure head" like structure for each Web page called ad_page_contract, and SQL injection is avoided by passing variable values to the PostgreSQL database via a separate channel (not via the SQL string) as colon variables. However, there are some exceptions where we had to bypass ad_page_contract, and these ~50 cases are causing trouble now.

So we are now going to add checks (integer, alphanum, nohtml, ...) for each of these parameters. We would need to test systematically that: 

• a) offensive parameter values are rejected and 
• b) that the parameter values actually used by the application are accepted. 

Example: We might want to say that the sort_column parameter in a page should only consist of alphanumeric characters. But a customer extension of the app may include a column name with a space in it, leading to a false positive security alert (space character not being an alphanumeric character).

I understand that ZAP should be designed for such a use-case, isn't it?

My idea for testing this would be to:

1. Manually navigate to each of these pages in proxy mode, 
2. tell ZAP to start spider all links on each page for one or two levels and finally
3. tell ZAP to start fuzzing on these URLs.

How can this be implemented in ZAP? I've got a basic understanding of ZAP and did some security testing of ]project-open[. I've read about a ZAP extension for scanning a list of URLs, but in our case we want to execute some specific ZAP actions on each of these URLs...

Simon wrote:

> I'd start by using the ZAP desktop so that you can control it and see exactly what effect it has.
> You can launch a browser, explore you app and then active scan the URLs you've found. 

This is what I did.

> The standard spider will find explore traditional apps very effectively but apps that make a 

> lot of use of javascript will probably require the ajax spider.

Nope, the "standard spider" fails in ]po[ because of the number of parameters in each page and the number of possible values for each parameters, leading to a combinatoric explosion. And correct, all PM applications heavily use JavaScript today (Redmine, Jira, OrangeScrum, ...).

Instead, we would need to start with the "1st level seed URLs" from manual browsing and spider breadth-first for just one or two levels in order to get "2nd level URLs" which should contain all these 1000 existing pages with reasonable parameters.

> You can also use the 'attack mode' which attacks everything that is in scope (which you define)
> that you proxy through ZAP. That just means the ZAP effectively just follows what you do and
> attacks anything new. If you don't explore part of your app then ZAP wont attack it.

That's all fine, there is no problem to attack anything in ]po[. However, the combinatoric explosion of parameter values (see above) renders this attack mode intractable.

> If you want to implement your own tests then I'd have a look at creating scripted active scan rules.
> We can help you with those but I'd just start with exploring your app and running the default rules for now.

I've done the "start with exploring" part, and I've found some XSS reflections within a few minutes. That's all fine.

Now we would have to start with the "systematic" and "exhaustive" part.

I just wonder if our use-case is rare or rather frequent. I thought that many other Web applications were tested like this, but I didn't find anything on the Web. I know there are these automatic testing tools like Selenium etc., but these focus more on GUI stuff rather than back-end security. There are literally thousands of Web applications in the market with a similar structure, and they are usually crawling with security and permission errors.

So maybe there is a more general use-case behind our requirements, where ZAP could make a big difference to the overall security of Web applications...

And congratulations to your work, by the way! ZAP is a show-case open-source project!

Cheers
Frank

[Frank Bergmann]

Just looked through my own post:

> 5 Banks

The banks are running their IT department or parts of it on ]po[, not their banking business, obviously. These IT departments are "project businesses", because they bill the cost of IT operations to the business departments owning the respective applications.

Frank

[Simon Bennetts]

Hiya Frank,

Thanks for posting here:)

I find this really interesting because you could argue that either this _is_ an ideal scenario for ZAP or that it is not :)

A large modern web app is going to be challenging for any tool to test, so I'm not sure any one tool will fit the bill, but ZAP is very flexible and so it could play a significant part in your solution.

I think a big problem you will have (for ZAP and any other tool) is exploring your application - finding all of the end points.

Can you tell us a bit more about the app?

It sounds like its a modern app, does this mean its interacting with the server via an API?

If so, do you have an API definition (eg Open API), or could you generate one, either in a standard definition or just generate requests that use as much of the API as possible?

Do you have a good set of integration tests that drive the API?

Cheers,

Simon

[Frank Bergmann]

Hi Simon,

> exploring your application

Well, we've got a script for manual testing that instructs a human tester to go to pages and to test certain actions there. That got a pretty good coverage of all pages. 

]project-open[ consists of about ~200 individually installable packages. The Web pages in these packages map directly into URLs. This is the approach if you want to be complete. Then just use a browser with this list (in order to execute the JavaScript) and compare the list of URLs with the available "pages"...

> Can you tell us a bit more about the app?

• Check our V5.0 Analyst Briefing document for a quick overview.
• Here is the Architecture Overview.
• Here is a list of all the packages that together make up ]po[. Package documentation also covers the APIs.
• Our Wiki has about 1000 pages.

I'm sure you are working with some Web project management of issue management software yourself, don't you? Bugzilla? Jira? Mantis, RT or OTRS? Redmine? The structure should be the - basically - same... At least with respect to security testing.

> its interacting with the server via an API?

There are "classic" parts of the application that uses plain old HTML 2 Web pages, and there are HTML 5 parts that use the REST API.

> generate requests that use as much of the API as possible?

Correct, there is actually a HTML "viewer" for the REST API data, in part designed to be suitable for spidering...

> integration tests that drive the API?

Nope, that didn't work out for us. We tried once, but they are quickly outdated. We actually got better results (coverage and test speed) with the manual test script mentioned above. I guess that's because of the ERP-like structure of ]po[, which is very "shallow" (no application state stored in memory!) and basically consists of formatting database records as HTML... 

Thanks for your interest!
Frank

[Simon Bennetts]

Have a look at my blog post for speeding up ZAP Scans, its getting on a bit so we do have more options, but its still very relevant: https://blog.mozilla.org/security/2013/07/10/how-to-speed-up-owasp-zap-scans/

One of the key things is to find pages / end points that have unique code behind them, ie are not data generated.

As you've no doubt realised theres no point in attacking 1000s of pages that actually use the same code but different data from a db, you just need to attack one example.

Re the modern parts of your app - have you tried the ZAP ajax spider yet?

https://www.zaproxy.org/docs/desktop/addons/ajax-spider/

That launches browsers so will handle modern web apps. However it is much slower than the standard spider :(

With a large complex app like this I think you'll need a blended approach, where you use the std spider on some parts, the ajax spider on others. You may well need to configure ZAP to understand which parts are data driven as well.

How much effort this takes and whether it brings enough value to you, I'm afraid only you can find out.

We can definitely give you advice and guidance but I'm afraid I dont have enough spare time to go through all of your docs or to try it out myself :/

So my advice would be to try some things out and see how effective they are. Report back and maybe we can make some more specific recommendations.

I'd also recommend breaking your app down into much smaller pieces and concentrating on one part at a time. What works well for one part of your app might not work well for the other part.

Also, if you cant get ZAP to work the way you want, try to describe how you would like it to work here. We would need details though 'just make it all work' isnt going to be enough :P

And if anyone else has any experience using ZAP on these sorts of apps then please chime in :)

Cheers,

Simon

[Frank Bergmann]

Thanks a lot!

I will look at your blog and come back here if I have specific questions. 

One specific question already:

How can I do a breadth-first spider / scan?

From what I've seen ZAP is doing depth-first by default, so that's the reason it spends so much time with parameters etc. (without having read your blog yet...). Breadth-first spidering could make sure ZAP sees all of the Web application pages first before entering too much into parameters...

I've checked on Google, but I couldn't find anything about breadth-first.

Cheers
Frank

[Simon Bennetts]

We dont currently explicity support a breadth first option.

However you could 'fake it' using the "Maximum depth to crawl" option: https://www.zaproxy.org/docs/desktop/ui/dialogs/options/spider/

You could either just spider once with a max depth that works for your app, or set max depth to 1, then spider again with 2 etc

[Simon Bennetts]

I've raised an enhancement request for a breadth first option: https://github.com/zaproxy/zaproxy/issues/6034

If anyone else would find this useful then either comment here or on the issue.

We do our best to prioritize the features that would be of most benefit to the most number of users.