Structural Modifiers and data driven content

[psiinon]

I've just implemented the first phase of support for data driven content: https://github.com/zaproxy/zaproxy/pull/1996#event-442717719
And I'm now writing the help to go with it, and to document structural parameters (which I should have done before).

So, without further ado, heres the help text I've got so far - this is planned to be linked off the Features section in 3 separate pages, the links wont work as they're local ones and I'll add other pages for the screens...

Structural Modifiers

Structural Modifiers are controls which change how ZAP represents the structure of the application.
The Sites tab show ZAP's representation of the application.
If it is not a good representation of the structure then ZAP will not be able to attack the application effectively.
There are currently 2 types of Structural Modifiers:

• Data Driven Content which identify URL paths that represent data
• Structural Parameters which identify parameters that represent application structure instead of user data

Structural Modifiers can only be applied to Contexts.

Data Driven Content

Data driven content is a type of Structural Modifier which identifies URL paths that represent data.

In 'traditional' web applications the structure of the application is typically defined by the URL paths and the data is contained in the URL parameters and POST data.
URLs like:

• https://www.example.com/app/aaa?ddd=eee
• https://www.example.com/app/aaa?ddd=fff
• https://www.example.com/app/bbb?ddd=eee

are represented in the Sites tab as two 'nodes' in the tree:

• Sites
  • https://www.example.com
    • app
      • GET:aaa(ddd)
      • GET:bbb(ddd)

The Sites tree is very important as it reflects ZAP's understanding of the application structure.
If it is not a good representation of the structure then ZAP will not be able to attack the application effectively.

Some applications include data in URL paths.
For example:

• https://www.example.com/app/company1/aaa?ddd=eee
• https://www.example.com/app/company2/aaa?ddd=fff
• https://www.example.com/app/company3/aaa?ddd=ggg

These 3 URLs represent the same page but with different data, but by default ZAP will represent them as three separate nodes:

• Sites
  • https://www.example.com
    • app
      • company1
        • GET:p1(ddd,page)
      • company2
        • GET:p2(ddd,page)
      • company3
        • GET:p3(ddd,page)

This is a problem because ZAP will now attack all 3 pages when it only needs to attack one of them.
In this case attacking the same page 3 times is not a big problem, but if you have hundreds or thousands of pages like this will significantly increase the time it takes to scan the application.

In ZAP terms the 'company' nodes are 'data driven content' - URL parameters that contain data instead of representing part of the application structure.
You can define data driven content by adding the application to a Context and then configuring them via the Context Structure page.
Once you have done this the pages will be correctly represented as 1 node:

• Sites
  • https://www.example.com
    • app
      • «company»
        • GET:p1(ddd,page)

The characters « and » are used to indicate that this is a 'special' node and the node name (in this case 'company') can be set by you to indicate what that node represents.

Structural Parameters

Structural parameters are a type of Structural Modifier which identify parameters that represent application structure instead of user data.

In 'traditional' web applications the structure of the application is typically defined by the URL paths and the data is contained in the URL parameters and POST data.
URLs like:

• https://www.example.com/app/aaa?ddd=eee
• https://www.example.com/app/aaa?ddd=fff
• https://www.example.com/app/aaa?ddd=ggg

are represented in the Sites tab as one 'node' in the tree:

• Sites
  • https://www.example.com
    • app
      • GET:aaa(ddd)

The Sites tree is very important as it reflects ZAP's understanding of the application structure.
If it is not a good representation of the structure then ZAP will not be able to attack the application effectively.

In 'single' page applications a parameter is used to indicate the logical 'page':

• https://www.example.com/app/aaa?page=p1&ddd=eee
• https://www.example.com/app/aaa?page=p1&ddd=fff
• https://www.example.com/app/aaa?page=p1&ddd=ggg

these 3 URLs represent different logical pages, but by default ZAP will still represent them as one node:

• Sites
  • https://www.example.com
    • app
      • GET:aaa(ddd,page)

This is a problem because ZAP will now not attack all of the application's functionality.

In ZAP terms the 'page' URL parameter is a 'structural parameter' - a parameter that defines part of the application structure.
You can define structural parameters by adding the application to a Context and then configuring them via the Context Structure page.
Once you have done this the pages will be correctly represented as 3 nodes:

• Sites
  • https://www.example.com
    • app
      • aaa
        • GET:p1(ddd,page)
        • GET:p2(ddd,page)
        • GET:p3(ddd,page)

Does that make any sort of sense??
Are the terms 'structural modifiers', 'data driven content' and 'structural parameters' ok or can you suggest better ones?
How can I improve it?

Cheers,

Simon

[ryerson...@gmail.com]

Great guide, thank you! The link Context and Context Structure page do not seem to work, are they supposed to be links?

In my case, I have

GET:index.cfm
POST:index.cfm
POST:index.cfm(a,b,c)
POST:index.cfm(d,e,f)

as an example. All 4 of those are the same page but the display is different based on the parameters, the search results might be filtered or something.

To add structural parameters i would go to Session Properties > Contexts > 2:My Custom Context > 2: Structure > Add

Type: Structural Parameter
Name: Do i do 1 separate entry for a, for b, for c, for d, for e, for f or do i do two entries such as (a, b, c) and (d, e, f) and what would be the syntax to associate these parameters back to index.cfm in the event that I have another parameter a, b, c on a different page called mailinglist.cfm
Regex: Not applicable to Structural Parameters

I think the names make sense though and this guide helped to clear up any confusions/misconceptions i had about the parameters.

[psiinon]

Replies inline:

On Wednesday, 4 November 2015 18:35:11 UTC, ryerson...@gmail.com wrote:

Great guide, thank you! The link Context and Context Structure page do not seem to work, are they supposed to be links?

Yes and no :P
As I mentioned in the post: "the links wont work as they're local ones"!
 

In my case, I have

GET:index.cfm
POST:index.cfm
POST:index.cfm(a,b,c)
POST:index.cfm(d,e,f)

as an example. All 4 of those are the same page but the display is different based on the parameters, the search results might be filtered or something.

Looking at those examples, the Sites tree should have a separate 'leaf' node for each one.
In which case the default is fine for you - you have one leaf node for each different way its accessed.
 

To add structural parameters i would go to Session Properties > Contexts > 2:My Custom Context > 2: Structure > Add

Type: Structural Parameter
Name: Do i do 1 separate entry for a, for b, for c, for d, for e, for f or do i do two entries such as (a, b, c) and (d, e, f) and what would be the syntax to associate these parameters back to index.cfm in the event that I have another parameter a, b, c on a different page called mailinglist.cfm
Regex: Not applicable to Structural Parameters

I think the names make sense though and this guide helped to clear up any confusions/misconceptions i had about the parameters.

I dont think you need to use Structural Parameters in this case.
You only need to use those if you have one Sites tree node that represents multiple different ways the page can be accessed.
For example:

• www.example.com/app/?aaa=bbb
• www.example.com/app/?aaa=ccc
• www.example.com/app/?aaa=ddd

will be represented as one leaf node in the Sites tree:

• www.example.com
• GET: app(aaa)

If the same code is being run, just with different data (bbb, ccc, ddd) then this is all well and good.
If in fact the data means that completely different code is run then you'll need to define aaa as a structural parameter. You'll then get:

• www.example.com
• app
• GET: bbb(aaa)
• GET: ccc(aaa)
• GET: ddd(aaa)

We then have different leaf nodes representing the different ways the page can be accessed, and we'll attack each one separately.

Does that make sense?

Cheers,

Simon

 

...

[Stephen Hookings]

Hello Simon

I hope my question is related to above. We would like to derive targets based on SAST scans. The way I do it right now is my own tool - grep for get's in the page content and prepend url. I have to do some manual work to prepare parameters.

Once ZAP, or other tools, are given this content they often make good findings. Or at least I can SQL fuzz the SQL parts. Related a a different post where NetSparkler demos the attack - well sometimes the code makes it so blatant - concatenated input to SQL statement with no casting. Or the fuzz, when you replay in browser

Originally I simply put the URLS in the ForceBrowse list. That visited the pages but without parameters nothing happens.

Would any of the above parameter handling techniques help? Structural modifier or DDC syntax?