Spider option on Active Scan dialog

[psiinon]

I've had a suggestion from Dave Wichers about adding a 'spider' option (checkbox) to the Active Scan dialog.
So this would allow you to kick off the (traditional) spider before launching the active scan, ie similar behaviour to the Quick Start tab.
We could also add a 'Ajax Spider' checkbox as well.

What do you all think?
Could it be useful?

Cheers,

Simon

[Cash]

I think it makes sense to add (both spider and ajax spider), assuming existing quick scan does not spider and the results aren't exclusive if ZAP doesn't know about most of the URLs.

Since the idea of the quick start is to add a new url and hit go, spidering should take place 1st.

Cash Williams
(979) 204-6683

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[kingthorin+owaspzap]

As I think about this I find myself waffling.

#1
My gut reaction was sure seems to make sense, as long as the default is off (always). [Perhaps a separate Tools:Options "option" for 'Remember how I set active scan spidering' in-case some newer users always want to spider before scanning.]

#2
My 2 mins more reflection thinking was that we add a "Quick Scan" context menu or hotkey (or both) along side "Active Scan". That way Active Scan continues to function like Active Scan (Scan) and Quick Scan continues to behave like Quick Scan (Spider and Scan)....

[Mark Rader]

Simon

From my perspective, I can see how that might be useful, but I would see it of being of more benefit under the Quick Start tab.  Probably a check box of "Spider Only" with two check boxes of Spider and Ajax Spider to let you select one or both of the spiders.  This would then feed active scan, and all of the other scanners.

Then one could active scan particular areas or just do the scan one wants at the time.

Mark

On Tue, Sep 1, 2015 at 10:50 AM, psiinon <psi...@gmail.com> wrote:

--

[Kevin W. Wall]

[Off list]

Simon,

I think that's a great idea. The main reason we've been told we can't download & use ZAP at Wells Fargo is that it pretty much always does active scans by default. I think ideally the powers that be would prefer a deployment of ZAP with NO active scans at all, but IMHO that would pretty much cripple it. However, I think we maybe could argue this as a happy middle ground.

-kevin
Sent from my Droid; please excuse typos.

--

[kingthorin+owaspzap]

I'm confused Kevin, how does adding an option to the active scanner to spider prior to active scanning address this WF concern? If you're active scanning you're active scanning, whether or not you spider first ....

Seems like modes are a more fitting solution to the issue you've outlined:
https://github.com/zaproxy/zap-core-help/wiki/HelpStartConceptsModes

Another alternative might be to the ascan *.zap module before launching ZAP, then there'd be no active scan "scanners" to run....I've never tried this, ZAP may not like it......

[Kevin W. Wall]

On Tue, Sep 1, 2015 at 8:06 PM, kingthorin+owaspzap
<kingt...@gmail.com> wrote:
> I'm confused Kevin, how does adding an option to the active scanner to
> spider prior to active scanning address this WF concern? If you're active
> scanning you're active scanning, whether or not you spider first ....
>
> Seems like modes are a more fitting solution to the issue you've outlined:
> https://github.com/zaproxy/zap-core-help/wiki/HelpStartConceptsModes
>
> Another alternative might be to the ascan *.zap module before launching ZAP,
> then there'd be no active scan "scanners" to run....I've never tried this,
> ZAP may not like it......

Sorry; read t hat wrong. I thought this was a spider-only mode that would not do
active scanning. Now that the cat is out of the bag (so much for
"off-list" :), what
would be helpful to us is to have deployment of ZAP that allows
completely passive
scanning...no attack mode available, etc. On the stuff needed to passively do
reconnaissance, but no ability to fuzz parameters, etc. Sure, it would mostly
cripple a lot of the ZAP functionality, but we'd probably be allowed to download
and use it at work. Keeping the interception proxy mode is okay where parameters
can be manually altered before sending through the response, but no automated
fuzzing / attack modes. Think of it as the ZAP equivalent to Fiddler2 (which BTW
we are permitted to use and which is what most of us use now). But this would
allow us to use Zest to write scripts to look for stuff we are
interested in that
is not always obvious from code inspections because many times the applications
are sitting behind reverse proxies and doing things like adding HSTS
and X-FRAME-OPTIONS
response headers, or adding Secure and/or httpOnly flags to the
Set-Cookie headers, etc.
Spidering only (especially AJAX spidering) could also help us locate cases where
external 3rd party URLs (for web service calls or JavaScript, etc.)
are used. Usually
we can find the fact that those are used in the code, but trying to
follow all the
AJAX logic to see from what pages that can be accessed from (important when
considering roles!) is extremely time consuming.

I think the Safe and Protected modes are pretty much what we want; the
"problem" is
that it's possible to use other modes like Standard and Attack modes
because it is
not possible for some central administrator to disable them. The fact
that it is _possible_
to use these modes is what makes it impossible for us to be allowed to use
ZAP at work. So if there were a separate ZAP build where the Standard and
Attack modes were disabled and couldn't be easily enabled (at least without some
significant coding) or where those packages for those modes simply were not
delivered, _then_ I think we'd have a chance of convincing the powers that be
that ZAP is safe enough for us to use as a tool in our daily work.

-kevin
--
Blog: http://off-the-wall-security.blogspot.com/
NSA: All your crypto bit are belong to us.

[kingthorin+owaspzap]

Thanks for the clarification. I can see what you're trying to accomplish... I haven't downloaded it and don't see much documentation about what is or isn't included (something the Team will have to tackle). But, "ZAP Core" might fit the bill here, it's a reduced functionality package.

https://github.com/zaproxy/zaproxy/wiki/Downloads#zap-241-core

[psiinon]

We could definitely generate a 'safer' version of ZAP, but I'm guessing a config option wouldnt be enough (as it could be trivially changed).
That would mean we would have to make code changes and build a completely separate version of ZAP.
We would also have to either disable the add-ons or classify then as 'safe' or 'unsafe'.
How much interest is there in a 'safer' ZAP?
And is there any agreement as to what its capabilities should be?

I guess another option would be to have a version of ZAP that required a lookup to a preconfigured location that specifies what features of ZAP the user can use on what, including url whitelists, blacklists etc.
But again this would require code  changes and unique builds.

If theres not significant interest, or no common agreement on what 'safer ZAP' should look like then this could be a significant overhead for us to maintain :/

Kevin (and anyone else interested in this), would you be prepared to generate your own versions of ZAP?
We could make code changes to support Safer ZAP and define the build options required to configure it.
You would then need to choose which options you want, build ZAP on the platforms you need to support and make it available within your organizations.

Cheers,

Simon

[psiinon]

A reply from Dave, who still hasnt joined the list ;)

---------- Forwarded message ----------
From: Dave Wichers <dave.w...@aspectsecurity.com>
Date: Tue, Sep 1, 2015 at 6:31 PM
Subject: RE: [zaproxy-develop] Spider option on Active Scan dialog
To: Mark Rader <msr...@gmail.com>, "zaproxy...@googlegroups.com" <zaproxy...@googlegroups.com>
Cc: "psi...@gmail.com" <psi...@gmail.com>

Simon, Can you forward my reply to this list?

That's kind of what I asked for at first, but Simon was hesitant to clutter the Quick Start tab with more features.

And, I want to be able to do this on the Active scan where I can also select a custom policy I want to use, unless you want to add select policy to the Quick Start tab too. (but that's even more features on Quick Start).

That's why I think adding this to Active Scan is best as it leaves Quick Start alone, but makes the Active scan work like Quick Start (or can easily be configured to do so, depending on the default settings).

-Dave

---

From: Mark Rader [msr...@gmail.com]
Sent: Tuesday, September 01, 2015 1:24 PM
To: zaproxy...@googlegroups.com
Cc: Dave Wichers
Subject: Re: [zaproxy-develop] Spider option on Active Scan dialog

--

OWASP ZAP Project leader

[psiinon]

I think this is an interesting proposal, and a difficult balance between usability and functionality (as is often the case).

I really want to keep the 'Quick Start' tab as simple as possible.
It was always intended as a simple 'point ad shoot' option for people new to ZAP, and I dont really want to add any more options.

I'm not _that_ keen on adding Spider options to the Active Scan dialog.
I like the fact that exploring an app is separated from attacking it.
And you have multiple options for exploring the app, including:

• Proxying
• Spider
• Ajax Spider

Both spiders also have their own set of options, so where does this end?

Having said that, if people want it then theres a use-case for it :)

One option would be a 'Advance quick scan' dialog (ok, ok, it would need another name :P )

This could include tabs for anything and everything.

I'd be wary of putting it on the Quick Start tab, but we could definitely have a toolbar button / menu item / hot key for it.

It would, of course, be implemented as an add-on, and whether it gets included by default can be decided later.

Another option is an add-on that adds a Spider tab to the existing Active Scan dialog - other add-ons already do this.

We might need to make some changes to ensure it can kick off the spider(s) before the active scan, but that shouldnt be too hard.

This would definitely _not_ be included by default, but could easily be installed by anyone who wants to use ZAP in this way.

Thoughts?

Simon

On Wednesday, 2 September 2015 09:24:49 UTC+1, psiinon wrote:

A reply from Dave, who still hasnt joined the list ;)

---------- Forwarded message ----------
From: Dave Wichers <dave.wichers@aspectsecurity.com>
Date: Tue, Sep 1, 2015 at 6:31 PM
Subject: RE: [zaproxy-develop] Spider option on Active Scan dialog
To: Mark Rader <msr...@gmail.com>, "zaproxy-develop@googlegroups.com" <zaproxy-develop@googlegroups.com>
Cc: "psi...@gmail.com" <psi...@gmail.com>

Simon, Can you forward my reply to this list?

That's kind of what I asked for at first, but Simon was hesitant to clutter the Quick Start tab with more features.

And, I want to be able to do this on the Active scan where I can also select a custom policy I want to use, unless you want to add select policy to the Quick Start tab too. (but that's even more features on Quick Start).

That's why I think adding this to Active Scan is best as it leaves Quick Start alone, but makes the Active scan work like Quick Start (or can easily be configured to do so, depending on the default settings).

-Dave

From: Mark Rader [msr...@gmail.com]
Sent: Tuesday, September 01, 2015 1:24 PM

To: zaproxy-develop@googlegroups.com

Cc: Dave Wichers
Subject: Re: [zaproxy-develop] Spider option on Active Scan dialog

Simon

From my perspective, I can see how that might be useful, but I would see it of being of more benefit under the Quick Start tab.  Probably a check box of "Spider Only" with two check boxes of Spider and Ajax Spider to let you select one or both of the spiders.  This would then feed active scan, and all of the other scanners.

Then one could active scan particular areas or just do the scan one wants at the time.

Mark

On Tue, Sep 1, 2015 at 10:50 AM, psiinon <psi...@gmail.com> wrote:

I've had a suggestion from Dave Wichers about adding a 'spider' option (checkbox) to the Active Scan dialog.
So this would allow you to kick off the (traditional) spider before launching the active scan, ie similar behaviour to the Quick Start tab.
We could also add a 'Ajax Spider' checkbox as well.

What do you all think?
Could it be useful?

Cheers,

Simon

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mark Rader]

Kevin

It may be easy to do from what you have said, at least as a internal special version.  Maybe the simplest route to achieve what you want is to simple remove the options from the dropdown menu, so the user can't access the active attack modes.  The attack code would stay intact, but unreachable. 

[Stephen Hookings]

Hi Kevin

I feel your pain about getting permission to use a security tool internally. Took me 6 months to get generic permission to use ZAP in SAP - and we are the Security Enablement team :-)

Each new release also requires some negotiation - we code scan the product, do some FOSS checks to ensure we are not signing our IP away etc etc. Having said this, of course our IT teams are trying to protect the internal infrastructure and reduce risk -- once you get people to agree we have a common goal (just at different parts of the lifecycle) then you can begin to progress. And of course we could show as codescan/dynamic test enablement team that these very tools can be helpful to their department -- win win.

It helps to get the IT department in the loop - take onboard their fears and reasons for "NO" - then reduce the risk so that "YES" becomes acceptable within their parameters. You learn where it is important to compromise (for example, do you want to be the reason behind DoS in your Corporate Internet? Do you want to crash customer data because there was no policy on how to avoid this??).

The most compelling argument I used was "If we do not do this, bad dudes, our competitors and our customers will!!". Once you then start with stand-alone "off premise" test and show progress [[I guess the idea is if you are sloppy and get caught then it was a non-sponsored individual who messed up]]. Then you can move to internal cloned environments with faked data (so no DPP concerns), use test users (so no stealing the credentials of your colleagues) you can move to cloud based sandboxes (we are experimenting with various VMs and clouds in SAP) with network isolation so accidentally attacking an external website/internal infrastructure becomes practically impossible -- and when that is all running you look back and say "why didn't we do this earlier!".

Simply to say rather than changing ZAP, why not work with your IT departments to find a mutually agreeable test landscape? Sure it will take time but it is well worth the effort.

Not least because if you use commercial tools you would NOT want them to only be passive would you? So why cripple ZAP. Worst case, reach inside the code and disable the features (ie fork your own) - it is open source after all.

But better to start the journey for controlled internal usage - your customers will ultimately thank you for doing this.

Regards

Steve H

[psiinon]

Thanks Steve - that was really useful.

Question for everyone - what could we do to make it easier for organisations to adopt ZAP internally?

Code changes, documentation, advice and guidance - anything :)

Many thanks,

Simon

[Mário Areias]

In my experience, we were more successful to adopt ZAP when we started to talk about automation. ZAP makes really easy to automate and that was the biggest difference among other competitors, including Burp Suite.

One thing that might help is the ability to ignore specific errors. I believe the work to implement has been started which is great! However, that was a major block for our team. To fix that I added a very simple filter on the grunt-zaproxy plugin. So at least we could ignore errors by priority which helped a little bit, although, it wasn't the ideal solution. I am curious to see this filter implemented :)

Cheers,

Mário

--