ZAP XML Report

[alla]

Hello there,

I am working on an import plugin for MagicTree (www.gremwell.com/
what_is_magictree) to parse ZAP XML reports. There is a problem.
MagicTree autodetects the XML format (that is, what tool generated the
XML) by looking at the top level element in the XML. In case of ZAP it
is <report>. Unfortunately OpenVAS 4 also has <report> as top-level
XML element, which makes it difficult to distinguish between the two.
I wonder what will break if ZAP started using some other name, for
example <ZAPreport> for its top level element? If nothing much will
break, I can submit a patch for ZAP.

Thanks.
Alla

[psiinon]

Hi Alla,

Nothing will break in ZAP, or at least nothing we cant fix ;)

However it could cause problems with other tools that use the reports that ZAP generates.

Having said that, we do want to enhance the reports (for example see http://code.google.com/p/zaproxy/issues/detail?id=139) so I have no problem with making changes.

Do you think you would be able to address any of the other suggestions in issue 139?

Ideally I'd like ZAP to allow the user to specify what they would like to include in the report via a series of checkboxes / pulldowns etc, but that would be more work.

Cheers,

Psiinon

[alla]

Hi Psiinon,

On Jan 12, 11:57 am, psiinon <psii...@gmail.com> wrote:
> Nothing will break in ZAP, or at least nothing we cant fix ;)
>
> However it could cause problems with other tools that use the reports that
> ZAP generates.

I thought so.

> Having said that, we do want to enhance the reports (for example seehttp://code.google.com/p/zaproxy/issues/detail?id=139) so I have no problem

> with making changes.
>
> Do you think you would be able to address any of the other suggestions in
> issue 139?

I can try. I am not familiar with ZAP code, but I have found the XML
report generation code :) , so at least I know where to start. I
understand the scan data comes from the DB. Do you have the DB schema
description somewhere?

> Ideally I'd like ZAP to allow the user to specify what they would like to
> include in the report via a series of checkboxes / pulldowns etc, but that
> would be more work.

That's probably a good idea for HTML or other human-readable formats,
but not for XML reports. XML is intended to be parsed automatically.
If user is allowed to decide what to include in XML, the parser needs
additional code that detects that the expected data is not present and
reports some meaningful error to the user. Otherwise it will lead to
confusing and difficult to troubleshoot errors. So I suggest putting
all available data in XML and let the consumer of the XML decide what
data it wants.

Alla

[alla]

On Jan 12, 11:57 am, psiinon <psii...@gmail.com> wrote:

> Do you think you would be able to address any of the other suggestions in
> issue 139?

Okay, I had a good look at it.

First part - "change the XML document root to something like the
following:
<OWASPZAPReport version="1.1.0b" generated="Fri, 17 Dec 2010
09:08:16">" is easy - I can send the patch.

As for the rest, I don't know where the data should come from. I
looked at org.parosproxy.paros.db classes and I can't figure out where
the list of scanned sites, plugins, spider info, dirbuster data and
port scanner data should come from. If you can give me some idea, how
I can get the data that should go into the report, I can handle it.

Something else: I can see that XML is produced by string
concatenation, with some escaping. I would suggest using some more
robust way of generating XML. I've been using xmlenc (http://
xmlenc.sourceforge.net/) in the past. It is lightweight and easy to
use.

Alla

[psiinon]

Hi Alla,

Sorry for the delay - I've been away and am still trying to catch up!

Thanks for the patches - I look at those asap.

I agree we should use a more robust way of generating XML.

Not heard of xmlenc - whats its escaping like, especially re special characters like null?

Other XML libraries seem to have problems with special chrs, which is why I created this class: https://code.google.com/p/zaproxy/source/browse/trunk/src/org/zaproxy/zap/utils/XMLStringUtil.java

Re the other data, most of that wont be in the db right now - maybe it should be!

I would like as much info as possible to be made available via the API.

However that is currently only available if enabled by the user.

Perhaps it should also be available internally all of the time - that wouldnt be a difficult change to make.

The other alternative, which would be easier to program against, would be for all Extensions to provide a clean programmatic API.

I've made a start with things like the Alerts extension, so you can write things like: 

ExtensionAlert extAlert = (ExtensionAlert) Control.getSingleton().getExtensionLoader().getExtension(ExtensionAlert.NAME);

if (extAlert != null) {

    List<Alert> alerts = getAllAlerts();

    // etc
}

So ideally there should be methods like:

List<integer> ExtensionPortScanner.getPorts(site)

The port scanner might be a good place to start - post any questions you have here :)

As an aside, it would be really nice if the 'generate report' gave a list of sites for you to select.

Or maybe you could select them in the hosts tab and then generate reports for the ones selected?

Many thanks,

Psiinon

[psiinon]

Hi Alla,

There is Issue 231 for API extensions, but thats a very generic one.

I would check xmlenc for handling things like the null character in particular - all of the libs I've looked at failed with this.

XmlStringUtil has some explicit checks for chrs that have caused problems with other libs, so it might be worth having a look at that.

And no, your email didnt seem to get through to the dev list, but I got it :)

Cheers,

Simon

On Mon, Jan 23, 2012 at 12:45 PM, Alla Bezroutchko <al...@gremwell.com> wrote:

I read your post. I'll look how you have implemented the API for Alerts,
and see if I can do the same for port scanner and other extensions. Do
you have an issue ID for API for extensions?

Regarding XML output, I believe xmlenc properly escapes special
characters in XML text. I did not test it extensively, but I had no
problem with all kinds of rubbish data.

Not sure if this will make it through to the mailing list. For some
reason it does not allow me to post from this address.

Alla

On 23/01/12 13:32, psiinon wrote:
> Hi Alla,
>
> I updated the dev group with a bit of info yesterday:
> https://groups.google.com/forum/#!topic/zaproxy-develop/6EaY2CCXGtI

> <https://groups.google.com/forum/#%21topic/zaproxy-develop/6EaY2CCXGtI>

> I think the port scanner might be the easiest one to start on as its
> self contained and fairly simple.
> Is that enough to get started with?
> If not let me know either via the dev group or via email :)
>
> Cheers,
>
> Simon
>
> On Mon, Jan 23, 2012 at 12:24 PM, Alla Bezroutchko <al...@gremwell.com

> <mailto:al...@gremwell.com>> wrote:
>
>     Hi Simon,
>
>     Yes, I had a look. My problem is, I don't know where ZAP keeps the data
>     I need in the report. Issue #139 asks for the following:
>
>     * List of scans in the current session
>     * List of enabled/disabled plugins
>     * Manually browsed URLs
>     * URLs discovered by spider
>     * URLs discovered by DirBuster
>     * Ports discovered by port scanner
>
>     In addition to that I would like the report to include the IP address
>     and port of the site scanned and the root URL of the application (if ZAP
>     has this concept)
>
>     I have no idea what class/method I need to call to get any of the above.
>     If you can give me some hints, I can probably take it from there.
>
>     Alla
>
>     On 23/01/12 13:09, psiinon wrote:
>     > Hi Alla,
>     >
>     > Looks good to me - thanks for that.
>     > I'll apply it tonight (hopefully).
>     >
>     > Have you had a chance to look at any of the other report changes?
>     > I realise they are not as straight forward!
>     >
>     > Cheers,
>     >
>     > Simon
>     >
>     > On Mon, Jan 23, 2012 at 12:03 PM, Alla Bezroutchko
>     <al...@gremwell.com <mailto:al...@gremwell.com>

>     > <mailto:al...@gremwell.com <mailto:al...@gremwell.com>>> wrote:
>     >
>     >     Sorry, my bad. Here it is.
>     >
>     >     Alla
>     >
>     >     On 22/01/12 18:42, psiinon wrote:
>     >     > Hi Alla,
>     >     >
>     >     > Just applied the NPE patch:)
>     >     > But the other one appears to be empty :(
>     >     > Can you try resending it?
>     >     >
>     >     > Many thanks,
>     >     >
>     >     > Simon
>     >     >
>     >     > On Tue, Jan 17, 2012 at 2:57 PM, Alla Bezroutchko
>     >     <al...@gremwell.com <mailto:al...@gremwell.com>
>     <mailto:al...@gremwell.com <mailto:al...@gremwell.com>>
>     >     > <mailto:al...@gremwell.com <mailto:al...@gremwell.com>
>     <mailto:al...@gremwell.com <mailto:al...@gremwell.com>>>> wrote:
>     >     >
>     >     >     Hi Psiinon,
>     >     >
>     >     >     I thought I better contact you off-list, to stop
>     spamming it.
>     >     >
>     >     >     I have attached a patch that changes the root element of XML
>     >     report to
>     >     >     OWASPZAPReport and includes generation date and ZAP
>     version as
>     >     >     attributes.
>     >     >
>     >     >     Regarding the rest of issue #139, I am willing to work
>     on it,
>     >     but I need
>     >     >     some help from you to figure out how to get the data
>     that goes
>     >     into the
>     >     >     report. See my post to the list.
>     >     >
>     >     >     I would also like to add the IP address of the scanned host
>     >     into the
>     >     >     report.
>     >     >
>     >     >     BTW, I have ran into a NullPointerException while
>     playing with
>     >     report
>     >     >     generation. The second patch fixes the NPE.
>     >     >
>     >     >     Regards,
>     >     >     Alla

>     >     > --
>     >     > Vote for ZAP as the Toolsmith Tool of The Year
>     >     >
>     >
>     <http://holisticinfosec.blogspot.com/2011/12/choose-2011-toolsmith-tool-of-year.html>
>     >     >
>     >
>     >
>     >
>     >
>     > --
>     > Vote for ZAP as the Toolsmith Tool of The Year
>     >
>     <http://holisticinfosec.blogspot.com/2011/12/choose-2011-toolsmith-tool-of-year.html>
>     >
>
>
>
>
> --
> Vote for ZAP as the Toolsmith Tool of The Year

> <http://holisticinfosec.blogspot.com/2011/12/choose-2011-toolsmith-tool-of-year.html>
>

--
Vote for ZAP as the Toolsmith Tool of The Year