SOAP scanner for ZAP (GSoC 2014 project) - Progress

[Alberto Verza]

Hello all,

Until now I have been reporting my progress only to my mentors, so we thought it was a good time to publish them to all.

This last month, I made a rough planning document that establishes some milestones, but it was subject to change a bit. Newest version can be viewed here [1].  The main goals were, on the one hand, to have something little but functional by midterm evaluation deadline. On the other hand, I thought it would be great to improve that basic functionality and to have something reasonably useful, so this would be done after midterm.

Then, I started to code the SOAP addon in alpha branch [2]. The first part was the WSDL file importer, which takes a single WSDL file and identifies the specified endpoints and available operations, and tests them sending some basic SOAP requests, supporting 1.1 and 1.2 SOAP versions (the sites tree is updated). Last changes made to the importer include adjustments to make it run as a Singleton so that other components like scanners can retrieve its information, and some code cleanup to make it more readable.

Having a functional WSDL importer, we thought it would be better to start implementing an active scanner to try SOAP Action Spoofing attacks, because it is simpler than others like SQL Injection, which will be implemented later. After developing the first version of it, this rule was tested against standalone Axis2 servers (which seemed to be completely vulnerable) and Tomcat-based Axis2 servers (which ignored altered requests). It remains to test others like JBoss or .NET WS.

Currently I want to continue the project by extending WSDL importing functions in order to make URL and proxy seeking possible, althought I will be working on fixing some existing issues too.

I will keep you up to date in this post.

Regards,

Alberto.

-------------------------------------------------------------------------------------------------------------------------------------

[1] SOAP scanner plans: https://docs.google.com/document/d/1YSf9yh5A02PX1sVYji6Wa3voxaRj83gDzJNiU7UhilA/pub

[2] SOAP scanner code: https://code.google.com/p/zap-extensions/source/browse/#svn%2Fbranches%2Falpha%2Fsrc%2Forg%2Fzaproxy%2Fzap%2Fextension%2Fsoap

[kingthorin+owaspzap]

Hey Alberto, I haven't gotten to play with it much yet but nice work.

Thanks for the update, it's really nice to hear what's going on with these projects. Can't wait to hear/see more.

One minor note, I had a quick look at the code you linked in seems that the lib directory under the soap extension contains a number of jars that are also in the alpha branch lib area.

Hopefully I'll get to doing some testing with this on the weekend or next week :)

[psiinon]

All add-ons need to include all of the jars they depend on in their libs directory, unless those jars are also in the core. The build mechanism then includes these in the add-on 'zap' file.
Those jars currently also have to be in the lib directory for the branch they are in so that the build works.
I need to document this better - hopefully in the next Hacking ZAP blog post ;)

Cheers,

Simon

[psiinon]

Thanks for the update - its great to see to progress you're making :)

Simon

[kingthorin+owaspzap]

Thanks for clarifying the lib thing Simon.

[Niklas Volcz]

Hi!
This was my first time running ZAP in development mode so my issues may not have been caused by the extension.
There seems to be some kind of problem when I try to import the WSDL file. I'm allowed to share the WSDL.
The GUI snaps to the Output tab which displays no sign of errors.

How do I enable debug output for the plugin? I've tried to change the log level to debug in the log4j.properties but I get this log output in all scenarios:
"log4j:WARN No appenders could be found for logger (com.predic8.wsdl.WSDLParser)."

I would also like to thank you for your work! I have been looking for this kind of extension for a very long time.

Thanks!

[Alberto Verza]

Hi Niklas,

Thank you for your interest! I'm glad you are trying out this Add-On :)

The login error happens because the Add-On includes a different log4j library version than ZAP, and it seems it is not taking current configuration from the log4j.properties file. I'm removing this library for the next check out, but meanwhile you could try it manually if you would like.

Regards,

Alberto.

[Alberto Verza]

Hi everybody,

I'm continuing with development, trying to add more ways to detect WSDL files in order to scan SOAP endpoints. I recently finished a menu option to insert a single URL pointing to a wsdl file, which seems to work fine. Now I'm trying to add functionality to automatically detect all wsdl files from a given root URL, but I think this would need to be handled by the ZAP spider. Does anybody know how could I work with it? What I need is the spider making requests to endpoints once a wsdl file is detected. Now it only detects wsdl files, but it does not explore them so far.

Thank you.

Regards,

Alberto.

[Cosmin Stefan-Dobrin]

Hi,

Sounds really nice. Great work! :)

Regarding parsing using the Spider, it's easy to add new 'detection' functionality by implementing a new SpiderParser and hooking it up. Check out the 'org.zaproxy.zap.spider.parser' package where you can find quite a few working parsers. The downside is that the Spider cannot be extended purely from 'extensions' as it is now as the list of parsers used is not public and you need to register it in the SpiderController.

But maybe the best way would be to actually change the spider so that it allows registering custom Parsers? Should we do this?

Cheers,
Cosmin

Cosmin

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cosmin Stefan-Dobrin]

Actually I've created https://code.google.com/p/zaproxy/issues/detail?id=1239 and I'll try to implement it asap. Please let me know if anyone has any other ideas of things that we should allow to extend in the Spider (via the Issue so we don't hijack Alberto's thread :P ).

Cosmin

Cosmin

[Alberto Verza]

Thanks for your support Cosmin! I will take a look to those parsers, see if I could make something similar. It will be useful at least to understand ZAP better :)

[Cosmin Stefan-Dobrin]

Sure, no problem! :) I believe this change was long overdue.

Anyway, I've just committed the changes for this in r5016. So now the Spider can be extended from any other extension via the #addCustomParser(), #addCustomFetchFilter() and #addCustomParseFilter() of the ExtensionSpider.

So what you could do now is fetch the latest changes, extend the SpiderParser abstract class implementing what you need to do, get a reference to the ExtensionSpider (be sure to null check if it's been loaded)  and add your custom parser via ExensionSpider#addCustomParser(). Then, this parser will be automatically invoked by the Spider for every message (first #canParseResource() and then #parseResource() ).

Cosmin

[Cosmin Stefan-Dobrin]

And one more thing. If you're interested in a few more Spider related details (concepts), feel free to take a look at the https://code.google.com/p/zaproxy/wiki/InternalSpider wiki page, especially the 'General Concepts' and the (newly created) 'Extending the Spider' sections.

Cosmin

[kingthorin+owaspzap]

Wow, now that's agile development :)

Nice work you two!

[Alberto Verza]

Hi again,

On Tuesday, 17 June 2014 20:29:00 UTC+2, kingthorin+owaspzap wrote:

Wow, now that's agile development :)

Nice work you two!

Thank you! While I continue with developing tasks, I leave you all a link showing how the Add-On currently works when you import a WSDL file from local file system, it would be similar for the single URL option. You have to enable captions since I haven't added audio to it, sorry for that. The video is available here: https://www.youtube.com/watch?v=kRZMzpbFD9c&feature=youtu.be

Regards,

Alberto.

[Alberto Verza]

Hi Cosmin,

I'm now working in 2.4 alpha branch, and when I try to call the new methods, Eclipse alerts me that they are not implemented. Should ZAP.jar be updated or am I doing something wrong?

Thank you,

Alberto.

[thc...@gmail.com]

Hi.

Yes, the zap.jar needs to be updated with the latest core changes.
Using a checkout of zaproxy repo at latest revision, run the Ant target "dist" (using the build.xml file in "build" directory), it will create the zap.jar in the directory build/zap.
Copy the newly built zap.jar to the 2.4 alpha branch lib directory and commit it.

Best regards.

[Alberto Verza]

Thank you thc202! I have just committed new version to 2.4 alpha branch and the Add-On now compiles correctly :)

I would want to ask another thing. Currently I have implemented a custom spider to identify endpoints they contain automatically. If I try to attack a site which has a wsdl file using ZAP's main window feature (introducing target URL and clicking "attack" button"), their endpoints are identified sucesfully, but it seems that if they belong to another different site (even if only remote port changes, ZAP treats it as a different site), active scans don't consider them, so they are not being checked automatically. However they can still be checked manually by specifying each of the identified endpoints one by one.

Could anybody help me about how to fix this issue? I did some research and I think that Active scan extension and its #startScan() method could be of my interest, but I'm not sure.

Thank you again.

Regards,

Alberto.

[Alberto Verza]

Hi everybody,

I'm glad to announce that the SOAP Add-On project has passed GSoC Midterm Evaluations, so this work will continue growing :)

I'm now investigating how ZAP currently supports SOAP messages, and it seems they are handled pretty well since they follow XML structure after all. I'm specifically taking a look to SQL injection scanners to observe their behavior.

While doing this, I have detected something I don't understand. XML input parameters are handled by Variants, so they are used to insert attack parameters using #setParameter() for unescaped values and #setEscapedParameter() for escaped ones (non-escaped parameter would be for example ", and its respective scaped code would be &quote;) [1]. However, even when scanners like [2] use #setParameter(), ZAP escapes them so attacks are not correctly executed. Do you know why are parameters escaped even when #setParameter() is used?

Thank you.

Best regards,

Alberto.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[1] https://code.google.com/p/zaproxy/source/browse/trunk/src/org/parosproxy/paros/core/scanner/VariantAbstractRPCQuery.java

[2] https://code.google.com/p/zap-extensions/source/browse/branches/beta/src/org/zaproxy/zap/extension/ascanrulesBeta/SQLInjectionMySQL.java#230

[Alberto Verza]

Hello everybody,

By writing this message I just want to keep you up to date with project progress. Current implementation seems to be compatible with some useful existing ZAP scanners like SQL Injection or XML External Entity attacks, so there is no need to implement them specifically for SOAP messages. Now I'm testing them against some different Web Service servers to check compatibility. I'm also working in WSDL file parsing improvements, since some of them could not be correctly parsed depending on its structure.

Thank you for your interest!

Best regards,

Alberto.

[kingthorin+owaspzap]

That's great Alberto, keep up the wonderful work!

[Visser, Dale]

Yes. Thank you, Alberto, for tackling this much needed feature. J

--

[Michal Kraus]

Is SOAP scanner for ZAP available for ZAP 2.5.0 ?

[thc...@gmail.com]

Hi.

Yes, it is, it can be installed from the marketplace. [1]


[1] https://github.com/zaproxy/zap-core-help/wiki/HelpUiDialogsManageaddons

Best regards.

On 19/09/16 12:05, Michal Kraus wrote:
> Is SOAP scanner for ZAP available for ZAP 2.5.0 ?
>
> W dniu sobota, 12 lipca 2014 20:01:04 UTC+2 użytkownik Dale Visser napisał:
>
> Yes. Thank you, Alberto, for tackling this much needed feature. J
>
>
>

> *From:*zaproxy...@googlegroups.com
> [mailto:zaproxy...@googlegroups.com] *On Behalf Of *kingthorin+owaspzap
> *Sent:* Saturday, July 12, 2014 9:19 AM
> *To:* zaproxy...@googlegroups.com
> *Subject:* Re: [zaproxy-develop] Re: SOAP scanner for ZAP (GSoC 2014

> project) - Progress
>
>
>
> That's great Alberto, keep up the wonderful work!
>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to zaproxy-devel...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to zaproxy-devel...@googlegroups.com

> <mailto:zaproxy-devel...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/zaproxy-develop/385f072e-012b-46e2-838a-09e25e18fcf6%40googlegroups.com
> <https://groups.google.com/d/msgid/zaproxy-develop/385f072e-012b-46e2-838a-09e25e18fcf6%40googlegroups.com?utm_medium=email&utm_source=footer>.