Add-on management

[psiinon]

The interface for managing add-ons was written when ZAP was very much a desktop tool.

ZAP is now also used a lot in CI and as part of services like ThreadFix and Minion, and the current options dont work as well in these environments.

In the UI you can browse the ZAP Marketplace and manually select add-ons to add, remove and update. You can also configure ZAP to automatically update add-ons that have changed when ZAP starts up.

In CI and ‘service’ environments ZAP is typically installed in one location and then launched in a new location every time it is run. If the ‘auto update’ options are set then these ZAP instances will download the same updates every time. And theres no way to say that new add-ons should be installed, such as the beta active scan rules.

I’m proposing to make the following changes:

Introduce a new optional ‘shared add-ons directory'.

If this is defined then a ZAP instance will include any add-ons in this directory when it loads. ZAP already loads add-ons from the installation and ‘local’ directories, so it will use the same code to work out which are the most recent versions.

The idea is that in service environments all new ZAP instances will point to the same shared add-ons directory so that new and updated add-ons will only have to be downloaded once.

There will also be an option to install add-ons to this directory when they are downloaded. If selected then the add-ons will be downloaded to the ‘local’ directory and only copied (permissions allowing;) to the shared directory when complete. This is so that instances started mid-download wont get incomplete files.

My expectation is that specific ZAP instances will be launched just to get new and/or updated add-ons - these instances wouldnt typically be used for scanning apps.

Enhance the API to allow full control of the add-on installation

There should be calls for things like:

• Viewing all of the add-ons on the marketplace

• Reporting the status of add-ons

• Downloading specific (new) add-ons

• Updating specific add-ons

• Updating all updated add-ons

• Uninstalling specific add-ons

Anything else I’ve missed out?

Enhance the command line to support:

• Installing specific add-ons

• Updating all add-ons prior to doing anything else

The idea is that you could set up a cronjob to do things like…

Install a specified add-on to the shared directory. eg:

• zap.sh -addon-install ascanrulesBeta -addon-dir shared

Update a specified add-on (if its been updated of course). eg:

• zap.sh -addon-update ascanrulesBeta

Update all add-ons. eg:

• zap.sh -addon-updateall

Install (for example) the beta ascan rules before scanning an app ‘inline’

• zap.sh -addon-install ascanrulesBeta -cmd -quickurl http://example.com/

Note that cmdline option names may change ;)

What do you all think?

Slightly related topic - does anyone want to run their own ‘marketplace’ so you can download ZAP add-ons from your own server? We could add an option for you to specify an alternate url...

Cheers,

Simon

[Mark Rader]

Simon

I like the idea of having the ability to have an "independent" marketplace as it may be termed.  That would let organizations develop and manage there own internal plugins' as well as use versions of the standard plugins' that may have custom modifications in a managed forum.

Mark Rader

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mário Areias]

HI Simon,

Great initiative!

I liked all ideas and I will give my 2cents based in my experience within big organisations and start ups. I think the idea for shared folder is incredibly helpful and I would like to give two or more suggestions:

• It would be good to have two shared folders, one global folder and one inside the home folder. The global folder could be managed by puppet/chef/ansible or by the sysadmin. So using puppet you would install ZAP and the most used plugins inside the organisation. However, this folder would be read-only for all users except by root, which means the user running the build wouldn't be able to add plugins or remove them. Then you have the shared folder inside the home folder. This folder would be used by the build agent user to install other plugins or update plugin without need to change the infrastructure code. That might be confusing so I will give an example here:

• The DevOps person/team creates a puppet module to install ZAP and addons A and B. This puppet module would use wget or similar to download the addons and put in the global folder, let's say /var/lib/zap-addons. ZAP and addons will be installed in all build agents from now on.
• Team A has a specific requirement and wants to use addon C. Instead to go to the DevOps team and ask them to add it, they add as part of the build one step to download this new addon. This addon will be added to the build agent user like /home/jenkins-slave/.zap-addons/
  
• As part of the build, it downloads any required addons specified by the build, then ZAP loads and looks at both folders and load all plugins. If there is any conflict (like Global folder having addon A in version 1.0, but home folder having addon A in version 1.5), the home folder has precedence.
  

I know the solution above seems a little complicated, however, I think it has both stability and flexibility required to help to improve CI support in ZAP. I am happy to be proven otherwise, though :)

Another idea is to have a debian/rpm file to install ZAP. It helps to do automation and it makes easier to install. If we could do both debian packages and shared folders, that would be very very helpful. The others ones including the command line I see as a nice to have.

Cheers,

Mário

[Stephen de Vries]

All 3 are great ideas and would be very useful for automation and CI integration.  +3  :)

[psiinon]

I like the idea as well :)
But it will require some changes, and even more documentation :(
So if theres no pressing need for it I'll probably leave that for now.
If a company is interested in this (and anyone feel free to ping me offline) then they'll be looking to make a fairly significant time/effort investment in ZAP.
So hopefully they'll also be happy to help with the changes and/or documentation :D

Cheers,

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[psiinon]

Hey Mário,

Thanks - thats really useful feedback!
In implementation terms theres not much difference in having one shared folder and an arbitrary number, so I might well go with that :)

I've had other requests for better linux packaging.
Thats not something I'm experienced with - can anyone on this list help out with that?
If we can change the build file to generate a proper linux package then we'll use that instead of the .tar.gzip :)

Cheers,

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[psiinon]

On Wednesday, 16 September 2015 09:29:49 UTC+1, psiinon wrote:

I've had other requests for better linux packaging.
Thats not something I'm experienced with - can anyone on this list help out with that?
If we can change the build file to generate a proper linux package then we'll use that instead of the .tar.gzip :)

And I've just remembered that someone already updated the build route to generate a debian package ;)
I've added the 2.4.2 one to the release: https://github.com/zaproxy/zaproxy/releases/download/2.4.2/zaproxy-2.4.2.deb

Can anyone confirm that it works for them?
If so I can add it to the Downloads page.

Cheers,

Simon

[Mário Areias]

Simon,

I just installed the package you mentioned in my vagrant box running Ubuntu. It seems it doesn't work as expected but it does the basic operation, let me explain:

• I started with a "wget https://github.com/zaproxy/zaproxy/releases/download/2.4.2/zaproxy-2.4.2.deb"
• Then I ran "sudo dpkg -i zaproxy-2.4.2.deb" where it happened two actions:
  
  • It unzipped the zap in "/usr/share/zaproxy" folder
  • It created a script file called "zap" in /usr/bin

So if you type "zap" on command line it does start ZAP, however it failed in my machine because I don't have GUI installed in my vagrant box. When I tried to start as daemon "zap -daemon -host localhost -port 8080", it was still trying to start the GUI option. So I realised any command passed through the command line, it wasn't being passed to zap.sh script.

In summary, it does work but not quite as expected :(

--

You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[psiinon]

Thanks for the feedback!

I'm guessing this file needs to pass the parameters across: https://github.com/zaproxy/zaproxy/blob/develop/build/debian/zap

Could you edit this file on your machine (/usr/bin/zap) and add  "$@" to the end of the second line and see if that fixes the problem?

Cheers,

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[Mário Areias]

Yeap, that worked :)

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[Mário Areias]

I was checking the build script to create this debian package.

https://github.com/zaproxy/zaproxy/blob/develop/build/build-debian.xml

It looks at version 2.0.0. It would be nice to update this automatically when we do a new release, it should be an easy task if we do releases in an automated fashion like using CI. Might be not that easy if we do manually, though

Cheers,

Mário

[Colm O'Flaherty]

This was done for the purposes of making a DEB that the Offensive Security guys could include in Kali Linux.  They're not using it, to my knowledge, but the DEB in question definitely used to work well for me on Kali 1.*, in place of the out-of-date 2.4.0 version that's (still) bundled there. That was a few months back, so YMMV now.

--

You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[psiinon]

The one I uploaded was created by this part of the main build script: https://github.com/zaproxy/zaproxy/blob/develop/build/build.xml#L518
I think thats the one you created Colm, and with the tweak to the zap script it seems to work well.
This means that we automatically get a debian build when we do a full build, and as long as it works well I'm happy to publish it along with the other packages we release.
The build-debian.xml file was my first attempt, is out of date and should probably be deleted ;)

Cheers,

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[psiinon]

I've just committed the command line part of this development as well (just the API part to go).
In the end I just implemented:
    -addoninstall <addon>    Install the specified add-on from the ZAP Marketplace
    -addonupdate             Update all changed add-ons from the ZAP Marketplace

-addoninstall will update the specified add-on if its already been installed.
I didnt bother implementing another option for specifying which directory to download add-ons to as that can already be set via the -config option.

I also tweaked the command line help output - it now looks like this:

Usage:
    zap.sh [Options]
Core options:
    -version                 Reports the ZAP version
    -cmd                     Run inline (exits when command line options complete)
    -daemon                  Starts ZAP in daemon mode, ie without a UI
    -config <kvpair>         Overrides the specified key=value pair in the configuration file
    -dir <dir>               Uses the specified directory instead of the default one
    -installdir <dir>        Overrides the code that detects where ZAP has been installed with the specified directory
    -h                       Shows all of the command line options available, including those added by add-ons
    -help                    The same as -h
    -newsession <path>       Creates a new session at the given location
    -session <path>          Opens the given session after starting ZAP
    -host <host>             Overrides the host used for proxying specified in the configuration file
    -port <port>             Overrides the port used for proxying specified in the configuration file
    -lowmem                  Use the database instead of memory as much as possible - this is still experimental
    -experimentaldb          Use the experimental generic database code, which is not surprisingly also still experimental
Add-on options:
    -addoninstall <addon>    Install the specified add-on from the ZAP Marketplace
    -addonupdate             Update all changed add-ons from the ZAP Marketplace
    -last_scan_report <path> Generate the 'Last Scan Report' into the specified path
    -quickurl [target url]: The URL to attack, eg http://www.example.com
    -quickout [output filename]: The file to write the XML results to
    -quickprogress: Display progress bars while scanning
    -script <script>         Run the specified script from commandline or load in GUI

The "-quick*" options still need to be changed, but they are in zap-extentions rather than the code, and should probably be changed just before a release.
That look better than the current help?

Cheers,

Simon

[T B]

Is it possible to install an addon from the MarketPlace while behind a proxy on a non-GUI (headless) server using the command line?  I've modified zap.sh to include a variable ${JAVA_OPTIONS} which expands to the following 4 settings:

-Dhttp.proxyHost=myproxyservername.com -Dhttp.proxyPort=3128 -Dhttp.proxyUser=myusername -Dhttp.proxyPassword=mypassword

$ZAPROXY_HOME/zap.sh -installdir $ZAPROXY_HOME -config database.recoverylog=false -addoninstall exportreport -cmd

Found Java version 1.8.0_121

Available memory: 7823 MB

Setting jvm heap size: -Xmx1955m

Check for updates call failed

I'm using Java version 1.8.0_121 and have tried with ZAP 2.6.0 and ZAP_D-2017-04-18

Thanks

TB

[thc...@gmail.com]

Hi.

It should be possible, but you also need to configure the outgoing proxy
in ZAP. [1] [2]


[1]
https://github.com/zaproxy/zap-core-help/wiki/HelpUiDialogsOptionsConnection#use-proxy-chain
[2] https://github.com/zaproxy/zaproxy/wiki/FAQconfigValues

Best regards.

>>> *Introduce a new optional ‘shared add-ons directory'.*

>>>
>>> If this is defined then a ZAP instance will include any add-ons in this
>>> directory when it loads. ZAP already loads add-ons from the installation
>>> and ‘local’ directories, so it will use the same code to work out which are
>>> the most recent versions.
>>>
>>> The idea is that in service environments all new ZAP instances will point
>>> to the same shared add-ons directory so that new and updated add-ons will
>>> only have to be downloaded once.
>>>
>>> There will also be an option to install add-ons to this directory when
>>> they are downloaded. If selected then the add-ons will be downloaded to the
>>> ‘local’ directory and only copied (permissions allowing;) to the shared
>>> directory when complete. This is so that instances started mid-download
>>> wont get incomplete files.
>>>
>>> My expectation is that specific ZAP instances will be launched just to
>>> get new and/or updated add-ons - these instances wouldnt typically be used
>>> for scanning apps.
>>>

>>> *Enhance the API to allow full control of the add-on installation*

>>>
>>> There should be calls for things like:
>>>

>>> -

>>>
>>> Viewing all of the add-ons on the marketplace

>>> -

>>>
>>> Reporting the status of add-ons

>>> -
>>>
>>> Downloading specific (new) add-ons
>>> -
>>>
>>> Updating specific add-ons
>>> -
>>>
>>> Updating all updated add-ons
>>> -

>>>
>>> Uninstalling specific add-ons
>>>
>>> Anything else I’ve missed out?
>>>

>>> *Enhance the command line to support:*
>>>
>>> -
>>>
>>> Installing specific add-ons
>>> -

>>>
>>> Updating all add-ons prior to doing anything else
>>>
>>> The idea is that you could set up a cronjob to do things like…
>>>
>>> Install a specified add-on to the shared directory. eg:
>>>

>>> -

>>>
>>> zap.sh -addon-install ascanrulesBeta -addon-dir shared
>>>
>>> Update a specified add-on (if its been updated of course). eg:
>>>

>>> -

>>>
>>> zap.sh -addon-update ascanrulesBeta
>>>
>>> Update all add-ons. eg:
>>>

>>> -

>>>
>>> zap.sh -addon-updateall
>>>
>>> Install (for example) the beta ascan rules before scanning an app ‘inline’
>>>

>>> -

[T B]

Thanks for the response thc202.  Based on the links you included and converting to dot notation, here's what I came up with:

$ZAPROXY_HOME/zap.sh -installdir $ZAPROXY_HOME -config database.recoverylog=false -config connection.proxyChain.hostName=myproxyservername.com -config connection.proxyChain.port=3128 -config connection.proxyChain.userName=myusername -config connection.proxyChain.password=mypassword -config connection.proxyChain.enabled=true -config connection.proxyChain.confirmRemoveExcludedDomain=true -config connection.proxyChain.authEnabled=true -config start.checkForUpdates=true -config start.downloadDir=$ZAPROXY_HOME/plugin -addoninstall exportreport -cmd

I receive the same error - "Checks for updates call failed"

Thanks

TB

[thc...@gmail.com]

OK, could you add the following to log4j.properties [1] file:
log4j.logger.httpclient.wire.header=DEBUG
log4j.logger.org.apache.commons.httpclient=DEBUG

and try running again?

That will log (to zap.log file, also in [1]) the connection attempts and
hopefully show what's wrong.


[1] https://github.com/zaproxy/zaproxy/wiki/FAQconfig

Best regards.

[T B]

Attached is the zap.log file with all the INFO lines removed for brevity.

Thanks

TB

[T B]

The workaround I found was to use the GUI to download the exportreport-alpha-3.zap plugin file from another instance and copy it to the ~/.ZAP/plugin directory on the headless machine.

Maybe this will help others having the same issue.

Thanks

TB

[thc...@gmail.com]

Thanks, I guess NTLM messages (challenge/authentication) exchange was
omitted from the log? Just want to make sure that something happened in
the ~15 seconds gap in the log (from 14:01:46,672 to 2017-05-10
14:02:16,501).

The log indicates that ZAP is able to download the add-on information
but "too late" and the add-on is not installed.

Would you mind raising an issue? [1]


[1] https://github.com/zaproxy/zaproxy/issues/new

Best regards.