ZAP Migration to GitHub

[psiinon]

Many people have asked me when we are going to migrate to GitHub, but I've resisted due to the disruption it will cause.
However we are rapidly running out of space on the main zaproxy Google Code repo and I've had no answers to my requests to increase our quota :(

So I think we'll have to migrate ZAP away from Google Code, and GitHub seems to be the obvious choice.
We're also probably going to have to do this _before_ 2.4.0 can be released.

We could migrate the repos as they are now. but I think it makes sense to reassess them to see if we could do anything better.
Some suggestions that have been mentioned before include:

• Using maven to manage the jars
• Having one repo for each add-on

So, my questions for you all:

1. Is GitHub the best place to migrate ZAP to?
2. What changes should we make?
3. Any other thoughts or suggestions??
   

We will need to move fairly fast to avoid delaying 2.4.0 too much, so please post your thoughts asap.

Simon

[Anant Shrivastava]

I vote for Github.

Plus it will also allow up to quickly manage wiki and pages in a better manner. we can design a full fleged website.
we have https://github.com/blog/1547-release-your-software also which can be explored.

[psiinon]

I'm all for a new website (been thinking about that for a while) but lets look at that after migrating ;)

[Kevin W. Wall]

Comments in-line, below.

On Mon, Nov 24, 2014 at 4:28 AM, psiinon <psi...@gmail.com> wrote:

Many people have asked me when we are going to migrate to GitHub, but I've resisted due to the disruption it will cause.
However we are rapidly running out of space on the main zaproxy Google Code repo and I've had no answers to my requests to increase our quota :(

​I take it you've already deleted your old "binaries" from Google Code since you've moved them to SourceForge, right? If not, doing that will recover a lot of space.​

So I think we'll have to migrate ZAP away from Google Code, and GitHub seems to be the obvious choice.

​Chris Schmidt had someone migrate ESAPI to GitHub.

Most (but not all) of the branches were migrated as well. I can put you in touch with the person if you need some operational advice for the migration.
​

 

We're also probably going to have to do this _before_ 2.4.0 can be released.

​That sucks. IMO, it would be better to do it after you push the new release so you automatically know what state you were in when you moved to GitHub...just in case you need to roll back. Probably won't be an issue though.

Are you also going to migrate from Google Issues to JIRA? Same person may be able to help you with pointers regarding that as well.
​

 

We could migrate the repos as they are now. but I think it makes sense to reassess them to see if we could do anything better.
Some suggestions that have been mentioned before include:

• Using maven to manage the jars

​This is definitely a win for everyone involved.​

 

• Having one repo for each add-on

​Long term, that probably would help the space issue *if* GitHub has similar size restrictions, although I'm not sure if they do.​

 

So, my questions for you all:

1. Is GitHub the best place to migrate ZAP to?

​If you wish to move to Git, then probably 'yes'. Although, supposedly Google Code supports Git as
well and you supposedly can convert your repository

from SVN to Git. (It's just not clear what happens to your version history, and especially any branches if you do that. I was unable to find any details regarding that.)​

OTOH, using Git takes a bit of getting used to and certainly a new tool set so it will take a bit of time for some to ramp up if they've not used Git before and/or need to retool.

1. What changes should we make?
2. Any other thoughts or suggestions??

We will need to move fairly fast to avoid delaying 2.4.0 too much, so please post your thoughts asap.

Simon

​FWIW,​

 

-

​kevin​

--

Blog: http://off-the-wall-security.blogspot.com/
NSA: All your crypto bit are belong to us.

[Björn Kimminich]

GitHub is perfectly fine. I never imported a SVN project there directly, but importing a SVN repo into a local Git repo and then publishing to GitHub will definetely work.

If there is no Maven expert on the team as of now, we might consider going the Gradle instead. The learning curve is similar, maybe even a bit easier. And it would be possible to use Maven artifacts while slowly getting rid of the old Ant stuff. It is a bit more "scripty" though, but has much less white noise compared to Mavens XML constructs.

If I remember correctly Simon has a ZAP org on GitHub already, so it would be easy to manage all plugins in separate repos inside that. If you want to use the OWASP org, we would lose this level of grouping ZAP stuff closely. Having individual repos for plugins is additionally nice because they can have their on documentation, which can then be linked from some core place like the main Wiki or so.

As it's free for Open Source we might consider using some nice GitHub-friendly services in the future, e.g. Travis-CI instead of the Mozilla-internal Jenkins, CodeClimate/CoverAlls etc. checking for code smells and test coverage, dependency analysis tools like Versioneye etc.

Regarding the development process I would recommend looking into "Git Flow" which means that the "master" branch is the latest stable release and all development happens on a "develop" branch (or feature branches on top of it): http://nvie.com/posts/a-successful-git-branching-model/

That's my 2ct so far... Cheers, Björn

[psiinon]

We have more space on Google Code: https://twitter.com/ImJasonH/status/537390443638833152 :D
"Your project is using approximately 5086 MB out of 6144 MB total quota."

As we're so close to releasing 2.4.0 we're going to stick with Google Code until after that release.

We can then take our time and decide what we want to do - both re GitHub and re Maven / Gradle etc.
I think we definitely need to do some restructuring (the dependencies in particular) but its not as urgent as it was before our quota was increased :)

Cheers,

Simon

[Niklas Volcz]

Moving to GitHub has lots of benefits. I've seen projects getting much more commits since it is easier to use and more visual.
I have experience in Maven and some in Gradle. Is it a decided that we want to migrate to Maven or Gradle? I would prefer Gradle since it is the next logical step in the evolution. I would be happy to contribute some time to migrate to Gradle.

Best regards,
Niklas

[Niklas Volcz]

I tried to make a copy of the svn repo and upload it to GitHub this weekend.
The first obstacle encountered was the 100MB file size limit. See: https://help.github.com/articles/what-is-my-disk-quota/
This will results in some changes of the development workflow of ZAP.

I noticed the GitHub migration wiki page. https://code.google.com/p/zaproxy/wiki/GitHubMigration
What is the progress on this? What help is needed?

Also noticed the mvn branch. Seems to be dead?

Best regards,
NVolcz

[psiinon]

That wiki page is pretty much where we're at right now :)
As we shouldnt / cant store add-ons and jars in source control the first task is the adoption of a dependency management tool.
I like the look of Gradle, using Maven Central for the add-ons and jars.
But I'm new to both of those so I'm just trying to get my head around how it will all work, while doing a dozen other things at the same time ;)

The kind of things that need to be decided:

• How do developers set up their ZAP development environment?
• How do developers update their ZAP development environment?
• Where do we publish artefacts and how?
  
• How to we build ZAP so that the release doesnt depend on a remote repository

Those kind of things :)

So right now I'm playing with Gradle scripts and investigating Maven Central.

Advice and guidance from anyone who knows about these things much appreciated :)

Cheers,

Simon

[Kevin W. Wall]

Simon,

Have you been in touch with Max Gelman <max.g...@contrastsecurity.com>?
Max migrated ESAPI as well as the ESAPI issues from Google Code to GitHub.
(Max is a colleague of Chris Schmidt.) He may have some scripts that he would
be willing to share (no promises) that you could tweak and get a head-start
in migrating. I must say I was pretty impressed with the migration of the issues
from Google code as it migrated all the labels and history and pretty much
everything. (OTOH, I personally like the Google issues interface a little
better, but I think there is also a JIRA interface to the GitHub issues.)

Note that I haven't mentioned this to Max so I don't promise how he'll
react or if he will be willing or able (as in time) to help you, but he seems
like a nice enough dude and fully capable so I think it is worth a shot.

Of course, OTOH, if I have already mentioned this to you before, just
ignore me and I'm remind you again in an other 6 months. Getting old is
a bitch. :)

-kevin

> --
> You received this message because you are subscribed to the Google Groups
> "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to zaproxy-devel...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Niklas Volcz]

Thanks,
I've now emailed Max.
Also played around with migrating the project to Gradle yesterday. Importing ant task was a breeze! Dependency management was also relatively painless, however I could not migrate all dependencies to be downloaded since all are not available in the maven repos.

What should the next steps be? Which tasks are candidates to be migrated to Gradle?
What project structure changes are suggested to be made in order to get away from the large files?

Best regards,
Niklas Volcz

On Wednesday, February 4, 2015 at 5:49:01 AM UTC+1, Kevin W. Wall wrote:

Simon,

Have you been in touch with Max Gelman <max.gelman@contrastsecurity.com>?

> email to zaproxy-develop+unsubscribe@googlegroups.com.

[psiinon]

Just seen that Google Code is going to close: http://arstechnica.com/information-technology/2015/03/google-to-close-google-code-open-source-project-hosting/

I'll focus much more on migrating ZAP to GitHub once 2.4.0 is released.

Simon

[Niklas Volcz]

I got some replies back from Max. Sends like they didn't have to deal we large binaries the same way ZAP is setup. Also got some feedback from a github staff that we can remove large binaries this way:
https://help.github.com/articles/remove-sensitive-data/

He also used a modified version of:
https://github.com/arthur-debert/google-code-issues-migrator

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

[kingthorin+owaspzap]

Wish the article actually linked to it:
"Exporting to GitHub is probably easiest, as Google has an export-to-GitHub tool"

[Cosmin Stefan-Dobrin]

I think it's this one: https://code.google.com/export-to-github/ . Haven't had time to play with it, but it looks promising.

Cosmin

On Thu, Mar 12, 2015 at 7:37 PM, kingthorin+owaspzap <kingt...@gmail.com> wrote:

Wish the article actually linked to it:
"Exporting to GitHub is probably easiest, as Google has an export-to-GitHub tool"

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[Nathan Jones]

I haven't read through the entire thread but since I just got a notification that Google code is shutting down soon I thought I should share the progress I have already made on this issue.

I've migrated the Zaproxy Googlecode Svn repository to Git. The large binaries prevent it from being pushed to Github so it's on Gitlab:

    https://gitlab.com/ncjones/zaproxy

It's a week since I did the export and I've lost my bash history but I think the commands were:

    $ git svn clone http://zaproxy.googlecode.com/svn

    $ git remote add origin g...@gitlab.com:ncjones/zaproxy.git

    $ git push origin --mirror

This should be a workable repository (`ant -f build/build.xml` succeeds on master) but there are two issues that you may want to resolve before making it official:

1. Author names:

I didn't use the `--authors-file` option so the author names are not pristine. You may want to run `git filter-branch` to tidy these up.

2. Large binary files:

There's a lot of binaries in the Zap source which need to be externalised (especially the OSX JDK!) which make this repo unnecessarily large (I think it's about 850MB compressed). Ideally these would be removed from the entire history to reduce the repo size. This is not trivial but I've done it before. The main issue is adding the dependency management and scripting etc.

Let me know what you think about this. I'm happy to help on boarding or resolving the above issues.

BTW, the reason I got this far is because I was keen on contributing some enhancements to Zaproxy but I can't stand working with Svn.

 - Nathan

[Kevin W. Wall]

You should have gotten a notification directly from Google Code.
They have some tools to migrate everything. I've not tried it,
but intend to try it to migrate the ESAPI C++ code to GitHub.
The link for the tool was in the email that Google Code sent.

-kevin

> --
> You received this message because you are subscribed to the Google Groups
> "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to zaproxy-devel...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Kevin W. Wall]

Google Code suggests using this:

https://code.google.com/export-to-github/

to migrate code to GitHub. It's supposed to migrate source,
wiki, and Google issues (all by the ones with restricted views)
to GitHub.

-kevin

> You received this message because you are subscribed to the Google Groups

> "OWASP ZAP Developer Group" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to zaproxy-devel...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[psiinon]

I did give the Google Export a quick go but it failed as we have too many issues (bug trackers) :(
There are other tools to address this but these take time to play with.
I will focus on the migration as soon as 2.4.0 is released.

Cheers,

Simon

> zaproxy-develop+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to zaproxy-develop+unsubscribe@googlegroups.com.

[psiinon]

Now that 2.4.0 is out I'm going to focus a lot more on this.

I've updated https://code.google.com/p/zaproxy/wiki/GitHubMigration with a list of tasks that I think need to be performed. Which will no doubt grow ;)

I've just done a test migration of the zap-extensions project and that worked fine, so the first steps might be to move the zap-extensions and zaproxy-test projects.

I'll keep updating this thread with progress and questions, especially about restructuring.

Offers of assistance gratefully received ;)

Cheers,

Simon

[psiinon]

I've just migrated the zaproxy-tests project to https://github.com/zaproxy/zaproxy-test
To make changes to this project submit pull requests :)

The wiki pages arent that pretty yet, but they are functional.

If no one reports any problems with this soon then I'll also migrate the zap-extensions project.

We plan to restructure the projects, for example by having one project per add-on, but the initial focus is to just get off Google Code.

Cheers,

Simon

[psiinon]

And the zap-extensions project has now been migrated to https://github.com/zaproxy/zap-extensions :)

Again, the wiki pages arent that great, which is why I havnt set up the project redirect url yet
Any suggestions as to how we could handle the tree index that Google Code supports? (ie on the left hand side of https://code.google.com/p/zap-extensions/wiki/Introduction?tm=6 assuming I havnt redirected that by the time you follow the link;)
Thats useful for the zap-extensions project but pretty much essential for the zaproxy wiki :/

Cheers,

Simon

[Cosmin Stefan-Dobrin]

First of all, I'm really glad we're finally doing the move to Github. :D

Re. the tree index structure, I'm not sure it's going to be easy to make it expand on click. On the other hand, I've discovered that, on Github, we can create a Markdown (or not only) document describing the menu and then make it float on the right side: a sidebar. Details: https://help.github.com/articles/creating-a-sidebar/. The problem is that I don't think Markdown allows creating 'expandable' tree structures.

Cheers,

Cosmin

> zaproxy-devel...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to zaproxy-devel...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--
Blog: http://off-the-wall-security.blogspot.com/
NSA: All your crypto bit are belong to us.

--

You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[psiinon]

Yeah, I've just updated the zap-extensions wiki to include that - how does this look?
https://github.com/zaproxy/zap-extensions/wiki

Cheers,

Simon

> zaproxy-develop+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to zaproxy-develop+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--
Blog: http://off-the-wall-security.blogspot.com/
NSA: All your crypto bit are belong to us.

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[Cosmin Stefan-Dobrin]

That looks good and I think it's going to be fine for most of our use cases.

The only suggestion I have is that it might be good to try and limit the levels to 4 and, if more are needed, link them directly from the 4th level page or embed them directly. For example, all the 3 pages relating to Port Scan could be merged together into the base page, with separate sections for the Options and the Tab.

I've just checked the wiki for the main project and everything seems to be up to level 4, except for `ZAP User Guide-> User Interface -> Dialogs -> Options` and 2 more. In these cases, I think we can either leave it as it is or somehow move things around (e.g. User Interface -> Dialogs -> Options to User Interface -> Options Dialog).

Cosmin

> zaproxy-devel...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OWASP ZAP Developer Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to zaproxy-devel...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--
Blog: http://off-the-wall-security.blogspot.com/
NSA: All your crypto bit are belong to us.

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[Dale Visser]

I notice that the core zaproxy project isn't up on GitHub yet. If the obstacle is large binaries in the repository, there are ways to purge...

I have successfully used the advice here in the past: https://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/

I remember that the script was really slow for me. In the comment thread, a Gist is pointed out where they claim optimizations for speed have been made: https://gist.github.com/nk9/b150542ef72abc7974cb

[psiinon]

Thanks, we'll probably need that :)
The first obstacle is the fact we have too many issues.
This tool is provided: https://code.google.com/p/support-tools/wiki/IssueExporterTool but I havnt had a chance to try it yet.
And if we have to remove the jars we depend on then we'll have to adopt a dependency management tool as well.
Which I'm all for, it just all takes time :(

Cheers,

Simon

[kingthorin+owaspzap]

I've been trying for a long time to cleanup our issue tracker and I've made some progress. Unfortunately due to work and life I kind of only get to tackle it about 30mins to and hour at a time and once a month (or every other month). With the pace things come in and giving requesters a reasonable chance to provide further details, it's really hard to get ahead.

We currently have ~350 open issues. Which to me doesn't seem like too much, do we expect/want to move all the closed/historic issues as well?

[java bean]

Could we migrate the Ant build to a more modern Maven build ? Let me know if i can help here.

I havel already updated the Maven plugin for ZAP to the latest Maven and jdk version and ZAP client api : https://github.com/javabeanz/zap-maven-plugin2

Cheers,

javabean.

Op donderdag 7 mei 2015 17:43:20 UTC+2 schreef kingthorin+owaspzap:

[Niklas Volcz]

Gradle provides nice functionality to call ant tasks. I would say gradle is to prefer over maven.

Best regards,
Niklas

--

[Björn Kimminich]

For an Ant-heavy project like ZAP Gradle definitely has the easier migration path. Maven will also do its best to prevent future scripting in the build process -  which you can either love or hate it for... I'd go for Gradle.

[yha...@gmail.com]

I haven't any experience in both Maven and Gradle. But I red about a Gradle daemon with a 1Gb heap that run in execution under the hood consuming lot of resources... is that true?
Besides that I saw that lots of Environment support Maven directly (e.g. Netbeans that is the IDE I use support it without any plugin). This was very useful for me, because when I migrated from SVN to Git I benefit by the native implementation of Git inside Netbeans and I'd only to solve issues related to project automation respect to IDE plugin compatibility...

Il giorno martedì 12 maggio 2015 16:05:02 UTC+2, Björn Kimminich ha scritto:

For an Ant-heavy project like ZAP Gradle definitely has the easier migration path. Maven will also do its best to prevent future scripting in the build process -  which you can either love or hate it for... I'd go for Gradle.

Am 12.05.2015 12:08 schrieb Niklas Volcz <niklas...@gmail.com>:

Gradle provides nice functionality to call ant tasks. I would say gradle is to prefer over maven.

Best regards,
Niklas

On May 12, 2015 10:11, "java bean" <java.dev...@gmail.com> wrote:

Could we migrate the Ant build to a more modern Maven build ? Let me know if i can help here.

I havel already updated the Maven plugin for ZAP to the latest Maven and jdk version and ZAP client api : https://github.com/javabeanz/zap-maven-plugin2

Cheers,

javabean.

Op donderdag 7 mei 2015 17:43:20 UTC+2 schreef kingthorin+owaspzap:

I've been trying for a long time to cleanup our issue tracker and I've made some progress. Unfortunately due to work and life I kind of only get to tackle it about 30mins to and hour at a time and once a month (or every other month). With the pace things come in and giving requesters a reasonable chance to provide further details, it's really hard to get ahead.

We currently have ~350 open issues. Which to me doesn't seem like too much, do we expect/want to move all the closed/historic issues as well?

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[java bean]

By supporting Maven, Eclipse, SpringSource STS, Netbeans and IntelliJ users can contribute to ZAP. Ant is rarely used anymore except for Netbeans users.

On Tue, May 12, 2015 at 6:43 PM, <yha...@gmail.com> wrote:

I haven't any experience in both Maven and Gradle. But I red about a Gradle daemon with a 1Gb heap that run in execution under the hood consuming lot of resources... is that true?
Besides that I saw that lots of Environment support Maven directly (e.g. Netbeans that is the IDE I use support it without any plugin). This was very useful for me, because when I migrated from SVN to Git I benefit by the native implementation of Git inside Netbeans and I'd only to solve issues related to project automation respect to IDE plugin compatibility...

Il giorno martedì 12 maggio 2015 16:05:02 UTC+2, Björn Kimminich ha scritto:

For an Ant-heavy project like ZAP Gradle definitely has the easier migration path. Maven will also do its best to prevent future scripting in the build process -  which you can either love or hate it for... I'd go for Gradle.

Am 12.05.2015 12:08 schrieb Niklas Volcz <niklas...@gmail.com>:

Gradle provides nice functionality to call ant tasks. I would say gradle is to prefer over maven.

Best regards,
Niklas

On May 12, 2015 10:11, "java bean" <java.dev...@gmail.com> wrote:

Could we migrate the Ant build to a more modern Maven build ? Let me know if i can help here.

I havel already updated the Maven plugin for ZAP to the latest Maven and jdk version and ZAP client api : https://github.com/javabeanz/zap-maven-plugin2

Cheers,

javabean.

Op donderdag 7 mei 2015 17:43:20 UTC+2 schreef kingthorin+owaspzap:

I've been trying for a long time to cleanup our issue tracker and I've made some progress. Unfortunately due to work and life I kind of only get to tackle it about 30mins to and hour at a time and once a month (or every other month). With the pace things come in and giving requesters a reasonable chance to provide further details, it's really hard to get ahead.

We currently have ~350 open issues. Which to me doesn't seem like too much, do we expect/want to move all the closed/historic issues as well?

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

[Niklas Volcz]

I'm a long time Maven user and have since some time back starting to
use Gradle wherever possible. It has been a blast so far. Haven't
noticed any excessive memory usage.
Big projects like Google Android and Spring have already migrated to Gradle.
Gradle has nice plugins that makes migration a breeze and for
generating project files for most IDE's (I've only used this with
Eclipse and IntelliJ).

Best regards,
Niklas

May 12, 2015 18:43, <yha...@gmail.com> wrote:
>
> I haven't any experience in both Maven and Gradle. But I red about a Gradle daemon with a 1Gb heap that run in execution under the hood consuming lot of resources... is that true?
> Besides that I saw that lots of Environment support Maven directly (e.g. Netbeans that is the IDE I use support it without any plugin). This was very useful for me, because when I migrated from SVN to Git I benefit by the native implementation of Git inside Netbeans and I'd only to solve issues related to project automation respect to IDE plugin compatibility...
>
>
> Il giorno martedì 12 maggio 2015 16:05:02 UTC+2, Björn Kimminich ha scritto:
>>
>> For an Ant-heavy project like ZAP Gradle definitely has the easier migration path. Maven will also do its best to prevent future scripting in the build process - which you can either love or hate it for... I'd go for Gradle.
>>
>> Am 12.05.2015 12:08 schrieb Niklas Volcz <niklas...@gmail.com>:
>>>
>>> Gradle provides nice functionality to call ant tasks. I would say gradle is to prefer over maven.
>>>
>>> Best regards,
>>> Niklas
>>>
>>> On May 12, 2015 10:11, "java bean" <java.dev...@gmail.com> wrote:
>>>>
>>>> Could we migrate the Ant build to a more modern Maven build ? Let me know if i can help here.
>>>>
>>>> I havel already updated the Maven plugin for ZAP to the latest Maven and jdk version and ZAP client api : https://github.com/javabeanz/zap-maven-plugin2
>>>>
>>>> Cheers,
>>>>
>>>> javabean.
>>>>
>>>> Op donderdag 7 mei 2015 17:43:20 UTC+2 schreef kingthorin+owaspzap:
>>>>>
>>>>> I've been trying for a long time to cleanup our issue tracker and I've made some progress. Unfortunately due to work and life I kind of only get to tackle it about 30mins to and hour at a time and once a month (or every other month). With the pace things come in and giving requesters a reasonable chance to provide further details, it's really hard to get ahead.
>>>>>
>>>>> We currently have ~350 open issues. Which to me doesn't seem like too much, do we expect/want to move all the closed/historic issues as well?
>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
>>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

>>>> To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

[java bean]

ok, if that's true, then Gradle works for me too. Does Netbeans have Gradle support too ? With all the Ant in ZAP now, I had the impression ZAP was built with Netbeans.

Cheers,

sytze.

[psiinon]

My plan is to adopt Gradle :)
It will probably be a gradual move over, as we have a lot of Ant tasks that we use, eg for generating add-ons and releases.

Cheers,

Simon

>>>> To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[mawoki]

Hi,

it sounds super great to move away from ANT to a more standardized
build tool.
Just to give my 2cents: I propose Maven.

I use Maven for years. Me and my team migrated a handful projects to
Gradle. I would consider my knowledge to be a little bigger for Maven,
than for Gradle

To my experience, the Maven tool ecosystem is way more sophisticated.
I use IntelliJ a lot. Gradle tooling/plugins are still missing a lot
of handy features.

If I compare build speed and performance, I don't see any difference.
This is a draw in my eyes.

Gradle had a strong development the last months. We where faced with
breaking plugins and in-compatible build descriptions.
Thus, we where forced to migrate our build scripts to newer Gradle
version a few times.
Maven development slowed down, due to its matureness, thus is more
stable.

What I like about Maven is its declarative nature.
Gradle is based on Groovy and thus offers direct "scripting" support.
This a chance and a risk in the same moment.
To me, the biggest risk is to use a lot of groovy magic to make things
happen. This solves the build problem in the first run, but makes it
harder for new/other developers to get on board.
Maven's XML is truly verbose compared to that, but offers at least an
easier starting point to Google for (XML) element names, when it comes
to understand others code

_If_ you still prefer to migrate to Gradle, I really want to encourage
you to use 'Gradle Wrapper' - this freezes the Gradle version somehow
and leads to more stability and less reasons to migrate your scripts
the next month.
The wrapper is the preferred way to start new Gradle builds, see
http://gradle.org/docs/current/userguide/gradle_wrapper.html


Regards
Martin

>>>> To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zaproxy-develop/H3GzoTf9MEI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zaproxy-devel...@googlegroups.com.

[psiinon]

I dont really know Gradle or Maven, so I'm easy with either :)
And as I dont know enough about them its difficult for me to decide which of these we should use and how we should use them :(

So I'm going to crowd source this, ie get you lot to help :D

Rather than try to solve all of our build problems in one go I'm going to try to get us to solve them one at a time one.

So the current situation is...

• We need to migrate from Google Code asap
• The zap-extensions project was migrated with few problems, so we can carry on using Ant to build the add-ons for now
• The zaproxy project is failing to migrate for a variety of reasons, one of which being the big files we have in the repo (esp the jars)
  

I would like to migrate the history to github, but thats proving very hard.

However it looks like the automated migration process to Source Forge works well. Before you all get too upset, no I'm not suggesting we move the source code there, but we could use it as the history archive and then just migrate the zaproxy trunk and 2.4 branch to github without any history.

The most pressing problem is then how to build and run ZAP without the jars and main add-ons in github.
Nearly all of the jars we use are on Maven Central, and I have no problem with us maintaining the rest (just one I think) as an artifact.

So ... starting with a standard version of Eclipse, whats needed to allow us to build ZAP?
I'm looking for a set of detailed instructions including which plugins to use, working config files, and anything else needed to make this work :)
We definitely also want to support other IDEs but for now I just want to be able to build and run ZAP in Eclipse from a github zaproxy project which contains no jars and add-ons.

Over to you lot!

Simon

[Colm O'Flaherty]

is the current thinking that we will not have the zaproxy history on github?

What's the specific issue?  Max file sizes?

--
You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[psiinon]

Yeah, we're struggling to import the history due to the large files.

If anyone can make this work then details would be very much appreciated :)

Cheers,

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-develop+unsubscribe@googlegroups.com.

[Colm O'Flaherty]

I've put together a working set of steps to build ZAP extensions (alpha, beta, release), using Eclipse Luna, on Kali Linux.  This is against GitHub, obviously.  It's a lot easier than running Eclipse against SVN, as it turns out, because it doesn't need any flaky third party plugins.

Since the Git repo for the zaproxy piece isn't yet fully set up, I can't test that piece.

Where do you want me to put it?

Colm

On 15 May 2015 at 10:47, psiinon <psi...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[Colm O'Flaherty]

I'm assuming that the zap core project (minus JARs) is: https://github.com/zaproxy/zaproxy-test

Oder?

Colm

On 15 May 2015 at 10:47, psiinon <psi...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "OWASP ZAP Developer Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zaproxy-devel...@googlegroups.com.

[mawoki]

I think no, because its stated in the headline:

"The zaproxy-test project contains all developer's test code for the OWASP Zed Attack Proxy (ZAP) and its extensions. This includes (but is not limited to)"

Regards

Martin

[mawoki]

Regarding the history ...

I'm curoius, what/where the actual problems are.

Is there a list/blog/whatnot describing what was alredy tried and how it failed?

[mawoki]

Hi,

I've started a Maven migration of Zaproxy.

It already compiles main sources and test sources and runs all tests automatically.

Alle dependencies, except one is retrieved from Maven Central.

It works smoothly from command line (Win 8.1) and from within IntelliJ IDE without any special tweaking.

My current work is available on https://github.com/nitram509/zaproxy-maven

At the moment I try to run ZAP.class directly from the IDE, which still gives me some headaches ;-)

I've wrote my thoughts down:

https://docs.google.com/spreadsheets/d/1XVkfq9L3ADDLlLt6ovZR14PCpFHKtk0Vckg6g32l_DU/edit?usp=sharing

Feedback is very welcome.

Regards

Martin

[Björn Kimminich]

Martin is right, it is the remains of the unit/integration tests that could not yet be moved into the core project under /src/test. The project should be obsolete in the long run.

Cheers,
Björn

[Colm O'Flaherty]

From what I can see, the 100MB GitHub file size issue only applies to the following files... (I'm looking at the 2.4 branch and the trunk).  I'm specifically excluding SVN internal files from the file size search, since I *assume* that these are merely incidentally processed by the Git migration tool, and are not the cause of the issue.  Can whoever has done this please confirm / refute?
    ./zaproxy-2.4/build/ZAP_2.4.0_Mac_OS_X.zip
    ./zaproxy-2.4/build/ZAP_Dev Build_Mac_OS_X.zip
    ./zaproxy/build/ZAP_Dev.zip
    ./zaproxy/build/ZAP_Dev Build_Mac_OS_X.zip

Assuming that is the case, I see no requirement to move to either Maven or Gradle just yet, since the JAR files are not the cause of the migration failing.  I'm not against using Maven/Gradle, but I'm against doing it right now (unless there is no alternative), since it just complicates the Git migration.

Since all of the large files named above are deliverables from the build process, I propose we try the following:
1) Delete the files in question from the Google Code SVN repo and history, using something like the following: http://subversion.apache.org/faq.html#removal

2) Include a Git ignore directive to prevent these files from being stored in the GitGub repo at all?

3) Since the number of issues in Google Code is still an issue (ahem!), we could then try the manual migration mechanism from Google Code to GitHub: https://code.google.com/p/support-tools/wiki/MigratingToGitHub

4) If all the above works, then we can look at migrating to Maven/Gradle/Anything But Ant (I like Ant, but each to their own).

It might work.. maybe.

Colm

[mawoki]

Hi,

this seems very pragmatic to me.

Decoupling Github migration and Build-Tool migration enables to do baby steps, and makes (as said) the Github migration easier.

I also think in any case its not a good idea to store deployable artefacts (like these huge files in build folder) to a source code management tool.

For me, my Github name is different from my google code ones.

Is there already a plan, how to map user names from the commits?

Regards

Martin

[psiinon]

The plan is to migrate the source code (minus big files), issues and wiki next week, ie the week beginning 1st June.
We have already done some test migrations but are still fine-tuning the process ;)

We will be keeping the jars in the repo for now to decouple the migration and and build changes required.

So ... if you have committed ANY changes to the zaproxy repo please email me (psiinon at gmail.com) your github username and the email address(es) you used to commit those changes ASAP.

I'll post here as soon as we have a definite date for the migration.

Cheers,

Simon

[psiinon]

More progress to report :)

We've decided to split the ZAP Core Help files out into a separate repo, and have migrated that as a test case for the full zaproxy repo.

Its now here: https://github.com/zaproxy/zap-core-help
With the html version of the user guide on the wiki: https://github.com/zaproxy/zap-core-help/wiki

Please have a look at it and let us know if you spot any problems.
Note that the add-on help pages have _not_ yet been generated for the wiki, so we know about that problem.
Add pull requests for the (English) help files much appreciated - and you can help translate them via Crowdin.

The GitHub accounts we knew about (up to some point yesterday) should have been correctly linked to the relevant changes.
I sent out another email to the people we dont have GitHub accounts for this morning - thanks to those of you who have replied, and if you did get an email but havnt replied then please do so asap!

We plan to start the final zaproxy migration .. (pauses for effect) .. Tomorrow!

Cheers,

Simon

[psiinon]

FYI all permissions to the Google Code zaproxy project have been removed pending the migration to GitHub.

[Dale Visser]

This is great! Getting the documentation updated is now as simple as providing a pull request. :-)

[psiinon]

The wiki has been migrated to https://github.com/zaproxy/zaproxy/wiki

Its not perfect, but most of the links should work.

We're still migrating the issues and source code - we keep getting blocked by GitHub anti abuse measures, and they havnt replied to my request for help with this yet :/
So source code and issues links may well not work.

I've just updated the Google Code help wiki pages to point to the GitHub wiki, eg see https://code.google.com/p/zaproxy/wiki/HelpIntro

I plan to do the same for the other wiki pages unless someone screams soon ;)

Note that we dont want to use the Google Code project redirection mechanism as this will break the 'check for updates' mechanism for older versions of ZAP.

Cheers,

Simon

On Wednesday, 3 June 2015 15:42:34 UTC+1, psiinon wrote:

[psiinon]

OK, the source code and issues have now been migrated as well :D

There will be a lot of tidying up to do - I've got a list of things to change (like the Hacking ZAP series) but feel free to post to this thread or email me directly if you notice ZAP links that still go to Google Code.
I'll also try to document how to set up a new ZAP dev environment asap.

We will need to rethink things like the development workflow and how we'd like to restructure the wiki, but I'll start separate threads for those.

Cheers,

Simon

[kingthorin+owaspzap]

That's great news.

Has anyone figured out how to "pickup" issues on github? I just went over there expecting a somewhat obvious assignment/ownership mechanism and am not seeing it.

I did manage to google'fu the following:
https://help.github.com/articles/assigning-issues-and-pull-requests-to-other-github-users/
But I don't see the checkbox mentioned in step 4 :(

[Dale Visser]

OK. This time on the actual Groups thread...

I assume the list of users who can assign issues is very short. I'm not clear on GitHub's permission model for this. It is also true that pull requests and issues are somewhat conflated in GitHub.

My suggestion would be to add a comment to your issue that you are working on it, maybe @tagging a project owner to get their attention so they'll assign it to you.

[psiinon]

I think its time for a new thread on the new workflow ;)
Will start one now...