Seeking migration path from large Maven based inhouse project to Bazel

[jochen.h...@kambrium.net]

How can i migrate a 700+ multimodule Maven build to Bazel for a POC? The sheer number makes manually editing BUILD files a torment. In addition, development can't halt, so chances are 300+ developers create changes faster than i can type BUILD files. Here's a couple of options found so far:

- generate_workspace utility works fine for external dependencies, but misses the corresponding generate_build utility

- there's Maven to Gradle migration, and Gradle to Bazel migration, and me being sure that this is not worth trying 

- Maven POMs can be accessed programmatically, so a Maven POM could be used as a POJO mapping into Bazel. Is there any way to persist a Bazel in-memory build into BUILD files? After all, it's Java, right?

Any experiences, hints, thoughts?

[Zalán Meggyesi]

Hi,

Perhaps this maven plugin I've written can give you a hand: https://github.com/zmeggyesi/migrator-maven-plugin

To be honest, I haven't tested it on such a large project, but I'm hoping it does what you need it to do. I'd be happy to have your feedback either way.

Best Regards,

Zalan Meggyesi

[Steren Giannini]

Hi,

Thanks a lot for the feedback.

Indeed, our migration story could be much better. We'd like to  improve the situation.

Would you be able to share with me more details on your current setup and expectations when performing this migration (on this list or privately to ste...@google.com)?

Are you using multiple repositories? It is OK for your team to maintain the 2 build systems while the migration is completed? Do you use Maven profiles or properties, Maven plugins? What is your test setup?

Also please let us know if the plugin suggested by Zalán worked for you.

Steren

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/d8908ff0-c23e-414c-b358-75cc0902df4f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jochen.h...@kambrium.net]

Thanks, looks exactly what i was looking for. Wonder why i did not stumble across it when googling. Maybe add some buzzwords such as "Maven", "Bazel", "Migration", "Conversion" to the README?

I'll give it a spin and let you know.

[jochen.h...@kambrium.net]

Other than the size, we have a pretty standard setup. 

We reference exactly one repository, our inhouse Maven repository server.

Yes, maintaining two build tools during a limited period of time is acceptable of course. Would be very helpful if one could use the "git rebase" pattern against changes in Maven world during this period.

Yes, we use profiles, but only to enable or disable certain plugins in certain builds. If Maven was faster, we'd not use any profiles in alignment w/ Maven best practices.

Yes, we use properties, mostly for pinning dependency versions.

Of course we use plugins, everything in Maven land is a plugin:

- clean, compiler, deploy, install, resources, site, surefier, ear, ejb, jar, war, source

- No reporting, this is done via Sonarqube and Sonargraph.

- Then we have a couple of inhouse and less common third party plugins, but i don't see this as part of a prebuilt migration path. Those have to be considered separately and maybe resolved as custom Skylark rules

Test setup is to (a) run JaCoCo on classes, and (b) feed everything into Sonarqube.

[Dan Fabulich]

We found that we needed to write our own thing, and that it was quickly covered in special cases for internal stuff.

I think we might be able to rip out some of our internal stuff and open source the framework for our thing, but for any medium-to-large sized project, I'd expect it to need a lot of custom code to get it working 100%.

[ittai zeidman]

I'm actually working on a migrator which tries to generate fine-grain bazel packages and targets from coarse grain maven modules. The rough idea is that the big performance gain in bazel requires small(ish) targets so caching and parallelization is more effective. Additionally since we're moving to a mono repo you want to try and remove the "noise" builds from code you're only dependent on "on paper".
To do that we leverage byte code, source code analysis, reverse lookup of classes to external jars and some maven parsing.
One could argue this refinement should be manual after you use one of the existing maven to bazel move with a coarse granularity but I think that refactoring root targets from 100 source files to around ten targets with ~10 source files is very hard since they will need to manually understand who of their consumers needs which finer grain target all the while maintaining also the coarse grain to not break people.
This effort will be open sourced (some of it leverages a third party we use for dependency analysis) but we're in an early stage. I hope to be able to have production code compilation working within the month.

[jochen.h...@kambrium.net]

Zalan,

i gave it a try, i get nice maven_jar() and java_library() entries for my dependencies. What i am missing, and i am also missing this from generate_workspace, is a java_library for my source project. If i throw generate_workspace at a minimal Maven project without any dependencies, it generates an empty BUILD and WORKSPACE file. My hope was to at least get an entry in the BUILD file

java_library(

    name = <artifact name from POM>,

    deps = [ <list of dependencies> ],

    srcs = <default source path such as src/main/java/**/*.java>

)

But thanks for your migrator, it its a nice start to get there.

[Paul Johnston]

Hi Jochen,

You might also try https://github.com/pubref/rules_maven, it computes transitive dependencies as an external WORKSPACE and creates java_library BUILD rules for those transitive dependencies, grouped by runtime/compile/testing dep sets.  For example:

# YOUR WORKSPACE

maven_repository(

    name = "jetty",

    deps = [

        "org.eclipse.jetty:jetty-jmx:9.3.10.v20160621",

        "org.eclipse.jetty:jetty-plus:9.3.10.v20160621",

        "org.eclipse.jetty.websocket:javax-websocket-client-impl:9.3.10.v20160621",

        "org.eclipse.jetty.websocket:javax-websocket-server-impl:9.3.10.v20160621",

    ],

)

# Creates the following java_library rule(s) in $(bazel info output_base)/external/jetty/BUILD:

# Only the 'runtime' rule is shown here...

# AUTO_GENERATED, DO NOT EDIT

java_library(

  name = 'runtime',

  exports = [

    '@javax_annotation_javax_annotation_api//jar',

    '@javax_servlet_javax_servlet_api//jar',

    '@javax_websocket_javax_websocket_api//jar',

    '@org_eclipse_jetty_jetty_annotations//jar',

    '@org_eclipse_jetty_jetty_http//jar',

    '@org_eclipse_jetty_jetty_io//jar',

    '@org_eclipse_jetty_jetty_jmx//jar',

    '@org_eclipse_jetty_jetty_jndi//jar',

    '@org_eclipse_jetty_jetty_plus//jar',

    '@org_eclipse_jetty_jetty_security//jar',

    '@org_eclipse_jetty_jetty_server//jar',

    '@org_eclipse_jetty_jetty_servlet//jar',

    '@org_eclipse_jetty_jetty_util//jar',

    '@org_eclipse_jetty_jetty_webapp//jar',

    '@org_eclipse_jetty_jetty_xml//jar',

    '@org_eclipse_jetty_websocket_javax_websocket_client_impl//jar',

    '@org_eclipse_jetty_websocket_javax_websocket_server_impl//jar',

    '@org_eclipse_jetty_websocket_websocket_api//jar',

    '@org_eclipse_jetty_websocket_websocket_client//jar',

    '@org_eclipse_jetty_websocket_websocket_common//jar',

    '@org_eclipse_jetty_websocket_websocket_server//jar',

    '@org_eclipse_jetty_websocket_websocket_servlet//jar',

    '@org_ow2_asm_asm//jar',

    '@org_ow2_asm_asm_commons//jar',

    '@org_ow2_asm_asm_tree//jar',

  ],

  visibility = ['//visibility:public'],

)

# And then can be used in your BUILD file like so:

java_binary(

  name = "app",

  main_class = "my.App",

  deps = ["@jetty//:runtime"],

)

It also will fetch all the sha1s for the jars too, which you can copy-paste back into step 1 to hermetically seal it.

Hope you find it useful, file issues if you need additional help.

--Paul

[gaura...@gmail.com]

On Sunday, January 22, 2017 at 1:42:43 PM UTC-8, jochen.h...@kambrium.net wrote:

Hi,

How did the transition go for the POC? I am in the same position as you were and would love to know if you found any tools/plugins to make the transition easier.

Thanks,
Gaurav