Incrementally converting large codebase from make to bazel

[Garrett Garcia]

I'm currently writing a proposal to convert a large corporate codebase (~250k source files) from make to bazel.  In order to do this effectively, we'll need to incrementally convert pieces of the codebase.  Is there a recommended method to do this?  What would a hybrid make/bazel build system look like?  Would make call bazel or vice versa?

I'd be very grateful to hear any ideas or recommendations you all have.

Thank you,

Garrett

[Austin Schuh]

How well structured are your makefiles?  You might be able to automatically generate some of the BUILD files to help bootstrap the process.  Even getting 90% of the way there automatically is a huge help.  I've been part of a gyp -> bazel conversion which was mostly that.

Calling make from Bazel will be a pain.  You aren't going to have fun writing rules to expose the intermediate libraries back to bazel.  I think it'll be more work than writing the rules themselves.  You are better off calling Bazel from make.

I think you'll find that you need to first start by converting the core set of libraries, and work out from there.  Bazel is going to generate a bunch of .a's and .so's (one per cc_library), so be aware of that.  If you can structure your rules to match that structure, you'll have a lot easier time with the conversion.

I'd highly recommend providing the compiler through a custom CROSSTOOL file and also controlling the version of bazel in use by either checking in a copy of bazel in at //tools/bazel or writing a shell script to fetch a specific version of bazel and use that.  With 250k source files, you are going to have a lot of developers. The more you can control the dependencies, the less configuration problems you are going to run into, and the more repeatable your build will be.

Good luck!

Austin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/981e4d85-3d1c-4fa6-b95f-66f102f65c28%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Garrett Garcia]

Austin,

Thanks for the detailed write-up!  I've gotten most of the way through a proof-of-concept building a single trivial library via bazel and calling making that bazel call via make.  That part has worked well so far with a single architecture.  However, I need to build for over a dozen architectures and am not sure what the best way to configure the toolchain is.

- Garrett

[Austin Schuh]

Garrett,

It all depends on if you need to build for multiple architectures as part of one single deployable.

For some code bases, we solve that by passing the --cpu=amd64 flag for amd64, --cpu=armhf-debian, --cpu=cortex-m4 flag into bazel and calling it multiple times.  We can get away with that because we only have 1 target platform at a time.  For more complicated repositories, we have written some independent cc_library rules which we then post-fix with the CPU it is targeting (cc_library_cortex_m4, cc_binary_...).  That gives us the ability to build for multiple architectures at once.   It isn't the nicest solution, but it gets us ready for the future.

There is active work upstream to enable building to enable "dynamic configurations".  The basic idea is that you want to be able to instantiate a rule for different platforms. (cc_library can then be re-used for amd64, arm, powerpc, and then combined in a .tar.gz).  Don't hold your breath for it.

Austin

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAEyi_-BHiPFSY-kYi1f5fCpe4XtdEJJDi%3Dz1_ohErZ-Uvt7%3D7A%40mail.gmail.com.