make and C++ dependencies

[Shubov, Igor]

Hi,

 

Interesting product guys!

 

A few questions:

1.       If I have an existing system built with a bunch of Makefiles or maven pom files or build.gradle files for that matter… how can I migrate to Bazel based builds? In a large org, it’s not realistic to ask developers to re-write all the Makefiles, etc. How did Google migrate from a make based build to Bazel?

2.       How do you handle dependency resolution for C++? To have the builds reproducible you need to make sure that the build variants (e.g. OS, Architecture, Compiler with flags) are exactly the same and dependencies are resolved appropriately. Therefore, one will need to have a coordinate system that fits C++ or native builds (i.e. this is where maven GAV falls on its face)

3.       Are you planning to support Nuget for .NET C# dependency resolution and NPM for node?

 

Thanks,

Igor Shubov

CREDIT SUISSE

Information Technology | Developer Services, KFYF

One Madison Avenue | 10010 New York | United States

Phone +1 212 325 0415

igor....@credit-suisse.com | www.credit-suisse.com

 

==============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================

[Maurizio Vitale]

On Fri, Mar 27, 2015 at 5:17 PM, Shubov, Igor <igor....@credit-suisse.com> wrote:

Hi,

 

Interesting product guys!

 

A few questions:

1.       If I have an existing system built with a bunch of Makefiles or maven pom files or build.gradle files for that matter… how can I migrate to Bazel based builds? In a large org, it’s not realistic to ask developers to re-write all the Makefiles, etc. How did Google migrate from a make based build to Bazel?

I cannot really answer for Google as I was there after the Makefile->Blaze migration was complete (and I'm not at Google anymore), but a rewrite is essentially what you have to accomplish. You can go in steps and try to automate as many as possible, but nobody can really help you in general: a makefile is essentially a shell script that can do arbitrary things.

The open source Bazel might not be compelling enough for large organizations yet, as it is lacking compiling in a compile farm, integration with version control and leaving artefacts in the cloud. But combined with distcc and cccache should be good enough for most midsize organizations and probably better than what most organizations have in-house.

2.       How do you handle dependency resolution for C++? To have the builds reproducible you need to make sure that the build variants (e.g. OS, Architecture, Compiler with flags) are exactly the same and dependencies are resolved appropriately. Therefore, one will need to have a coordinate system that fits C++ or native builds (i.e. this is where maven GAV falls on its face)

Not entirely sure what you mean here, but the general philosophy is that all dependencies are declared. So, differently from other build systems where C++ is inspected for #includes the BUILD file contains the information.

There're debug modes that ensure that compilation don't use more than what is declared and you can (and should) use off-line tools for populating that information. With even better tools, you don't have to do anything at all: you use C++ features, some tools (look for include-what-you-use) figures out which includes you need and adds them to your file. The same or separate tool add the necessary dependencies to the BUILD file.

3.       Are you planning to support Nuget for .NET C# dependency resolution and NPM for node?

 

Thanks,

Igor Shubov

CREDIT SUISSE

Information Technology | Developer Services, KFYF

One Madison Avenue | 10010 New York | United States

Phone +1 212 325 0415

igor....@credit-suisse.com | www.credit-suisse.com

 

==============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To post to this group, send email to bazel-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/DD0E7E593AE6704E9D80F7E52A93B1CC14D9659A%40USW20015565.gbl.ad.hedani.net.
For more options, visit https://groups.google.com/d/optout.

[Kristina Chodorow]

On Sat, Mar 28, 2015 at 11:19 AM, Maurizio Vitale <mrz...@gmail.com> wrote:

On Fri, Mar 27, 2015 at 5:17 PM, Shubov, Igor <igor....@credit-suisse.com> wrote:

Hi,

 

Interesting product guys!

 

A few questions:

1.       If I have an existing system built with a bunch of Makefiles or maven pom files or build.gradle files for that matter… how can I migrate to Bazel based builds? In a large org, it’s not realistic to ask developers to re-write all the Makefiles, etc. How did Google migrate from a make based build to Bazel?

I cannot really answer for Google as I was there after the Makefile->Blaze migration was complete (and I'm not at Google anymore), but a rewrite is essentially what you have to accomplish. You can go in steps and try to automate as many as possible, but nobody can really help you in general: a makefile is essentially a shell script that can do arbitrary things.

The open source Bazel might not be compelling enough for large organizations yet, as it is lacking compiling in a compile farm, integration with version control and leaving artefacts in the cloud. But combined with distcc and cccache should be good enough for most midsize organizations and probably better than what most organizations have in-house.

One thing you could do to start with is writing genrules (http://bazel.io/docs/build-encyclopedia.html#genrule) that call make.  You'll probably run into issues with Bazel clearing the environment before running make, so you'll have to figure out what environment variables make is actually depending on and provide them in the genrule.

 

2.       How do you handle dependency resolution for C++? To have the builds reproducible you need to make sure that the build variants (e.g. OS, Architecture, Compiler with flags) are exactly the same and dependencies are resolved appropriately. Therefore, one will need to have a coordinate system that fits C++ or native builds (i.e. this is where maven GAV falls on its face)

Not entirely sure what you mean here, but the general philosophy is that all dependencies are declared. So, differently from other build systems where C++ is inspected for #includes the BUILD file contains the information.

There're debug modes that ensure that compilation don't use more than what is declared and you can (and should) use off-line tools for populating that information. With even better tools, you don't have to do anything at all: you use C++ features, some tools (look for include-what-you-use) figures out which includes you need and adds them to your file. The same or separate tool add the necessary dependencies to the BUILD file.

At the moment the tools you use are linked to tools on your local filesystem.  To get reproducible builds, you'll have to commit the tools you're using to your repository or reference a remote "golden tool" everyone can use in the WORKSPACE file.

3.       Are you planning to support Nuget for .NET C# dependency resolution and NPM for node?

I'm not familiar with Nuget, but for node we'd love to see someone add Skylark rules for it (http://bazel.io/docs/skylark/index.html). 

 

 

Thanks,

Igor Shubov

CREDIT SUISSE

Information Technology | Developer Services, KFYF

One Madison Avenue | 10010 New York | United States

Phone +1 212 325 0415

igor....@credit-suisse.com | www.credit-suisse.com

 

==============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To post to this group, send email to bazel-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/DD0E7E593AE6704E9D80F7E52A93B1CC14D9659A%40USW20015565.gbl.ad.hedani.net.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To post to this group, send email to bazel-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAAeLbQ%2BHPjgUwhbU7%2BO1jDa7gV9%2Bq8QXrr2fjP%3DcJbtGvsgH9g%40mail.gmail.com.

[Shubov, Igor]

Thanks for your answers! I’m interested in contributing and will take a look at the rules for building node, etc.

 

One of your answers: At the moment the tools you use are linked to tools on your local filesystem.  To get reproducible builds, you'll have to commit the tools you're using to your repository or reference a remote "golden tool" everyone can use in the WORKSPACE file.

 

This is relevant to my question about coordinates to resolve C++ libraries… if the files are not on the files system and need to be resolved from a remote Definitive Software Library then you’ll need something along the lines of: http://mark.donszelmann.org/maven-nar-plugin/aol.html

 

Committing tools to a repository is not great, unless ALL of your projects are using a single repository.

 

Igor Shubov

Developer Services

+1 212 325 0415 (*105 0415)

[Kristina Chodorow]

 

 One of your answers: At the moment the tools you use are linked to tools on your local filesystem.  To get reproducible builds, you'll have to commit the tools you're using to your repository or reference a remote "golden tool" everyone can use in the WORKSPACE file.

 

This is relevant to my question about coordinates to resolve C++ libraries… if the files are not on the files system and need to be resolved from a remote Definitive Software Library then you’ll need something along the lines of: http://mark.donszelmann.org/maven-nar-plugin/aol.html

Well, note https://github.com/google/bazel/blob/master/tools/cpp/CROSSTOOL, which seems to serve a similar function (not that that's optimal, there's work in progress on configurations). 

 

Committing tools to a repository is not great, unless ALL of your projects are using a single repository.

It doesn't necessarily need to be committed to your source repository, but it needs to be somewhere everyone can access or people will be using different tools to build.

[Paulo Gallo]

On Saturday, March 28, 2015 at 8:19:03 AM UTC-7, Maurizio Vitale wrote:

On Fri, Mar 27, 2015 at 5:17 PM, Shubov, Igor <igor....@credit-suisse.com> wrote:

Hi,

 

Interesting product guys!

 

A few questions:

1.       If I have an existing system built with a bunch of Makefiles or maven pom files or build.gradle files for that matter… how can I migrate to Bazel based builds? In a large org, it’s not realistic to ask developers to re-write all the Makefiles, etc. How did Google migrate from a make based build to Bazel?

I cannot really answer for Google as I was there after the Makefile->Blaze migration was complete (and I'm not at Google anymore), but a rewrite is essentially what you have to accomplish. You can go in steps and try to automate as many as possible, but nobody can really help you in general: a makefile is essentially a shell script that can do arbitrary things.

The open source Bazel might not be compelling enough for large organizations yet, as it is lacking compiling in a compile farm, integration with version control and leaving artefacts in the cloud. But combined with distcc and cccache should be good enough for most midsize organizations and probably better than what most organizations have in-house.

2.       How do you handle dependency resolution for C++? To have the builds reproducible you need to make sure that the build variants (e.g. OS, Architecture, Compiler with flags) are exactly the same and dependencies are resolved appropriately. Therefore, one will need to have a coordinate system that fits C++ or native builds (i.e. this is where maven GAV falls on its face)

Not entirely sure what you mean here, but the general philosophy is that all dependencies are declared. So, differently from other build systems where C++ is inspected for #includes the BUILD file contains the information.

There're debug modes that ensure that compilation don't use more than what is declared

Are you referring to debug modes in blaze itself?

If yes, how can we enable this?

If not, can you elaborate on this one.

Any help is appreciated.

Thanks,

-Paulo

[Kristina Chodorow]

Since the last post on this thread, we've implemented Bazel doing this by default (on Linux).  If you use anything you don't declare in your BUILD files, it won't be included in the build's sandbox and your build will fail.  See http://bazel.io/blog/2015/09/11/sandboxing.html.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/38c7b8c3-121c-42f8-8d21-fd6717527187%40googlegroups.com.

[Paulo Gallo]

Ok, thanks a lot.

-Paulo