Custom/multiple toolchains + use specific toolchains for specific targets?

[Marc-Antoine Courteau]

Hi,

I do embedded development on multiple platforms (e.g. AVR, ARM, etc.), and would like to:

1. add the toolchains (e.g. avr-gcc, gcc-arm-none-eabi) for these platforms to my bazel workspace, and
2. be able to specify which toolchains to use when building specific targets (e.g., "this cc_binary should be built for the AVR chip X using the avr-gcc toolchain, and this cc_binary should be built for ARM chip Y using the gcc-arm-none-eabi toolchain").
   

I also have code that is platform-agnostic that I'd like to reuse for all platforms, and have unit tests for that code that I'd like to run on the host computer.

Is that something that can be done with bazel? Any pointers on how to do this?

Thanks!

[Kristina Chodorow]

1. For the toolchains, you can add them to the CROSSTOOL file[1] or create your own CROSSTOOL file and then specify --crosstool_top[2] when you build.

2.  Have you seen the configuration machinery[3] bazel has?  You could do something like:

cc_binary(

    name = "a",

    srcs = select({

        ":platform-a": ["a.cc"],

    }),

)

config_setting(

    name = "platform-a",

    values = {

        "cpu": "arm-chip-y-or-something",

        "crosstool_top": "gcc-arm-none-eabi",

    },

)

Then you'll only be able to build :a with the for ARM chip Y using the gcc-arm-none-eabi toolchain.  You have to specify the options on the command line, though, e.g.,

$ bazel build --crosstool_top=gcc-arm-none-eabi --cpu=arm-chip-y-or-something //my:a

You can't specify the build options in the cc_ rule itself.

[1]: https://github.com/google/bazel/blob/master/tools/cpp/CROSSTOOL

[2]: http://bazel.io/docs/bazel-user-manual.html#flag--crosstool_top

[3]: http://bazel.io/docs/build-encyclopedia.html#select

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c417d4c4-1bff-41de-b0fd-29957b58e6ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marc-Antoine Courteau]

Thanks for the fast reply! More questions inline below.

On Tue, Jun 16, 2015 at 4:58 PM, Kristina Chodorow <kcho...@google.com> wrote:

1. For the toolchains, you can add them to the CROSSTOOL file[1] or create your own CROSSTOOL file and then specify --crosstool_top[2] when you build.

I currently symlink the tools directory to the one from Bazel, to stay up-to-date as I update Bazel. Can I put the CROSSTOOL file elsewhere than in tools/cpp/ and use that version without specifying --crosstool_top for every build? E.g. by putting the location of the CROSSTOOL file in the WORKSPACE file in the root of my repository?

 

2.  Have you seen the configuration machinery[3] bazel has?

A bit, yes.

 

You could do something like:

cc_binary(

    name = "a",

    srcs = select({

        ":platform-a": ["a.cc"],

    }),

)

config_setting(

    name = "platform-a",

    values = {

        "cpu": "arm-chip-y-or-something",

        "crosstool_top": "gcc-arm-none-eabi",

    },

)

Then you'll only be able to build :a with the for ARM chip Y using the gcc-arm-none-eabi toolchain.

SG - thanks!

 

  You have to specify the options on the command line, though, e.g.,

$ bazel build --crosstool_top=gcc-arm-none-eabi --cpu=arm-chip-y-or-something //my:a

You can't specify the build options in the cc_ rule itself.

Ah, that's what I was hoping I could do. :)

Is that something that I could do with Skylark (or some other way), or is there really no way to have this set per-target (except maybe forcing these flags when building a target via e.g. some shell scripts)?

 

[1]: https://github.com/google/bazel/blob/master/tools/cpp/CROSSTOOL

[2]: http://bazel.io/docs/bazel-user-manual.html#flag--crosstool_top

[3]: http://bazel.io/docs/build-encyclopedia.html#select

On Tue, Jun 16, 2015 at 4:03 PM, Marc-Antoine Courteau <macou...@gmail.com> wrote:

Hi,

I do embedded development on multiple platforms (e.g. AVR, ARM, etc.), and would like to:

1. add the toolchains (e.g. avr-gcc, gcc-arm-none-eabi) for these platforms to my bazel workspace, and
2. be able to specify which toolchains to use when building specific targets (e.g., "this cc_binary should be built for the AVR chip X using the avr-gcc toolchain, and this cc_binary should be built for ARM chip Y using the gcc-arm-none-eabi toolchain").
   

I also have code that is platform-agnostic that I'd like to reuse for all platforms, and have unit tests for that code that I'd like to run on the host computer.

Is that something that can be done with bazel? Any pointers on how to do this?

Thanks!

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c417d4c4-1bff-41de-b0fd-29957b58e6ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marc-Antoine Courteau, ing.
macou...@gmail.com

[Greg Estren]

Not today, but at some point in the future there should be much better options for this. Gory details: https://docs.google.com/document/d/1uoU8t7loTOu6uyzez-ilhcYgEXg4xjViJu0aqrf69TY/edit

Practically I expect the machinery to support this to evolve incrementally over time - the more time goes on the closer we'll be able to get to this ideal.

 

 

[1]: https://github.com/google/bazel/blob/master/tools/cpp/CROSSTOOL

[2]: http://bazel.io/docs/bazel-user-manual.html#flag--crosstool_top

[3]: http://bazel.io/docs/build-encyclopedia.html#select

On Tue, Jun 16, 2015 at 4:03 PM, Marc-Antoine Courteau <macou...@gmail.com> wrote:

Hi,

I do embedded development on multiple platforms (e.g. AVR, ARM, etc.), and would like to:

1. add the toolchains (e.g. avr-gcc, gcc-arm-none-eabi) for these platforms to my bazel workspace, and
2. be able to specify which toolchains to use when building specific targets (e.g., "this cc_binary should be built for the AVR chip X using the avr-gcc toolchain, and this cc_binary should be built for ARM chip Y using the gcc-arm-none-eabi toolchain").
   

I also have code that is platform-agnostic that I'd like to reuse for all platforms, and have unit tests for that code that I'd like to run on the host computer.

Is that something that can be done with bazel? Any pointers on how to do this?

Thanks!

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c417d4c4-1bff-41de-b0fd-29957b58e6ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marc-Antoine Courteau, ing.
macou...@gmail.com

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAH_sd0VfWtaWDHCv8gEq6LUS2bycTg2WgmgwNrrogFPDfys-5A%40mail.gmail.com.

[jerem...@gmail.com]

I'm also interested in using Bazel for embedded software development.

Any update on the "dynamic configurations" or setting a --crosstool_top per binary? It would be nice to build for multiple targets and package all in bazel.

Thanks!

> add the toolchains (e.g. avr-gcc, gcc-arm-none-eabi) for these platforms to my bazel workspace, andbe able to specify which toolchains to use when building specific targets (e.g., "this cc_binary should be built for the AVR chip X using the avr-gcc toolchain, and this cc_binary should be built for ARM chip Y using the gcc-arm-none-eabi toolchain").

[Lukács T. Berki]

There haven't been significant changes. Greg is working on it, but it's not easy work :(

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c376c9be-d8e7-4311-a38c-a7123a7a832b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[Greg Estren]

Jeremy,

How, specifically, would you like to build multiple targets with different toolchains? For example, is it important that you could call:

$ bazel build //foo:all

and every target under //foo builds with its appropriate toolchain? Or would something like:

$ bazel build //foo:target1   # Automatically builds with target1's toolchain

$ bazel build //foo:target2   # Automatically builds with target2's toolchain

or even

$ bazel build //foo:all --crosstool_top=toolchain_x  # Automatically builds all target's under //foo that need toolchain x

Short story is that we have more of the internal pieces together to make these things possible. But some UIs are more possible than others because of ongoing technical limitations.  We can't do everything you might want right now, but we can do more than what's exposed right now. 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c376c9be-d8e7-4311-a38c-a7123a7a832b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Marc-Antoine Courteau]

I can't speak for Jeremy, but in my case, that first option (bazel build //foo:all) would be best.

More on my use case: I'd like to have one repo for all my embedded projects, e.g. some building with the ARM toolchain, and others building with the AVR toolchain. I might add other toolchains later. The plan is to also have some common code that I can use on all platforms, and also some tests for that code, which would run on the host.

I have some generic libraries that I want to use on all platforms, some platform-specific libraries (e.g. drivers for some hardware functionality on a specific family of chips), and executables to build for one specific chip. I would like to be able to build any executable without having to specify the crosstool to use on the command-line.

Does that make sense?

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c376c9be-d8e7-4311-a38c-a7123a7a832b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[Greg Estren]

Oh, I hadn't forgotten about you, Marc-Antoine. :) Glad to see you're still following this thread.

That absolutely makes sense, and is a perfectly reasonable request. I'm asking specifically about "all" vs. building a single executable at a time because that has impact on output path naming, which has efficiency and correctness problems if we're not careful. See here for details (both sections "Corrupted Outputs" and "Cacheable and Distributed Execution").

I think we can support building a single executable with the right toolchain before we'll be able to mix different executables with different toolchains together in the same build. So if even that, limited, scenario, is useful to you, it's something we can consider opening up soon. Especially as the work on platform definitions (which imply crosstool settings) moves along.

We could also conceivably open "//foo:all"-style builds sooner if there's a mutual understanding it could have performance consequences. For example, common platform-independent dependencies might get unnecessarily built twice for different executables.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c376c9be-d8e7-4311-a38c-a7123a7a832b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

--

Marc-Antoine Courteau, ing.
macou...@gmail.com

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAH_sd0WO-kwO3RX69SvAED_KFwivSTVQcGey7%3DYZicifUinB_A%40mail.gmail.com.

[jerem...@gmail.com]

Hi Greg,
Thanks for sharing the doc about Bazel platforms. It looks really interesting and is specifically helpful for embedded systems.

I think that only the "$ bazel build //foo:all" solution is helpful.

The other two examples: automatically selecting the toolchain based on the target or automatically selecting the targets based on the toolchain specified with --crosstool_top isn't really helpful, because it's easy enough to pass those arguments to bazel manually. I always call bazel through a script anyways because --cpu, --compilation_mode, --crosstool_top, --spawn_strategy, etc... so many options need to be included.

A common embedded platform includes an Arm-based system-on-a-chip and 1 or more low power microcontrollers like an AVR or MSP430. I would like to call something like

$ bazel build //:firmware_package

which would depend on rules to build //:arm_app with toolchain_arm and //:avr_app with toolchain_avr and combine them with a pkg_tar rule into a firmware package.

Thanks for your effort.

[Marc-Antoine Courteau]

On Mon, Apr 3, 2017 at 5:07 PM, Greg Estren <gre...@google.com> wrote:

Oh, I hadn't forgotten about you, Marc-Antoine. :) Glad to see you're still following this thread.

That absolutely makes sense, and is a perfectly reasonable request. I'm asking specifically about "all" vs. building a single executable at a time because that has impact on output path naming, which has efficiency and correctness problems if we're not careful. See here for details (both sections "Corrupted Outputs" and "Cacheable and Distributed Execution").

Right; in my case, building "all" is not super important, if I can build one executable without specifying which crosstool to use, which CPU and so on on the command line.

 

I think we can support building a single executable with the right toolchain before we'll be able to mix different executables with different toolchains together in the same build. So if even that, limited, scenario, is useful to you, it's something we can consider opening up soon. Especially as the work on platform definitions (which imply crosstool settings) moves along.

We could also conceivably open "//foo:all"-style builds sooner if there's a mutual understanding it could have performance consequences. For example, common platform-independent dependencies might get unnecessarily built twice for different executables.

Yeah, that makes sense. :)

Thanks!

 

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c376c9be-d8e7-4311-a38c-a7123a7a832b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

--

Marc-Antoine Courteau, ing.
macou...@gmail.com

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAH_sd0WO-kwO3RX69SvAED_KFwivSTVQcGey7%3DYZicifUinB_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[fl...@google.com]

Adding my voice here.

I have a sub-tree of my bazel repository that needs to be built with a different toolchain than the rest. It's ugly keeping track, for my many users, of which crosstools specification they need to use in different subtrees.

In the long term, parts of the tree need to be compiled to multiple target architectures, and tested together on the different systems.

Any ideas how I should address my short term problem?

Thanks,
Paul

[todd....@gmail.com]

I'm also interested in this.

Is there a way to create a macro or new rule to specify a compiler (or toolchain) without invoking bazel twice?

I'd like to have something like:

# use the gcc compiler
gcc_cc_binary(
name = "hello-world-arm",
srcs = "hello-world.c",
)

# use the icc compiler
icc_cc_binary(
name = "hello-world-x86",
srcs = "hello-world.c",
)

I'm okay with the duplication of the sources for now. I'd like to do one "bazel build :all" and it would build both.

I've got something similar working with a new CROSSTOOL but I have to run bazel build twice one with the --config=icc.

Are dynamic configurations needed for this?

Thanks,
-Todd

[Austin Schuh]

We wrote our own version of cc_{library,binary} in skylark to support this use case.  It took a good chunk of work and doesn't have all the features, but it lets us compile for 5 architectures and package the results up in a .deb.

Dynamic configurations should remove the need for our own rule though when they are ready.

Austin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8a9b1c60-51ad-4342-8760-eaf73c561528%40googlegroups.com.

[Zack Lalanne]

We solved this with writing our own cc_ skylark rules as well. Is there a preliminary doc discussing dynamic configurations?

-Zack

[Todd Foggoa]

Thanks Zack and Austin for the replies. I was also thinking about writing custom rules but thought I'd ask before jumping in.

Did you model your new rules off anything? I see the cc_library and cc_binary are implemented in java for bazel itself. I'm just wondering if there were better skylark examples to follow. I'll likely check out the skylark scala and go rules, to see how they work.

The doc I found on dynamic configurations is here:

  https://docs.google.com/document/d/1uoU8t7loTOu6uyzez-ilhcYgEXg4xjViJu0aqrf69TY/edit

I haven't seen an update on the topic in a month or two.

Thanks for the tips,

-Todd

[Zack Lalanne]

I modeled mine off of a set of rules for MSP430 (a 16-bit microcontroller).

https://github.com/sergiocampama/bazel-msp430