Bazel configuration binary output locations

[Charles Nicholson]

I have an extremely simple cross-compilation toolchain building arm cortex-m binaries in Bazel.

I build them by invoking bazel with my own crosstool_top and Bazel's host_crosstool_top:

bazel build --crosstool_top=//build/toolchain/host-linux:toolchain

            --cpu=armv7e-m

            --host_crosstool_top=@bazel_tools//tools/cpp:toolchain

            :hello

This works, and I get my ARM AAPCS elf file output to bazel-bin/hello.

If I then build "hello" for the host architecture (Linux-x64, "k8" to Bazel) using Bazel's built-in host toolchain:

bazel build :hello

This works too, and I get my x64 elf output to bazel-bin/hello.

Unfortunately, this overwrites the earlier ARM elf file. When I alternate back and forth between the two toolchains, I don't see an actual recompilation happen, so I assume Bazel is just fetching the latest up-to-date version from the cache.

I don't expect to have identically-named executables, but for static libraries this isn't uncommon. Either way, this seems subtle.

Similarly, I see that when I build for my local host architecture via:

bazel build :hello -c dbg

bazel build :hello -c opt

The "dbg" configuration "hello" elf just gets overwritten with the "opt" version.

I want to have one top-level "b" script that builds everything it can on the current host, probably host C++ tools, C99/ASM firmware, some python scripts, and organize them somewhere.

In previous projects, I've set CMake up directly so that the output tree looks more like:

bin/dbg/hello-x64

bin/dbg/hello-armv7e-m

bin/opt/hello-x64

bin/opt/hello-armv7e-m

Is any of this kind of configuration possible with Bazel? I guess I could write extra tooling that renames the bazel-bin results and copies them into my own 'bin', but now that's outside of Bazel's dependency graph so I can't ask it where my binaries came from etc, Bazel can't clean them, etc. 

If there's a way that Bazel can be configured to do this, I'd prefer that.

Thanks,

Charles

[Damien Martin-Guillerez]

Hello,

I think you are mistaking the convenience symlink with the actual output directory. If you actually look at the convenience symlink:

$ readlink bazel-out

~dmarting/.cache/bazel/_bazel_dmarting/4c68128ea7d752c9535d956d14249eea/execroot/io_bazel/bazel-out

$ readlink bazel-bin

~dmarting/.cache/bazel/_bazel_dmarting/4c68128ea7d752c9535d956d14249eea/execroot/io_bazel/bazel-out/local-fastbuild/bin

Then bazel-bin is under a "local-fastbuild/bin" directory under bazel-out. If you look at bazel-out itself you will find the configuration speicific directories:

 $ ls bazel-out

host  local-fastbuild  stable-status.txt  _tmp  volatile-status.txt

The -c opt will create a local-opt and you can also see the host configuration there.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACfYaaWx3NXHjBKEpmfL%3DBLequAiwH_7TJCNB_QWut%3DEOxFO%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Charles Nicholson]

Thanks for the response, that seems great.

Is there a way I can get a manifest of outputs from Bazel? 

If possible I'd like to interrogate Bazel for a list of all of my final executable outputs, but none of the object files, dependency files, unit test executables, or other "intermediate".

Otherwise, if I want to copy them out into an artifact bundle, I need to keep an explicit list of artifacts, which needs to be synchronized with the Bazel targets. That's not the end of the world, but it's outside of Bazel's knowledge.

Charles

On Wed, Jul 26, 2017 at 2:54 AM, Damien Martin-Guillerez <dmar...@google.com> wrote:

Hello,

I think you are mistaking the convenience symlink with the actual output directory. If you actually look at the convenience symlink:

$ readlink bazel-out

~dmarting/.cache/bazel/_bazel_dmarting/4c68128ea7d752c9535d956d14249eea/execroot/io_bazel/bazel-out

$ readlink bazel-bin

~dmarting/.cache/bazel/_bazel_dmarting/4c68128ea7d752c9535d956d14249eea/execroot/io_bazel/bazel-out/local-fastbuild/bin

Then bazel-bin is under a "local-fastbuild/bin" directory under bazel-out. If you look at bazel-out itself you will find the configuration speicific directories:

 $ ls bazel-out

host  local-fastbuild  stable-status.txt  _tmp  volatile-status.txt

The -c opt will create a local-opt and you can also see the host configuration there.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

[Robert Tsai (Verb Ext.)]

"final executable" is kind of subjective. What if your system image contains a shell script that invokes other C programs?

Some options:

• Write some kind of zip/tarball genrule to bundle your desired artifacts. (I guess that's an explicit list.)
• Use "bazel query" to find all your cc_binary rules (maybe too broad, if you inevitably write your own internal tools)
• Use the "tags" attribute with cc_binary, etc., you can tag the special ones with a special tag. But that's effectively an explicit list, too.

--Rob

On Wed, Jul 26, 2017 at 11:08 AM, 'Charles Nicholson' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Thanks for the response, that seems great.

Is there a way I can get a manifest of outputs from Bazel? 

If possible I'd like to interrogate Bazel for a list of all of my final executable outputs, but none of the object files, dependency files, unit test executables, or other "intermediate".

Otherwise, if I want to copy them out into an artifact bundle, I need to keep an explicit list of artifacts, which needs to be synchronized with the Bazel targets. That's not the end of the world, but it's outside of Bazel's knowledge.

Charles

On Wed, Jul 26, 2017 at 2:54 AM, Damien Martin-Guillerez <dmar...@google.com> wrote:

Hello,

I think you are mistaking the convenience symlink with the actual output directory. If you actually look at the convenience symlink:

$ readlink bazel-out

~dmarting/.cache/bazel/_bazel_dmarting/4c68128ea7d752c9535d956d14249eea/execroot/io_bazel/bazel-out

$ readlink bazel-bin

~dmarting/.cache/bazel/_bazel_dmarting/4c68128ea7d752c9535d956d14249eea/execroot/io_bazel/bazel-out/local-fastbuild/bin

Then bazel-bin is under a "local-fastbuild/bin" directory under bazel-out. If you look at bazel-out itself you will find the configuration speicific directories:

 $ ls bazel-out

host  local-fastbuild  stable-status.txt  _tmp  volatile-status.txt

The -c opt will create a local-opt and you can also see the host configuration there.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACfYaaXjh4QXPy1cJH68BSx9_97-F6rPPgB05EjXNDU8njaM%3DQ%40mail.gmail.com.

[Charles Nicholson]

Agreed re: subjectivity and that's not something I'd want bazel to solve. Sorry if I mis-expressed myself.

I'm happy to maintain a list of targets (e.g. "//tools:foo", "//app:firmware", ...), I was just wondering if there's a way to go from that list plus a bazel invocation (-c, --crosstool_top, etc) to a final list of files.

Bazel knows this information, since it puts the files there.

On Wed, Jul 26, 2017 at 11:18 AM, Robert Tsai (Verb Ext.) <rober...@verbsurgical.com> wrote:

"final executable" is kind of subjective. What if your system image contains a shell script that invokes other C programs?

Some options:

• Write some kind of zip/tarball genrule to bundle your desired artifacts. (I guess that's an explicit list.)
• Use "bazel query" to find all your cc_binary rules (maybe too broad, if you inevitably write your own internal tools)
• Use the "tags" attribute with cc_binary, etc., you can tag the special ones with a special tag. But that's effectively an explicit list, too.

--Rob

[Charles Nicholson]

Sorry, I didn't finish my thought:

Thanks for the responses, tags plus a zip/tarball genrule seems like a good combination here! :)

On Wed, Jul 26, 2017 at 11:24 AM, Charles Nicholson <nicho...@google.com> wrote:

Agreed re: subjectivity and that's not something I'd want bazel to solve. Sorry if I mis-expressed myself.

I'm happy to maintain a list of targets (e.g. "//tools:foo", "//app:firmware", ...), I was just wondering if there's a way to go from that list plus a bazel invocation (-c, --crosstool_top, etc) to a final list of files.

Bazel knows this information, since it puts the files there.

On Wed, Jul 26, 2017 at 11:18 AM, Robert Tsai (Verb Ext.) <rober...@verbsurgical.com> wrote:

"final executable" is kind of subjective. What if your system image contains a shell script that invokes other C programs?

Some options:

• Write some kind of zip/tarball genrule to bundle your desired artifacts. (I guess that's an explicit list.)
• Use "bazel query" to find all your cc_binary rules (maybe too broad, if you inevitably write your own internal tools)
• Use the "tags" attribute with cc_binary, etc., you can tag the special ones with a special tag. But that's effectively an explicit list, too.

--Rob

[Austin Schuh]

For one of our targets, we are generating a "deploy" shell script (name + '_deploy') that has a runfiles directory with everything in it (zip/tarball would also work) and will deploy the executables when run.  We are using this to deploy code to a microcontroller.  This is an implicit output of our linking rule.  I've also used this method to rsync a list of executables to a linux box when the target is run.

Austin

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACfYaaWx3NXHjBKEpmfL%3DBLequAiwH_7TJCNB_QWut%3DEOxFO%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACfYaaXjh4QXPy1cJH68BSx9_97-F6rPPgB05EjXNDU8njaM%3DQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CACfYaaWm79DUhO4xv%3D3ibc1VcfSb3aFExyF77g-xpj%3Dhn0_D3A%40mail.gmail.com.