What is the recommended way to deploy a bazel build?

[Chris Brown]

What is the recommended best practice for shipping a project built with Bazel to customers? Obviously customers can't be expected to install Bazel and use `bazel run`. They need an installer that installs the package onto their system. My understanding is that internally, Google uses a separate packager, independent from Blaze, to package and deploy builds. How should I do this with Bazel? The best I can come up with is:

Write an "install" xx_binary target in bazel with data dependencies on all the targets that it wants to install. The installer's function is to copy the data dependencies to the desired final locations. Then, tar up the entire runfiles tree of the "install" target using the `--dereference` option to `tar` to make sure symlinks are dereferenced in the tarball. Instruct users to download the tarball and then run the "install" executable to install the package.

However, I'm not sure how the above will work when I want to ship and install multiple cc_binary targets from a single package. The runfiles tree of each target will have independent symlinks to any shared dependencies, so I'm pretty sure I will wind up with multiple copies of data files or dynamically linked libraries. For simple projects, the above approach seems to work ok, but I'm not confident it will do what I want when I have a large complex project with many data dependencies and dynamic linking. Is there a recommended approach?

[Austin Schuh]

Does https://docs.bazel.build/versions/master/be/pkg.html do what you want?  We use it to build packages to install.

Austin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/07b8f66d-8ca9-4c5d-8b88-16cb50087402%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alpha Lam]

If you just have one runfiles directory you can simply tar with --deference.

I believe you want to share a runfiles directory for multiple binary targets. In this case you can create a merged runfiles. Essentially a smaller execroot. For each {cc, java}_binary target there is a {name}.runfiles_manifest sitting next to it. I created a python script that reads and merges multiple of these files to generate a single runfiles directory which contains all dependencies for all targets of concern. Because of this you can even build with --nobuild_runfile_links to eliminate separate runfiles tree which is expensive for us (many tests). After that you can create symlinks to point to the copied binaries in the merged runfiles tree. This works for us for cc_binary and java_binary. py_binary is broken for this use case but I'm fixing.

Alpha

2017-11-14 1:02 GMT-05:00 Austin Schuh <austin...@gmail.com>:

Does https://docs.bazel.build/versions/master/be/pkg.html do what you want?  We use it to build packages to install.

Austin

On Mon, Nov 13, 2017 at 9:02 PM Chris Brown <chris...@zoox.com> wrote:

What is the recommended best practice for shipping a project built with Bazel to customers? Obviously customers can't be expected to install Bazel and use `bazel run`. They need an installer that installs the package onto their system. My understanding is that internally, Google uses a separate packager, independent from Blaze, to package and deploy builds. How should I do this with Bazel? The best I can come up with is:

Write an "install" xx_binary target in bazel with data dependencies on all the targets that it wants to install. The installer's function is to copy the data dependencies to the desired final locations. Then, tar up the entire runfiles tree of the "install" target using the `--dereference` option to `tar` to make sure symlinks are dereferenced in the tarball. Instruct users to download the tarball and then run the "install" executable to install the package.

However, I'm not sure how the above will work when I want to ship and install multiple cc_binary targets from a single package. The runfiles tree of each target will have independent symlinks to any shared dependencies, so I'm pretty sure I will wind up with multiple copies of data files or dynamically linked libraries. For simple projects, the above approach seems to work ok, but I'm not confident it will do what I want when I have a large complex project with many data dependencies and dynamic linking. Is there a recommended approach?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/07b8f66d-8ca9-4c5d-8b88-16cb50087402%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DGcrjkruoAWQKAd4NNp_NtZn8Grt4iS2gWiYXFi7oJk0Q%40mail.gmail.com.

[Alpha Lam]

There are a couple corner cases to get it right. The format of .runfiles_manifest is undocumented I believe. But I tested it to behave like this:

1. There is only 1 space for each line. That's the only field separator.

2. Each line describes the file in the runfiles tree ($1) and the corresponding real file ($2).

3. The second column can be empty. Create an empty file in this case.

4. Second column can be either a real file (copy), a symlink (dereference and copy) and a directory (copy the whole tree).

[Josh Powell]

Can you deliver a Docker image to your customers? If so https://github.com/bazelbuild/rules_docker might be of interest to you.

[Paul Johnston]

I think this is quite variable based on the type of software you're building, which is going to look very different for C/C++ vs Java vs NodeJs project.  

What's your current method look like?  Maybe there are best practices people can suggest based on your current setup and pain points vs some generic mechanism.

On Monday, November 13, 2017 at 10:02:47 PM UTC-7, Chris Brown wrote:

[Matthew Moore]

Generally within google we deal in what I call "mono-archives", which look like:

 - Java/Scala/Groovy => :foo_deploy.jar

 - C++/D/Rust/Go => mostly static binary

 - Python => PAR file (see github.com/google/subpar)

These are all single files, which are relatively easy to redistribute**.

You can loosely think of all of these as "statically linking" at the language level, so that all you need is the language runtime environment.

-M

** - However, not by traditional means because they bundle their deps unlike traditional redistribution systems like Maven Central / PyPI

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/d1661ebc-f61a-41eb-85d0-cb7de9dd609e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Matthew Moore

Container Development Uber-TL

Developer Infrastructure @ Google

[cns...@gmail.com]

Thanks Matthew for the suggestion! What's the best way to handle data dependencies (https://docs.bazel.build/versions/master/build-ref.html#data) in this case? Thanks!

On Wednesday, November 15, 2017 at 4:07:10 PM UTC-8 matt...@google.com wrote:

Generally within google we deal in what I call "mono-archives", which look like:

 - Java/Scala/Groovy => :foo_deploy.jar

 - C++/D/Rust/Go => mostly static binary

 - Python => PAR file (see github.com/google/subpar)

These are all single files, which are relatively easy to redistribute**.

You can loosely think of all of these as "statically linking" at the language level, so that all you need is the language runtime environment.

-M

** - However, not by traditional means because they bundle their deps unlike traditional redistribution systems like Maven Central / PyPI

On Wed, Nov 15, 2017 at 3:31 PM, Paul Johnston <pcj...@gmail.com> wrote:

I think this is quite variable based on the type of software you're building, which is going to look very different for C/C++ vs Java vs NodeJs project.  

What's your current method look like?  Maybe there are best practices people can suggest based on your current setup and pain points vs some generic mechanism.

On Monday, November 13, 2017 at 10:02:47 PM UTC-7, Chris Brown wrote:

What is the recommended best practice for shipping a project built with Bazel to customers? Obviously customers can't be expected to install Bazel and use `bazel run`. They need an installer that installs the package onto their system. My understanding is that internally, Google uses a separate packager, independent from Blaze, to package and deploy builds. How should I do this with Bazel? The best I can come up with is:

Write an "install" xx_binary target in bazel with data dependencies on all the targets that it wants to install. The installer's function is to copy the data dependencies to the desired final locations. Then, tar up the entire runfiles tree of the "install" target using the `--dereference` option to `tar` to make sure symlinks are dereferenced in the tarball. Instruct users to download the tarball and then run the "install" executable to install the package.

However, I'm not sure how the above will work when I want to ship and install multiple cc_binary targets from a single package. The runfiles tree of each target will have independent symlinks to any shared dependencies, so I'm pretty sure I will wind up with multiple copies of data files or dynamically linked libraries. For simple projects, the above approach seems to work ok, but I'm not confident it will do what I want when I have a large complex project with many data dependencies and dynamic linking. Is there a recommended approach?

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.