How could I deploy a build rule which runs a docker image?

[Filip Filmar]

Hi folks.

I have this piece of software that I'd like to use in a bazel build rule, which I think may be far too complex to try and make a hermetic version of.  That option is not off the table yet, but at least in principle what I'd want to attempt instead seems like it may be faster to do.

To avoid having to figure out how to have bazel pull it in as an external repository. It's a Python program that pulls in a gigabyte of dependencies, and a clean installation on any given system is a headache as is. I thought, I might avoid repeated installation pains if I managed to package it in a container, then have a build rule executing the said program inside an appropriately configured container.

I tried to do the above, and got to a point where I can invoke the build rule; but the build rule fails to run because all the files that Bazel makes available to it are symlinks to paths which are not mounted inside of the container. I didn't see an easy way for me to know what paths these would be ahead of time so that possibly I could mount them into the container as it starts.

This left me puzzled a bit, and at this point I figured better ask: is this even a viable option?  If not, what alternatives do I have?

I am aware that it is possible to run bazel completely inside a container.  Though, the utility of that approach is under question since all bazel-* symlinks are invalid once container exits, which could be a drag on productivity.

I am also aware that it is possible to have bazel run docker sandboxes.  Which, I suppose, could allow me to extend the docker image to contain my binaries and then invoke my program non-hermetically inside a container. IIUC my use case is not the intended use case, but might just work.

Advice?  Anything I missed?

Thanks,

F

[Luke Gehorsam]

Hi Filip, 

Are you looking to build the entire program in your container via some `docker container run` command on your host, or are you looking to have a Bazel installation on your host machine that executes some build steps in a container?

If the former, I think you could build to a predefined output directory inside the container by passing in --output_user_root=/some/predefined/path and then mounting to that directory.

If the latter, I could go into further detail there.

- Luke

[Filip Filmar]

On Thu, May 14, 2020 at 6:26 PM Luke Gehorsam <lugeh...@gmail.com> wrote:

Are you looking to build the entire program in your container via some `docker container run` command on your host, or are you looking to have a Bazel installation on your host machine that executes some build steps in a container?

I would like to build on host, with some build steps done in a container (invoking a binary that's in the container itself).

F

[Luke Gehorsam]

Take a look at `container_run_and_extract` -- it may be what you need https://github.com/bazelbuild/rules_docker/blob/master/docker/util/README.md  

However, I've found that the rule does not cache certain steps - specifically it runs `docker load` on your image from a `.tar` even if it already exists in your local registry. It seems like in the current active PRs there is work being done to improve that. If not I will add it once some of the dust settles. Also, you will need to migrate your Dockerfile to either a `container_image` rule or a `container_run_and_commit` rule (see repo-wide README) in order for Bazel to integrate your image with its build system.

[Filip Filmar]

Thanks for the advice.  I'll try it out.

F

[Filip Filmar]

Hi Luke, others.

It took me a while to get to trying this out again.  I got to a certain place, but I'm not sure how to proceed from there.  I managed to use the instructions in rules_docker (and in general looking at the WORKSPACE file and some trial and error) to build my own container image in Bazel with the needed necessary.  Here's what I get in the end.

So now I have a bazel-built docker image with the tools that I need.  Cool.  However, I now still have the same issue as in my original question: to do an equivalent of `docker run`, with exactly the right directories mounted as volumes so that the source code that I want to access is visible within the container.  I think I somehow need to mount in the workspace root and output user root (so that the symlinks in the output directory work).  Are there variables that allow me to access those? Am I missing something?

Best, 

F

[Luke Gehorsam]

There may be better suggestions here but the best I could do for myself was actually clone my own repository at a specific hash (for reproducibility) inside the `run_and_extract`.

[Elliot Murphy]

Hi Luke, Filip!

This discussion has been super helpful for me. I've been trying out `container_run_and_extract` as well. 

What I did was make a tarball that contained the source files I wanted to process inside the docker image, and added the tar to my `container_image` rule. You can see a working example here: https://github.com/kindlyops/talks/blob/ad1a5a56a82ddb17cd9041ce0f40aa8317536f36/bazelpresenter/BUILD.bazel#L9

My experiments with volume mounts have been more complicated, the tar approach felt simpler for me.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CANREG33iUnAofBznDg1sYdsm7TofyWSg4%2BqEyT7t7Nuv1czYzw%40mail.gmail.com.

--

Elliot Murphy | Kindly Ops is the DevOps accelerator for regulated startups

Secure Infra as Code, Security Assessments, Pentests, AWS Security & Compliance Automation

Get our best analysis and insights each week https://www.kindlyops.com/newsletter/

[Luke Gehorsam]

Glad this has been helpful! Along these lines, if your source code that is required to make the build from within the container is not going to change much, you could also bake the code into the image by doing a git clone inside a container_run_and_commit rule.

[Filip Filmar]

Hi folks.  Thank you for the advice. I tried the approach out, and you can see the result here.  So far results have been interesting.  I.e. what y'all advised is doable.  Sadly I am not convinced of the practicality of the approach:

1. It takes 40 seconds to merge the "main" image layer with a layer containing the source files.
2. The in-container run step is completely opaque: if there's an error somewhere in the multi-step build process, the only thing you have as result is an error code and lack of the expected output file.  Here I'd expect the "regular" compiler error output with file and line number and such.

I think (2) can be solved, if I take care to redirect the command output into files, and pull them out of the container. But even if it were, there's still (1).  With multiple files to process, the compilation time is going to be likely impractical for processing source files.  I haven't given up yet, but am not convinced that this particular approach will work for me. 

Back to the drawing board,

F

[Filip Filmar]

On Mon, May 25, 2020 at 5:25 PM Filip Filmar <fil...@gmail.com> wrote:

Back to the drawing board,

FYI, gone back to the original `docker run` approach, and changed the angle a bit, and I think I have it.  Details in the pull request, sorry it's not very tidy. 

Earlier, the main stumbling blocks were that the source directory and the build root directory were not available to starlark. Turns out, if you delegate the invocation of `docker run` to a shell script, the shell script is able to figure the missing information out.  You can get at the source directory by following `readlink -m ${INPUT}` as far as it will go, and taking `dirname`.  The build root can be obtained by cutting the full path of the input file before the string "/_bazel_". I'll probably burn in bazel hell for this, but I suspect the naming scheme won't change soon.

Once you have the above info, you can mount the source dir and build root into the container at host-identical paths, which will make all the copious bazel symlinks valid in the container too. From there it is easy to run the command line.

My first example builds from clean in 2.5s.  This is much better than ~40s I saw with the container approach.

Of course, the approach is not cross-platform and *may* interact badly with build caches and whatnot, but that bridge can be crossed in due time.

Cheers, and thank you for the help and pointers,
F

[Filip Filmar]

On Mon, May 25, 2020 at 10:51 PM Filip Filmar <fil...@gmail.com> wrote:

On Mon, May 25, 2020 at 5:25 PM Filip Filmar <fil...@gmail.com> wrote:

Back to the drawing board,

FYI, gone back to the original `docker run` approach, and changed the angle a bit, and I think I have it.  Details in the pull request, sorry it's not very tidy. 

To give this thread some closure, and in case someone else is interested in running bazel build actions using docker run in the future, here's a proof of concept that seems to work locally:

https://github.com/filmil/ebook-example/tree/b8f4f8251f84132f6bb11b6e4133b63f2b259ff0

The rule `asymptote` is run inside a custom container via docker run.  Example command line:

bazel build //:test

This will build some png files.  You need docker, and you need to be OK with downloading and running a container I keep on docker hub.  

HTH,

F