"not all outputs were created": running a tool that reaches outside of Bazel

[Gregg Reynolds]

I'm writing Bazel rules for OCaml and I've run into a problem that I think is due to a tool that wants to work outside of Bazel's control.  In short, tool `ocamlfind` runs the compiler `ocamlopt`, which runs clang.

Now this works fine, except when it doesn't. I "think* the problem is traceable to the tools reaching outside of Bazel. I'm looking for confirmation and suggested remedies.

This is a little complicated, because OCaml compilation is, let's just say, a little eccentric.  A module has two parts, an interface and an implementation. So a compiled  module Foo would have foo.cmi (Compiled Module Interface) and foo.cmx (the implementation file).  You can compile the cmi first, then the cmx, or you can pass them both to the compiler at once.  The rules are a little odd: the compiler looks for an mli, and it if finds it, looks for the cmi. If it does not find it, it generates the cmi from the ml file.

In any case, I have a rule to compile mli files and a rule to compile ml files; the latter has an (optional) attribute for a cmi dependency.  This works fine:

```

INFO: From ocaml_module(RefList), compiling impl :
+ clang -arch x86_64 -Wno-trigraphs -c -o 'bazel-out/darwin-fastbuild/bin/src/refList.o' '/var/folders/wz/dx0cgvqx5qn802qmc3d4hcfr0000gp/T/camlasmca0598.s'
Effective set of compiler predicates: pkg_bytes,autolink,native
+ ocamlopt.opt -verbose -g -bin-annot -c -I bazel-out/darwin-fastbuild/bin/src -I src -I bazel-out/darwin-fastbuild/bin/src/tmp -o bazel-out/darwin-fastbuild/bin/src/refList.cmx -I /Users/gar/.opam/4.07.1/lib/bytes src/refList.ml
Target //src:RefList up-to-date:
  bazel-bin/src/refList.cmx
  bazel-bin/src/refList.o
  bazel-bin/src/refList.cmi
  bazel-bin/src/refList.cmt

```

(Ignore the *.cmt file)

Note the clang output and input locations.

Now I've enhanced the rule to take an mli (source) file OR a cmi file. Passing both source files (.mli, .ml) works if I construct a compile command by hand on the CLI.  The rule generates a command that looks correct. But when I run it, the .cmi file is produced but I get an error:

```

INFO: From ocaml_module(RefList), compiling impl :
+ clang -arch x86_64 -Wno-trigraphs -c -o 'src/refList.o' '/var/folders/wz/dx0cgvqx5qn802qmc3d4hcfr0000gp/T/camlasm35a097.s'
Effective set of compiler predicates: pkg_bytes,autolink,native
+ ocamlopt.opt -verbose -g -bin-annot -c -I bazel-out/darwin-fastbuild/bin/src -I src -I bazel-out/darwin-fastbuild/bin/src/tmp -o bazel-out/darwin-fastbuild/bin/src/refList.cmx -I /Users/gar/.opam/4.07.1/lib/bytes src/refList.mli src/refList.ml
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.cmt' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.cmx' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.o' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: not all outputs were created or valid

```

Note the output of the clang command - it wants to write its output to src rather than one of the bazel-* dirs.  I believe that is the problem - am I right?

(Related: if that  is indeed the problem, should not Bazel recognize it and issue a meaningful error message?)

I have encountered this previously in a different context, and the solution I came up with was to copy the source file to a temp dir created by the Bazel rule and run the compile action from there.  Then clang writes its output there, which works.

Is there a better way to handle this?

Thanks,

Gregg

[Herrmann, Andreas]

Hi Gregg,

I'm not terribly familiar with the OCaml compiler and it's hard to diagnose such issues without looking at the rule definitions and an example. Nonetheless

On Sun, Oct 4, 2020 at 5:49 PM Gregg Reynolds <d...@mobileink.com> wrote:

```

INFO: From ocaml_module(RefList), compiling impl :
+ clang -arch x86_64 -Wno-trigraphs -c -o 'src/refList.o' '/var/folders/wz/dx0cgvqx5qn802qmc3d4hcfr0000gp/T/camlasm35a097.s'
Effective set of compiler predicates: pkg_bytes,autolink,native
+ ocamlopt.opt -verbose -g -bin-annot -c -I bazel-out/darwin-fastbuild/bin/src -I src -I bazel-out/darwin-fastbuild/bin/src/tmp -o bazel-out/darwin-fastbuild/bin/src/refList.cmx -I /Users/gar/.opam/4.07.1/lib/bytes src/refList.mli src/refList.ml
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.cmt' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.cmx' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.o' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: not all outputs were created or valid

```

Note the output of the clang command - it wants to write its output to src rather than one of the bazel-* dirs.  I believe that is the problem - am I right?

You're referring to `clang ... -o 'src/refList.o' ..`, correct? Indeed, that seems to be an issue. However, IIUC it doesn't explain the missing `.cmx` output. What changed compared to the other clang command earlier in your email? In that case clang was given a `-o` flag with the correct `bazel-out/...` prefix.

 

(Related: if that  is indeed the problem, should not Bazel recognize it and issue a meaningful error message?)

The error message that Bazel prints out states that these declared outputs were not created, in this case meaning they were not created in the expected location.

 

I have encountered this previously in a different context, and the solution I came up with was to copy the source file to a temp dir created by the Bazel rule and run the compile action from there.  Then clang writes its output there, which works.

That is one way to work around such issues. A difficulty is that this approach can invalidate things like relative include paths for files from dependencies, e.g. `#include "./foo/bar"`, unless enough of the source tree is reconstructed in the temp dir. Another approach is to add a step to the action that moves output files to the correct location. Either by defining the action using `ctx.actions.run_shell` or by pointing `ctx.actions.run` to a script that performs these steps in the end. So long as build actions are sandboxed, intermediate outputs in the source tree (meaning outside of `bazel-out/...`) should not cause issues. However, if you run such build actions without sandboxing, e.g. on Windows, then you can get conflicts between parallel build actions that write to the same files in the source tree. Ideally, the compiler can be configured to read sources from the source tree and write outputs to the expected output paths right away.

Btw, you can pass the flags `-s --sandbox_debug` to `bazel build` to get a little more information from Bazel about the actions being performed. `-s` will print the commands that Bazel invokes, and `--sandbox_debug` will print out details about the sandbox including the sandbox working directory and will preserve the sandbox directory so that you can inspect the outputs after the build action runs.

Best, Andreas

[Gregg Reynolds]

On Mon, Oct 5, 2020 at 6:46 AM Herrmann, Andreas <andreas....@tweag.io> wrote:

Hi Gregg,

I'm not terribly familiar with the OCaml compiler and it's hard to diagnose such issues without looking at the rule definitions and an example. Nonetheless

On Sun, Oct 4, 2020 at 5:49 PM Gregg Reynolds <d...@mobileink.com> wrote:

```

INFO: From ocaml_module(RefList), compiling impl :
+ clang -arch x86_64 -Wno-trigraphs -c -o 'src/refList.o' '/var/folders/wz/dx0cgvqx5qn802qmc3d4hcfr0000gp/T/camlasm35a097.s'
Effective set of compiler predicates: pkg_bytes,autolink,native
+ ocamlopt.opt -verbose -g -bin-annot -c -I bazel-out/darwin-fastbuild/bin/src -I src -I bazel-out/darwin-fastbuild/bin/src/tmp -o bazel-out/darwin-fastbuild/bin/src/refList.cmx -I /Users/gar/.opam/4.07.1/lib/bytes src/refList.mli src/refList.ml
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.cmt' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.cmx' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: output 'src/refList.o' was not created
ERROR: /Users/gar/mina/ocaml-extlib/src/BUILD.bazel:596:13: not all outputs were created or valid

```

Note the output of the clang command - it wants to write its output to src rather than one of the bazel-* dirs.  I believe that is the problem - am I right?

You're referring to `clang ... -o 'src/refList.o' ..`, correct? Indeed, that seems to be an issue. However, IIUC it doesn't explain the missing `.cmx` output. What changed compared to the other clang command earlier in your email? In that case clang was given a `-o` flag with the correct `bazel-out/...` prefix.

The difference is that the one that worked had only refList.ml as input, and a dependency on another rule that compiled refList.mli separately to produce refList.cmi. So it only had to compile refList.ml.  The rule that did not work has both source files as input, so it must compile them both, first the interface file (refList.mli) then the implementation file (refList.ml) - this is how the OCaml compiler works, it checks the implementation against the interface.  In this case, the interface file gets compiled, and then, as the error message shows, the compile of the implementation file fails.  BTW, when it compiles the implementation file it outputs both a *.cmx file and a *.o file.  The *.cmt file contains info that tools use, it's an optional output controlled by a compiler flag.

So the problem is essentially that the OCaml compiler actually acts as a kind of compiler driver.  The clang command is generated by the OCaml compiler, not by my build rule.  The compiler has some flags that can be used to pass arguments to "the linker" and so forth, but its all pretty opaque, so I think it's best to treat the clang stuff as a compiler internal, rather than trying to control it from my Bazel rule. If there was a clear and reliable way to tell the compiler "tell clang to use this directory as its work dir" that would be great, but I'm not aware of such a method. Maybe there is one, though; I'll ask on the OCaml list.

 

 

(Related: if that  is indeed the problem, should not Bazel recognize it and issue a meaningful error message?)

The error message that Bazel prints out states that these declared outputs were not created, in this case meaning they were not created in the expected location.

It's not a very useful error message, IMO. It doesn't tell me what happened, only what did not happen, not very different from "Error: refList.ml failed to compile".  It says the *output* was not created, but that has special meaning with Bazel; it does not say that the compile failed. And "not all outputs were created or valid" - doesn't Bazel know which? 

 

I have encountered this previously in a different context, and the solution I came up with was to copy the source file to a temp dir created by the Bazel rule and run the compile action from there.  Then clang writes its output there, which works.

That is one way to work around such issues. A difficulty is that this approach can invalidate things like relative include paths for files from dependencies, e.g. `#include "./foo/bar"`, unless enough of the source tree is reconstructed in the temp dir. Another approach is to add a step to the action that moves output files to the correct location. Either by defining the action using `ctx.actions.run_shell` or by pointing `ctx.actions.run` to a script that performs these steps in the end. So long as build actions are sandboxed, intermediate outputs in the source tree (meaning outside of `bazel-out/...`) should not cause issues. However, if you run such build actions without sandboxing, e.g. on Windows, then you can get conflicts between parallel build actions that write to the same files in the source tree. Ideally, the compiler can be configured to read sources from the source tree and write outputs to the expected output paths right away.

Thanks for the info.  I'm not currently supporting Windows but that's good to know.

 

Btw, you can pass the flags `-s --sandbox_debug` to `bazel build` to get a little more information from Bazel about the actions being performed. `-s` will print the commands that Bazel invokes, and `--sandbox_debug` will print out details about the sandbox including the sandbox working directory and will preserve the sandbox directory so that you can inspect the outputs after the build action runs.

My user.bazelrc contains both, always. ;)

Thanks

Gregg 

[Clément Hurlin]

Hi Gregg, that's nice you're authoring Ocaml rules :-) Are they related to https://github.com/jin/rules_ocaml? Are you extending them?

Cheers,

Clément

[Gregg Reynolds]

On Thu, Oct 8, 2020 at 10:24 AM Clément Hurlin <clement...@tweag.io> wrote:

Hi Gregg, that's nice you're authoring Ocaml rules :-) Are they related to https://github.com/jin/rules_ocaml? Are you extending them?

Hi Clément,

I started with jin's rules. They turned out to be a good place for a beginning rules author to start - I learned some key things from them - but they are far too elementary. My original goal was to add Bazel support to the Mina (formerly: Coda) project; I was hoping a quick-and-dirty conversion would do, and estimated about a month.  That was four months ago, hahaha!  It became clear by degrees that I would need to implement a first-class rules_ocaml package that covers all the use cases Dune covers, as well as a tool to automate conversion of Dune files to BUILD.bazel files., etc. etc. Turned out to be a good deal more complicated than I anticipated.  The OCaml toolchain is, I gather, somewhat notorious for its opacity and complexity.  I can testify to that! So between that and learning the ins and outs of Bazel, here we are four months down the road. I expect to make a (pre-) Beta version of my rules available for public scrutiny sometime this month.

(In the meantime I encourage you to take a look at Mina, it's a fascinating project, and there are still 100s of places available in their Genesis program.  If you sign up please tell them gar#6872 (my handle on Mina's discord server) sent you so I get some brownie points.)

Gregg

[Clément Hurlin]

Hi Gregg,

Thanks for the explanations, very much appreciated. I would gladly give a try to your rules once you release them, as I could have use of them in the context of the Tezos blockchain. Keep going :-)

Cheers,
Clément

ps: I had a look at Mina before, it looks like a nice project indeed.