Handling dynamic dependencies with Bazel. What is the status? What are today's workaround?

[Marc Delorme]

I am currently working on a game engine in my company, and I am investigating different build system in order to escape our current "batch script + MSBuild" situation which is a nightmare to maintain.

The engine build is quite complex, there are a lot of tools, multiple target platforms, code generators, and asset conventer

to generate runfiles. (you can have more details on my previous post). The build include some custom tools which all have to deal with dynamic dependencies (like header files in C++).

In my case, the issue can be split in three situations:

Situation 1: Code generators
code_generator.exe --output_header=my_service.h --output=my_service.cpp my_service.idl
In that situation my_service.idl could reference other .idl file defining some data type. Meaning that my_service.h and my_service.cpp can dynamically depends on other .idl file.

Situation 2: Shader Compiler
ShaderCompiler.exe --target=Win64 --include_dir=. --include_dir=../shader_include --output=my_shader.bin my_service.glsl
The situation is very similar to C++, my_service.glsl could depends on header files available in given include directory.

Situation 3: Asset Compiler
TextureCompiler.exe --target=Win64 -o my_texture.bin my_texture.tex
In that situation my_texture.tex is a file describe the texture properties, inside it references an image file (e.g my_texture.png), and it could reference other preset files.

I have read some years ago it was not yet possible to deal with dynamic depnedencies in Starlark, and I have not read anything new about the issue since that time. Is there anything new about it. Is it on the long term roadmap?

In the meantime, what are the workaround possible for the different scenario above?

For the Situation 1, have been adviced to add all .idl files as dependencies of any .idl target (Solution A). I think it is fine in that case because code generator is a very fast build step. Regenerating everything everytime I do a change is accetable. But it would not be acceptable in the Situation 2. Indeed compiling shader is much slower. If by changing one header file it trigger the rebuild of all the shaders, that would be very painful. And regarding Situation 3, that solution would be impossible because we cannot predict in advance which image file
could be used. Would it be .png? .bmp? .tga? We kind of name? A lot a different folder possible, etc.

Another solution would be for those build step to make bazel action run another build system behind the scene, anything which could skip running the real action if not necessary (Solution B). Example FileTracker of MSBuild's .vcxproj or calling ninja managing depfile, ... But those action would not be hermetic, by the would read of write dependency file unmanaged by Bazel. So I guess it would not work with Bazel? Or would be dangerous/error prone?

Otherwise I could generate code before the bazel build and compile shader and asset after the bazel build (Solution C).
It is not very satisfying because I would be back to today's situation where my build system would be some custom script
stiching other build system together. It feels I would lose the major benefit of Bazel.

Conclusion,

1. I wish the Bazel Community can provide some tips about how I can handle dynamic dependencies today. :)
   
2. Also I have a meta-question about the dynamic dependencies situation. I guess if the issue is not handle by Bazel today, its because it does not appear as an important feature for the majority of user. It is very surprising to me, because from my point of view dynamic dependencies is every where regarding the concept of build system (C/C++, shader compilers, protobuf, other code generator, data converter, etc.). Am I biased because it is actually just a problem for the game industry? What is your opinion?

[lcid...@gmail.com]

See comments further down.

On Thursday, January 2, 2020 at 9:44:24 AM UTC+1, Marc Delorme wrote:

I am currently working on a game engine in my company, and I am investigating different build system in order to escape our current "batch script + MSBuild" situation which is a nightmare to maintain.

3 years ago I was in the same situation and managed to convert a multi-million 3D asset pipeline to Bazel. Back then Bazel was a bit less mature but it nevertheless was a big success.

 

The engine build is quite complex, there are a lot of tools, multiple target platforms, code generators, and asset conventer

to generate runfiles. (you can have more details on my previous post). The build include some custom tools which all have to deal with dynamic dependencies (like header files in C++).

Bazel basically operates on a DAG. So any complexity you have needs to be mapped to this DAG. Platforms can be mapped by now quite ok, although AFAIK documentation is rather lackluster around these parts last I saw.

Your biggest problems might be to split up (you should aim for that) and/or clean up your current tools. From experience I would advise not to take any shortcuts (e.g. disabling Sandboxing). Resist the urge to fight the basic concepts of Bazel.

 

In my case, the issue can be split in three situations:

Situation 1: Code generators
code_generator.exe --output_header=my_service.h --output=my_service.cpp my_service.idl
In that situation my_service.idl could reference other .idl file defining some data type. Meaning that my_service.h and my_service.cpp can dynamically depends on other .idl file.

Situation 2: Shader Compiler
ShaderCompiler.exe --target=Win64 --include_dir=. --include_dir=../shader_include --output=my_shader.bin my_service.glsl
The situation is very similar to C++, my_service.glsl could depends on header files available in given include directory.

Situation 3: Asset Compiler
TextureCompiler.exe --target=Win64 -o my_texture.bin my_texture.tex
In that situation my_texture.tex is a file describe the texture properties, inside it references an image file (e.g my_texture.png), and it could reference other preset files.

In the end it all boils down to:

1. Find out what the exact input for each run is

2. Find out what the exact output for each run is

3. Code that into a rule

(4. Find tune using Aspects)

5. Repeat

 

I have read some years ago it was not yet possible to deal with dynamic depnedencies in Starlark, and I have not read anything new about the issue since that time. Is there anything new about it. Is it on the long term roadmap?

In the meantime, what are the workaround possible for the different scenario above?

You can produce directories from rules but AFAIK this is a last resort type of thing. Directories as dependencies lead very often to bad Bazel cache utilization. 

And I cannot emphasize it enough - try not to fight Bazels concepts. If you want (very few scenarios actually command them) "dynamic dependencies" I would take a look at CMake or SCons.

 

For the Situation 1, have been adviced to add all .idl files as dependencies of any .idl target (Solution A). I think it is fine in that case because code generator is a very fast build step. Regenerating everything everytime I do a change is accetable. But it would not be acceptable in the Situation 2. Indeed compiling shader is much slower. If by changing one header file it trigger the rebuild of all the shaders, that would be very painful. And regarding Situation 3, that solution would be impossible because we cannot predict in advance which image file
could be used. Would it be .png? .bmp? .tga? We kind of name? A lot a different folder possible, etc.

What we did back then is split Shader compilation into various stages to maximize caching.

Basically try to codify your shader input/outputs into ever smaller files until you get the desired cache utilization. And yes - one of the later Shader steps for us really did produce (predictable) directories.

In the end it is all about optimizing Bazel caching.

Should that not be enough you could enable Remote Caching or even Remote Execution in Bazel.

 

Another solution would be for those build step to make bazel action run another build system behind the scene, anything which could skip running the real action if not necessary (Solution B). Example FileTracker of MSBuild's .vcxproj or calling ninja managing depfile, ... But those action would not be hermetic, by the would read of write dependency file unmanaged by Bazel. So I guess it would not work with Bazel? Or would be dangerous/error prone?

What we did is indeed have a repository rule, which generated some of the actions from "not so great"-Input formats. The repository rules outputs need to be hermetic, too I am afraid.

If you absolutely cannot manage that you can always wrap Bazel with another script which generates some BUILD files for you (<- which we also did).

 

Otherwise I could generate code before the bazel build and compile shader and asset after the bazel build (Solution C).
It is not very satisfying because I would be back to today's situation where my build system would be some custom script
stiching other build system together. It feels I would lose the major benefit of Bazel.

Conclusion,

1. I wish the Bazel Community can provide some tips about how I can handle dynamic dependencies today. :)
2. Also I have a meta-question about the dynamic dependencies situation. I guess if the issue is not handle by Bazel today, its because it does not appear as an important feature for the majority of user. It is very surprising to me, because from my point of view dynamic dependencies is every where regarding the concept of build system (C/C++, shader compilers, protobuf, other code generator, data converter, etc.). Am I biased because it is actually just a problem for the game industry? What is your opinion?
   

Dynamic dependencies are only great if you don't have so scale. And my theory is if you don't need to scale (< multi million assets and building in seconds/minutes) I think there are more forgiving (and less scalable) build systems than Bazel.

If you want to scale you want to (correctly!) hit caches. This problem is very well understood inside of games. In order to hit caches you have to align and format your data correctly so that the underlying system can correctly reason about dependencies and only do a minimal set of work. This is basically not that different between CPU and Bazel caches.

Hope these comments helped a bit.

Feel free to post more concrete examples, though.

[Marc Delorme]

@lcidfire, thank you very much for your answer. I am glad to get feedback about Bazel from other people of the game industry.

First, I would to like to explain how shader compilation could work according to your suggestion. I want to make sure I have understoop your solution:

1. Because it is not possible to have dynamic dependencies, whatever the action graph required to compile a shader, in the BUILD file, a shader somehow depends on a list of all the header files which could be included.
2. The build can be split in two steps, for instance a pre-processing step gathering all included file into one intermediate file, then a regular compilation step of that intermediate file.
3. Every time I am changing a header file of the list above, every shader will have to be recompiled. First the pre-processing step will be run for all the shaders. Then thanks to bazel cache, the compilation step will run only for shaders whose pre-processing step gave a result which is not in cache. Other shader will be fetch from the cache.
   
4. Because the pre-processing step is much faster than the compilation, its kind of okay to pre-process all the shader everytime.
   

Is my understanding correct?

Then, some more comments:

In the end it all boils down to:

1. Find out what the exact input for each run is

2. Find out what the exact output for each run is

3. Code that into a rule

(4. Find tune using Aspects)

5. Repeat

I am not very familiar with Aspects, I have read the documentation but it never came to my mind how Aspects could help me better defining the build graph. Do you have any example to show how Aspects could be useful in our context (game engine build, game assets build)?

Dynamic dependencies are only great if you don't have so scale.

I feel I am misunderstanding something, because today I have the opposite opinion: a build system which does not support dynamic dependencies is intended not to scale at some point.

In the scenario where it is impossible to know all the inputs during the build declaration (for instance building C++, or the shader compilation scenario above), you have to overdefine the list of your input. For instance for C++ you can make your target depend on all the .h file existing in the include directory list. If you do that all the .cpp files will be recompiled for any header which as been modified. You can do the trick of preprocessing first and then only compile what cannot be fetched from the cache. It is definitely better, but I would not say that method "scale". Because if you increase the number of .cpp file to be compiled, at some point pre-processing all the file will not be neglictable anymore.

We could consider the solution where "the scenario where it is impossible to know all the inputs during the build declaration" is forbidden and considered as a bad practice. But it feels to me it is not sustainable. Not only because legacy languages are forcing us to deal with that problem, but because behind the issue of dynamic dependencies lies the general concept of composability. As an input data author (whatever it is C++ source file, shader, asset, ...), by compositing together data referenced in other file than the input, you are creating a dynamic dependency, i.e a dependency which lies in the data and which cannot be known during the build declaration.

To give a concrete example, let's consider some data which define a texture material for a 3D model. This data could be a complex image processing graph defining how to merge several textures together. Building that data would output one texture. The input of that build step is the processing graph, but inside the graph lies the reference to all input texture. Those input texture cannot be defined in the BUILD because the artist is free to select and add any texture from a large list of textures. You could ask the artist to also change the BUILD file every time he adds/remove a texture from the graph but that would kill his user experience. You could re-generate the BUILD file according to the input data, but because you cannot predict which BUILD file needs to be re-generated, this generation step would need to be run everytime before any build system for every similar asset.

Forbidding dynamic dependencies means forbidding composability inside your data.

If you want to scale you want to (correctly!) hit caches. This problem is very well understood inside of games. In order to hit caches you have to align and format your data correctly so that the underlying system can correctly reason about dependencies and only do a minimal set of work. This is basically not that different between CPU and Bazel caches.

I see a conceptual difference between CPU cache and Bazel cache.

In the context of a CPU a trade off has been made. Your data has to be as cache-friendly as possible and we give up on user friendliness. Those data are not intended to be authored by human but came from an upstream process (loading or build in advance).

On the other hand, in the context of Bazel, because the purpose of a build system itself is to convert from a user friendly data format to another format, the trade-off cannot be all in toward cache friendliness. Otherwise people will starting building a build system on top of Bazel to generate bazel optimized input data from user friendly data, defeating the all purpose.

[lcid...@gmail.com]

Let me try to sum this up:
Bazel works differently than e.g. Unreal Build Tool. In Bazel all input is kept into files whereas in other tools data might be stateful in Process or on a Server.
Bazel also has a focus on correctness. With Unreal Build Tool or Unity it is not uncommon to have to delete caches because of dynamic or implicit dependencies.
So IMO Bazel will be a good choice if you really want to be sure that you produced the correct output (a lesson we had to learn the hard way) and have to process millions of files.
A few downsides to Bazels approach are:
- you need to be super explicit about dependencies
- File IO will easily be a limiting factor and often it forces you to optimize your persisting due to that

My following answer all make the assumption that you care about correctness.

On Sat, 4 Jan 2020 at 16:41, Marc Delorme <delorm...@gmail.com> wrote:

@lcidfire, thank you very much for your answer. I am glad to get feedback about Bazel from other people of the game industry.

First, I would to like to explain how shader compilation could work according to your suggestion. I want to make sure I have understoop your solution:

Because it is not possible to have dynamic dependencies, whatever the action graph required to compile a shader, in the BUILD file, a shader somehow depends on a list of all the header files which could be included.

It does depend how and where you normally configure your shaders.
For us we had configuration split into material properties, texture properties and shader sources (am talking only about the parameters here, not the references). Each of these were processed by rules and ended in more concrete (M/T/S) shader-permutations (I am oversimplifying a bit ;) ).
Anyway, personally I would extract as many clear-cut parameters as possible (into distinct files) and use rules to eventually generate e.g. cpp files for these parameters for your code. Could this work for you? 

The build can be split in two steps, for instance a pre-processing step gathering all included file into one intermediate file, then a regular compilation step of that intermediate file.
Every time I am changing a header file of the list above, every shader will have to be recompiled. First the pre-processing step will be run for all the shaders. Then thanks to bazel cache, the compilation step will run only for shaders whose pre-processing step gave a result which is not in cache. Other shader will be fetch from the cache.
Because the pre-processing step is much faster than the compilation, its kind of okay to pre-process all the shader everytime.

How many shaders are you expecting to have? Where do you store the shaders (HDD, SSD, RAMDISK)? For us touching all shaders everytime did not scale and thus we had to make pre-processing a "manual" step (e.g. backed by a pre-commit hook).

(4. Find tune using Aspects)

5. Repeat

I am not very familiar with Aspects, I have read the documentation but it never came to my mind how Aspects could help me better defining the build graph. Do you have any example to show how Aspects could be useful in our context (game engine build, game assets build)?

IMO aspects are often (complex) icing on the cake. Maybe start defining your graph and once you have a first version working iterate and optimize.
It is easy to get overwhelmed with/in Bazel.

Dynamic dependencies are only great if you don't have so scale.

I feel I am misunderstanding something, because today I have the opposite opinion: a build system which does not support dynamic dependencies is intended not to scale at some point.

Hopefully I could bring across the problem in the first paragraph. Dynamic dependencies scale IMO if you let go of correctness or build times. Which for most Games seems to be an acceptable tradeoff.
If you persist everything into files on disk then dynamic dependencies easily get problematic the more edges in your graph you have. One limit then is that you cannot parallelize actions as well as using (fine grained) static dependencies.
Again, at least we chose Bazel because we wanted both correctness and small build times.

 

In the scenario where it is impossible to know all the inputs during the build declaration (for instance building C++

It is never impossible. I do agree that it is very common in gaming not to have static dependencies because it may make things like iterating easier. It is often hard to transition to static dependencies when you did start of with dynamic dependencies. Especially in C++ you normally know exactly which headers or inlining to include.

(for instance building C++, or the shader compilation scenario above), you have to overdefine the list of your input. For instance for C++ you can make your target depend on all the .h file existing in the include directory list. If you do that all the .cpp files will be recompiled for any header which as been modified.

My advice would be to try to narrow shader dependencies down as much as possible. You will perhaps not reach totally static dependencies but every step in a more rigid expression of dependencies can help.

 

You can do the trick of preprocessing first and then only compile what cannot be fetched from the cache. It is definitely better, but I would not say that method "scale". Because if you increase the number of .cpp file to be compiled, at some point pre-processing all the file will not be neglictable anymore.

For us we easily had 100x asset sources compared to C++ files. As stated earlier preprocessing did not scale at these file counts. Then the trick is mostly to find the point in your workflow where to best do the processing (background worker on Workstations, SCM hooks, RAMDISK to minimize IO) to not halt the pipeline.

 

We could consider the solution where "the scenario where it is impossible to know all the inputs during the build declaration" is forbidden and considered as a bad practice. But it feels to me it is not sustainable. Not only because legacy languages are forcing us to deal with that problem, but because behind the issue of dynamic dependencies lies the general concept of composability.

Bazel IMO is a very opinionated software. I would phrase it more on the lines of: "if you cannot go into a direction of static dependencies you might not get that much benefit from Bazel".

 

As an input data author (whatever it is C++ source file, shader, asset, ...), by compositing together data referenced in other file than the input, you are creating a dynamic dependency, i.e a dependency which lies in the data and which cannot be known during the build declaration.

What you could do is e.g. instead of having files implicitly referencing other names (or files) express this information directly in BUILD files and using rules than generate your composited files. But yes if you want to take advantage of benefits of Bazel you might need to rework the way your builds work.  

 

To give a concrete example, let's consider some data which define a texture material for a 3D model. This data could be a complex image processing graph defining how to merge several textures together. Building that data would output one texture. The input of that build step is the processing graph, but inside the graph lies the reference to all input texture. Those input texture cannot be defined in the BUILD because the artist is free to select and add any texture from a large list of textures. You could ask the artist to also change the BUILD file every time he adds/remove a texture from the graph but that would kill his user experience. You could re-generate the BUILD file according to the input data, but because you cannot predict which BUILD file needs to be re-generated, this generation step would need to be run everytime before any build system for every similar asset.

The latter is nearly what we did.
Imaging a directory llike this:

- glider
|- BUILD
|- glider.scene (references meshes, materials, shader, tex) 
- meshes
|- BUILD
|- some.mesh
|- more.mesh  
- materials_1
|- BUILD
|- green.material
|- rust.material
- materials_2
- shader
|- BUILD
|- superduper.shader
- texture
|- BUILD
|- metal.tex

So if our artists worked on glider.scene, we knew which meshes, materials, shaders and textures were referenced and which package these belonged to. And we only had to APPEND to these BUILD files. So we only ensured, that all permutation that the artists wanted were actually expressed in the files. Permutations not used mean a bit of loading time in Bazel but not enough that we had to remove them again. So over time these files grow and you reach a sweet spot where all permutations are mostly in (garbage collecting in SCM would not have been to hard, though). And then only let Bazel figure out what it needs to build.

 

Forbidding dynamic dependencies means forbidding composability inside your data.

If you want to scale you want to (correctly!) hit caches. This problem is very well understood inside of games. In order to hit caches you have to align and format your data correctly so that the underlying system can correctly reason about dependencies and only do a minimal set of work. This is basically not that different between CPU and Bazel caches.

I see a conceptual difference between CPU cache and Bazel cache.

In the context of a CPU a trade off has been made. Your data has to be as cache-friendly as possible and we give up on user friendliness. Those data are not intended to be authored by human but came from an upstream process (loading or build in advance).

On the other hand, in the context of Bazel, because the purpose of a build system itself is to convert from a user friendly data format to another format, the trade-off cannot be all in toward cache friendliness. Otherwise people will starting building a build system on top of Bazel to generate bazel optimized input data from user friendly data, defeating the all purpose.

If you do commit to the analogy fully, then also BUILD files may not be user-facing and can be generated (I would recommend that).

Hope that input helped.