Action wrapper script, perhaps?

[Jonathan Perry]

I’m looking to integrate bazel with nixos in a natural way. Specifically, I’m hoping that I can come up with a system that can decorate an action with the set of nix packages it requires, and from that decoration have bazel invoke the action using either a dynamically generated flake or a nix-shell —pure invocation. 

In order to do this I think I want a hook in bazel which can wrap all actions in a sort of ‘env setup’ script, similar to what the sandbox execution does (at least as I understand it). 

Does this make sense as an approach? I think this will result in a totally safe and hermetic setup, trivially remoteable (as long as the target remote execution env has nix of course!). 

Thank you!

[James Sharpe]

I think you want to look at tweags rules_nixpkgs 

https://github.com/tweag/rules_nixpkgs?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAPDgU36C65P0gVEp2H3gZciprz72bTktbibqst5NkiJmTVo5TQ%40mail.gmail.com.

--

James Sharpe | Software Architecture & Security | Zenotech

	+44 (0) 7950 592277
	james....@zenotech.com
	www.zenotech.com

Company Registration No : 07926926 | VAT No : 128198591 Registered Office : 1 Larkfield Grove, Chepstow, Monmouthshire, NP16 5UF, UK Address : Bristol & Bath Science Park, Dirac Cres, Emersons Green, Bristol BS16 7FR

[Jonathan Perry]

I have thank you. It does a good job if you have a remote environment which already has the packages you want installed. But adding a new set of stuff requires you to go and install that stuff everywhere else, which loses the power of a declarative package manager like nix. I think doing it the way I’m envisioning would be an improvement as a totally clean nix env would be sufficient in your remote build and your local one - and in fact there would be zero difference between the two, which is where we all want to be. 

[Jonathan Perry]

Is this a crazy approach to consider? Is there some other way that people can think of please?

Thank you!

[pcl...@google.com]

There is an upcoming BazelCon talk you might be interested in: https://opensourcelive.withgoogle.com/events/bazelcon2022?talk=day1-talk4

[Jonathan Perry]

Thanks for that! I’m in contact with aherrmann via an issue on the rules_nixpkgs repo. There seems to be some gentle consensus that a better approach is possible, but we need to keep exploring how to go about it. 

https://github.com/tweag/rules_nixpkgs/issues/180

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c51c787b-22c6-4341-a6b6-1a840437e0bbn%40googlegroups.com.

[Jonathan Perry]

Hello again!  I would like to push on this a liittle more with the collected wisdom of this group please.

After some amateur perusal of the bazel codebase, I think the right place to end up here would be a Skylark-driven API around a derivation of (or delegating implementation of) SpawnStrategy, which would allow us to generate a commandline prefix per target based on the transitive closure of values of some set of providers across the dependencies of the target.  This would then provide us with finegrained control of the execution environment, with whatever setup/teardown code the build platform requires.

I am not sure what that api would look like, as it's a completely cross cutting concern which we would want to apply across all kind of rules (i.e. in-workspace rules, thirdparty skylark-native rules, and java-native rules (such as JavaCompileAction).  Perhaps adding a new Provider class, with an instance per toolchain would be a good start.  Then perhaps an aspect for the collection and merge into a PrefixProvider?  And then when you configure the new SpawnStrategy, you point it at that aspect.

Thoughts?  In fact perhaps the only thing we need is the last thing, as everything else can be totally user-defined...

Thanks for your time!

[Jonathan Perry]

One more point.  It's worth noting that from the nixos perspective, we can either generate command-line arguments specifying the required packages (thus requiring us to change the command line prefix per target) or else we can point at a generated file (thus 'only' requring generation of the file per target).  Perhaps the second approach would be better, as then we could use the usual mechanisms to ship files around the place for the environment definition, and point the stable command like prefix at that.  Hope that made sense...

I can trivially see how to change Bazel to add a new Module that would delegate to DynamicSpawnStrategy, and would add a prefix to every inbound command line.  I can't at the moment see how to inject a per-target file (perhaps the result of an aspect) into the Spawn's toolFiles (presumably).

[Jonathan Perry]

Does anyone have any insight here? Should I be asking this kind of question in a different forum?

Thank you!

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a69d1a62-4bfa-4d19-b30e-8c903b8570dbn%40googlegroups.com.

[Jonathan Perry]

Gentle ping for advice please 

[Philipp Schrader]

I am admittedly ignorant of NixOS and extending the list of available spawn strategies, apologies.

One thing that comes to mind is that if you can indeed add a new spawn strategy (defined via Starlark?) or add support for an action wrapper, you could possibly get away with the following?

• Create an external repository filled with empty marker files. Each marker file represents one NixOS package (right term?).
• Any target can then add "data" dependencies on any number of these marker files.
• When the spawn strategy starts or the action wrapper executes, it takes inventory of the present marker files. It can then build(?) the nix environment and delegate to nix-shell --pure.
  • It should be able to take inventory of the requested packages by either reading the manifest or listing the contents of the external/nix-markers/ runfiles directory.

It's likely that I have the wrong mental model for how Nix actually works. Really all I wanted to communicate is that you can use "data" dependencies to communicate with the spawner/wrapper instead of adding support for decorating actions.

Phil

[Alex Eagle]

Andreas led a session at the BazelCon community day about this. Those weren't recorded, but maybe someone can provide notes.

[Jonathan Perry]

@philip - thank you for your thoughts! That will help with passing the info to the spawn for sure, but I still think I will need to figure out the transitive closure of external dependencies so that passing references through providers will work.

@alex thanks for the info; I hope someone has details!

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/b9e19739-1ff0-45dc-95be-0f8fd0bd999dn%40googlegroups.com.