Re: Providers and GN

[Dirk Pranke]

Thanks for the post, it's very educational. And I apologize for my delay in replying.

GN was never intended to have a full-blown programming language like Starlark as part of it. And as you might have picked up from the threads Andrew linked to, quite a bit of its design is constrained by ensuring that we can do as much in parallel as possible. I have never been that familiar with these constraints, and so it's hard for me to say how easy

it would be to preserve the parallelism and still get something closer to Starlark and a cleaner way of implementing new rules and target types. But, certainly you can imagine better APIs and designs that would make things like what you describe possible. I'm not sure how big the "can't think only in terms of labels" problem is (I think it mostly comes up when defining templates, not using them), but it's certainly true that toolchains are limited and that writing high quality rules is hard compared to having something like Starlark.

I think in general the historical answer to this from people like Brett and myself is that if we got to a point where the logic got complicated and it really needed to be in GN itself (to preserve GN's knowledge of the build graph), then we should hard-code the logic into the binary in C++. You can see cases where we did that, for how things work on iOS and Mac, and in the Rust code, and cases where we didn't (e.g., for Android and Java). Obviously, hard-coding things makes it much harder to change. I don't think that's been a problem in practice for the iOS/Mac rules, as those haven't changed much over the years. I am not close to the Rust work so I don't know how well that has worked out, and I cannot say if the Android rules stabilized enough that we would've felt safe hard-coding them. But, I think over time we haven't explored hard-coding things as much as we should have. I think part of that is that no one works on GN full-time, and so people have probably felt reluctant putting code in someplace where it might be harder to maintain. I think we've also been leery about hard-coding stuff that might be Chrome-specific or Fuchsia-specific and not generally usable across projects.

I do want to call out something about your last bullet specifically, however. You wrote that "build rules feel like code, rather than config". This is true, and IMO it's actually a good thing in one specific sense: the procedural flow through a target's definition feels very natural to programmers used to conventional imperative languages, and this makes it *very* easy to understand how conditionals work and interact. I have never been able to think of a more declarative way to do the same thing that gets anywhere close to the expressiveness and ease of use of GN's approach. And the lack of good conditional support at the user level is probably the major thing stopping a project like Chrome from moving to something like Bazel (unless something has changed since I last looked at this problem). Obviously a language like Starlark does have this kind of expressiveness, but then you'd be making users read and write Starlark code to declare targets, and that would produce the same problem you object to.

-- Dirk

On Wed, May 28, 2025 at 5:35 PM Matt Stark <ms...@google.com> wrote:

I stated earlier in another thread that I think that providers could remove about half of the issues I have with GN, and I got some interest in going into more detail, so I'm starting a new thread here.

Coming from bazel, my biggest complaints about GN are:

• A user cannot think purely in terms of labels, they must think in terms of files
• Eg. Any time you have to use rebase_path
• Eg. Any time you have to rely on naming conventions
• Eg. Any time you have to distinguish between root_build_dir, root_gen_dir, etc.
• Modifying toolchains requires modifying GN itself (sort of - GN has "toolchains", but they're pretty weak by comparison - clang modules, for example, require cooperation from GN itself)
• There is no way to write good quality reliable rules.
• Build files feel like code, rather than config

A quick overview of bazel, for context (with some minor complexities simplified a little). Bazel has a few really fundamental concepts to it that work extremely well together:

• A label is a way of referring to a target. It can be absolute or relative. GN has this already (//foo/bar:baz and :baz).
• Every target has associated metadata in the form of providers
• Each different provider is a different topic of information (eg. CcInfo, RustInfo, etc.)
• They are basically just structs
• They usually contain files, but may contain arbitrary information
• A rule is a function that creates targets.
• It outputs providers
• It can consume providers from other rules
• It can declare files
• And if it does, it must declare an action which generates it
• An action is:
• A list of input files
• A list of output files
• A command-line to run
• DefaultInfo is a provider that almost every rule outputs. It contains:
• Files (set of files that are built whenever you run `bazel build //path/to:target`)
• Runfiles (set of files that are built whenever you want to run this target, either as `bazel run //path/to:target`, or in another build action)

Providers can solve or mitigate all of the problems I mentioned above. Consider the following rule:

CcInfo = provider(fields = ["headers", "transitive_libs"]

CcToolchainInfo = provider(fields = ["ccflags", "ldflags"])

def cc_binary_impl(ctx):

  deps = [dep[CcInfo] for dep in ctx.attr.deps]

toolchain = ctx.toolchain[CcToolchainInfo]

  compiled = ctx.actions.declare_file(ctx.label.name + ".o")

  ctx.actions.run(

    inputs = [...] + [dep.headers for dep in deps],

    outputs = [compiled],

   args = ... + [dep.headers for dep in deps] + toolchain.ccflags

 )

  binary = ctx.actions.declare_file(ctx.label.name)

  ctx.actions.run(

    inputs = [...] + [dep.transitive_libs for dep in deps],

    outputs = [binary],

   args = ... + [dep.transitive_libs for dep in deps] + toolchain.ldflags

 )

  return DefaultInfo(files = [binary], runfiles = [dep.runfiles for dep in deps])

cc_binary = rule(

implementation = cc_binary_impl,

attrs = {

"srcs": attr.label_list(allow_files = True),

"headers": attr.label_list(allow_files = True),

"deps": attr.label_list(providers = [CcInfo]),

  "toolchain": attr.label(default = "//toolchains:cpp", providers = [CcToolchainInfo]),

},

)

This rule:

• Depends on .o files declared by other rules without ever mentioning their file paths, or following any sort of naming conventions
• Has complete control over the toolchain (this isn't quite how bazel does it, but I've simplified things, and this would work), and complete control over how the command-line gets created
• Can depend on any rule that outputs CcInfo. This could be static_library, shared_library, prebuilt_shared_library, or any rule that you create
• Puts the complexities in the rule definition, allowing the build files to be trivial.
• Completely removes the need for complexities such as "public deps" or "public config". Instead, you can just add a field "transitive_deps", and then put your direct dependencies' transitive_deps into your CcInfo (this is performant via the use of a depset type).

This also allow you to create rules like:

• Alias - eg. foo is an alias to foo_linux on linux, and foo_windows on windows
• Filegroup - Group a bunch of sources together (which can be rules)
• Config-only rules
• Eg. I wrote rules_toolchains for bazel, which allows users to specify arbitrary toolchains via build files (example)

--

Thanks, Matt.

[Matt Stark]

Would I be correct in saying that the only reason we don't do this, then, is performance and implementation complexity?

Re performance, I think your concerns are very fair about serialization of an otherwise parallel task. I've been working on a prototype so I can evaluate the performance impact. I think with my design, the performance impact should be zero if you don't use providers, and relatively small if you do.

Re implementation complexity, I was actually able to make the implementation quite simple. I simply:

• Changed the loading of a build file to, when it sees a target, schedule loading of the target instead of actually doing it
• When it sees a call to get_provider("//foo/bar:baz", "DefaultInfo"), it:
• Schedules a load of //foo/bar/BUILD.gn
• Waits for the target //foo/bar:baz to be loaded
• Note that this can only happen inside target definitions

--

Thanks, Matt.

[Dirk Pranke]

I'm not sure what you're referring to with the "this" in "don't do this"? What is it you're proposing we do?

-- Dirk

[Matt Stark]

Ah, sorry. The "this" is to add some kind of metadata to targets visible to other targets.

--

Thanks, Matt.

[Dirk Pranke]

Then I think it would depend on the proposal :).

-- Dirk

[Roland McGrath]

https://docs.google.com/document/d/1v3RN2Ut4wbD0RGnSfBeVBmyp9JOqNj1Wj3uYc-oCM9w/edit?usp=sharing has the original proposal for the richer `metadata` feature.  This design was an attempt to split the difference between the richer inter-target data visibility without changing GN's basic evaluation model.  It didn't get implemented, and we found the `generated_file` approach to metadata to be adequate at the time; its limitations have not become more pleasant over time...  Certainly if it's feasible to do metadata queries directly via evaluation ordering without introducing the `metaresult` type and its nontrivial delayed-evaluation semantics, that will be more straightforward in using the feature and understanding it.