WASM builds/toolchains for Dawn

[Loko Kung]

Hello build folks,

I'm Loko from Dawn team and I was looking into Chromium's build/config/BUILDCONFIG.gn file, because I wanted to see if it would be possible to add wasm targets much like how Skia does in theirs an was wondering a couple of things...

• Is it possible to "extend" the existing BUILDCONFIG.gn file in Dawn only to provide a new toolchain to build wasm targets? I don't want to change it for all of Chromium since it doesn't make sense for Chromium, but I also don't want to branch Dawn entirely off of the same config of chrome.

• If it's not possible and I do need to change the existing file, does it make sense to add a dependency to emsdk (which contains Emscripten toolchain for building) as a DEP?

I'm not too familiar with the build / toolchain stuff so feel free to correct me if I'm wrong on how some of these things work heh...

- Loko

[Dirk Pranke]

On Tue, Feb 11, 2025 at 12:49 PM 'Loko Kung' via build <bu...@chromium.org> wrote:

Hello build folks,

I'm Loko from Dawn team and I was looking into Chromium's build/config/BUILDCONFIG.gn file, because I wanted to see if it would be possible to add wasm targets much like how Skia does in theirs an was wondering a couple of things...

• Is it possible to "extend" the existing BUILDCONFIG.gn file in Dawn only to provide a new toolchain to build wasm targets? I don't want to change it for all of Chromium since it doesn't make sense for Chromium, but I also don't want to branch Dawn entirely off of the same config of chrome.

Probably? It might depend on the specifics of what you want to do. I am not entirely opposed to adding stuff to Chromium's BUILDCONFIG that might only make sense for other Chromium-related subprojects.

• If it's not possible and I do need to change the existing file, does it make sense to add a dependency to emsdk (which contains Emscripten toolchain for building) as a DEP?

It might, again, I'd have to look at the specifics.

Depending on how things work, it might make sense to directly add support for a wasm toolchain unconditionally to Chromium's BUILDCONFIG.gn, or it might make sense to guard it behind a conditional. 

We have a crude mechanism for doing project-specific configuration of stuff in GN files using files that are checked into a //buildconfig directory, e.g., //build_overrides/dawn.gni. We could possibly add a //build_overrides/buildconfig.gni file. But I probably wouldn't do this unless we really needed to.

I suggest you post a CL with the changes you'd like to make and we can review it that way.

-- Dirk

I'm not too familiar with the build / toolchain stuff so feel free to correct me if I'm wrong on how some of these things work heh...

- Loko

--
You received this message because you are subscribed to the Google Groups "build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to build+un...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/build/CAFAqEFgiWf0ii2C2b2LR8fx46O%2B656oW%2BFx%3DQ6vv-3PiTFqE9A%40mail.gmail.com.

[Andrew Grieve]

There are a few spots in chromium that are using wasm, and just going the route of checking in the .wasm files. That's pretty not great, so I do think there'd be appetite for adding proper wasm support. The main tricky thing is that we'd want to use chrome's copy of LLVM and not the one bundled with emsdk.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/build/CAEoffTB0ixzZXO%3D1-QNrVs5RGXZdR1k%2BzTvJtXhgxHen1c4J%2BA%40mail.gmail.com.

[Loko Kung]

Thanks Dirk and Andrew for the replies!

I think that support for wasm should almost certainly be behind a conditional or flag. Trying to add the DEPS and use emsdk with Chrome's copy of LLVM seems like a rather daunting task for me to dive into, so I'm leaning towards trying to do it just for Dawn for now. Do you know, for the other places that Chromium is using wasm and checking in the .wasm files, whether they are building those .wasm files with Chromium's LLVM or the one shipped with emsdk? Could you also link me to some of the examples of such users for reference?

Regarding Dirk's reply to "extending" the BUILDCONFIG, what would the proper way to go about it look like? Are there any examples or previous changes similar to this that I could reference? Or is this unknown territory? I would love to put together a CL or a prototype and get more reviews with this, but I'm trying to find a starting point so any pointers would be super helpful here.

- Loko

[Dirk Pranke]

If we weren't going to just modify BUILDCONFIG.gn unconditionally, then what I would do would be to:

1. Create a //build_overrides/buildconfig.gni file, add something like `enable_wasm_toolchain = false` to it.

2. Modify //build/BUILDCONFIG.gn to include the code you wanted inside a `if (defined(enable_wasm_toolchain) { ... }` block, and add `import("//build_overrides/buildconfig.gni")` to the top of the file.

3. Land that CL, roll it into Dawn (which will require you to add a matching //build_overrides/buildconfig.gni file in the Dawn repo with that flag set to true). We'd probably also expect you to roll that version of //build into the other repos that use it, so this is an annoying amount of work.

-- Dirk

[Andrew Grieve]

Here's the one I did:

https://chromium.googlesource.com/chromium/src/+/main/tools/binary_size/libsupersize/viewer/caspian/README.md

[Loko Kung]

Taking a look at Andrew's README, doesn't that patch still use the LLVM shipped with emsdk to do the compilation?

[Andrew Grieve]

Yes, which is why I can't check it in properly :P. 

[Loko Kung]

Ok, so I've been trying to pull the toolchain in and messing around with the changes... One thing I notice that seems to be a massive pain if I'm trying to add the toolchain globally is that a lot of dependencies are checking `current_cpu`, which I am setting to "wasm" when `target_os = "wasm"`, and asserting since it's not a platform they support. The intention is that these dependencies won't ever be compiled using wasm anyways, but updating all their BUILD/*.gni files doesn't make sense to me. Is there a way to specify not to build or even check certain dependencies when running `gn args <dir>`? Or better, is there a way to create rules to explicitly specify if something is even wasm buildable, otherwise ignore them?

- Loko

[Dirk Pranke]

When you say "add the toolchain globally", do you mean you're just declaring a `target_os = "wasm"` and trying to build? 

By default, that'll try to define all the targets transitively reachable from //BUILD.gn, so you almost certainly don't want that. Wrap the targets in //BUILD.gn in a `if (target_os != 'wasm') { ... }` block, and then declare only the targets you want in an `else` block.

Or, probably better, use your normal target_os (e.g., `target_os = "linux"`) and then take some normal target and add a dependency on your new target in the wasm toolchain, e.g., add it to the //:gn_all target (`if (enable_wasm) { deps += [ "//your/target(//build/toolchain/wasm:wasm)" ] }` or something like that.

-- Dirk

[Loko Kung]

Ah, yes I was originally trying to set `target_os = "wasm"`, and that was causing a lot of failures. For using the "//target(//toolchain)" syntax, will I need to include the toolchain in the deps of recursive deps as well? Or does the build system just know that the target and all its deps should be built via the toolchain override?

[Dirk Pranke]

When GN resolves a dependency into a different toolchain, all of that target's dependencies will be in the same toolchain unless explicitly specified otherwise. Put differently, once you switch toolchains, you stay there unless you ask to switch again.

-- Dirk

[Loko Kung]

Ok, so I managed to make a ton of progress towards getting it to work! I think I'm able to use the LLVM provided by Chromium, but I am running into issues with standard system libraries, specifically, the one I am seeing errors for is `libc++abi`. Normally, using Emscripten, we would build against the `libc++abi` that is provided from Emscripten, but when I was looking through the `/build` directory, it seems like Chromium wants to use its own copy/version of the standard libraries (see the comment here: https://source.chromium.org/chromium/chromium/src/+/main:build/config/c++/c++.gni;l=14?q=deprecated&ss=chromium%2Fchromium%2Fsrc:build%2F&start=11), so I am forcing the builds to use the one built from the source. Unfortunately, there seems to be some Emscripten specific functions that are only available in the Emscripten source for the library (basically this block: https://github.com/emscripten-core/emscripten/blob/6fd5a1faf03c1d4784fa11bf8fe6baf133d830a6/system/lib/libcxxabi/src/private_typeinfo.cpp#L1549). Note that the equivalent file used in Chromium (https://source.chromium.org/chromium/chromium/src/+/main:third_party/libc++abi/src/src/private_typeinfo.cpp) doesn't have that block, nor does the upstream repository that Chromium depends on (https://github.com/llvm/llvm-project/blob/main/libcxxabi/src/private_typeinfo.cpp). 

I'm not sure what to do here in this case. I do see that some build targets do still use a custom libcxx (I think it looks like maybe some Windows stuff and Android?). Would it be fine to use the libraries shipped with emsdk in this case? Or do we need to find another way to build the library to make this compatible?

- Loko

[Dirk Pranke]

I think you could probably try turning off the GN arg `use_custom_libcxx` for your toolchain via `toolchain_args` in your toolchain definition, and then just make sure the toolchain picks up the library you want.

But I'm not 100% sure here, I've never been that solid on how the custom libc++ stuff works; maybe thakis@ or someone else can chime in.

What we would actually want to do (which library to use for real) is probably a different question.

-- Dirk 

[Loko Kung]

So with some tinkering, I've managed to get it to work, at least for a small prototype in Dawn. Some things I want to get clarified before I try to push this any further.

While I was able to get emsdk to use the LLVM shipped with Chromium/Dawn, I was informed that other binaries that it may depend on could be problematic if we ever want to get these builds on CQ. Specifically, I would assume that the node and Binaryen binaries that ship with emsdk could be problematic? I think node is an easy fix as I could just point it to a node version/package that is downloaded from google storage, but for Binaryen, do I have to assume that I would need to build that from source to ensure that it's a "safe" binary? Moreover, if that's the case, I'm wondering whether I even need a dependency on emsdk, and could instead just depend on emscripten directly, setting the necessary envvars to make it work?...

For iterative development in the meantime, however, would it be fine to add emsdk as is, and use the Binaryen shipped with it? We don't need to have these builds on CQ right now, so maybe it's fine for local development, and if needed, we could try to get Binaryen building in gn and pointing to it later on?

- Loko

[Andrew Grieve]

Sounds like great progress!

There's no requirement to have things built from source (that I know of). We just need a higher level of trust than "devs upload binaries from their machine". This can be done via the 3pp bot, which uploads things to cipd, which can then be pulled in via DEPS file. The JDK one is probably a good one to copy.

IIRC, the reason for using our own copy of llvm is not about trust, but about not having to support multiple compilers (or multiple versions of the same compiler).

[Loko Kung]

Hello, so here's what I have so far... https://chromium-review.googlesource.com/c/chromium/src/+/6282629. I'm running into issues where it seems like there's some automation that figures out I added a new toolchain and is trying to build stuff with it? I'm assuming there's a way to change this behavior, but I am not sure where to find it? Ideally, we should only use the toolchain if we explicitly ask for it.

- Loko

[Dirk Pranke]

That looks plausibly correct to me, bearing in mind that I know virtually nothing about the details of using wasm and emscripten :). I'd definitely want Nico to give his stamp of approval as well, and this is possibly something that should be raised on chrome-at...@google.com as per https://chromium.googlesource.com/chromium/src/+/main/docs/adding_to_third_party.md#before-you-start, although maybe it's not necessary if this is just for standalone Dawn at this point.

-- Dirk

[Loko Kung]

Yea, the goal is only standalone Dawn right now. I'll address your comments in the CL, but how can I get rid of the CQ errors that are trying to build stuff with the new toolchain?

[Dirk Pranke]

Ah, I didn't actually look at the errors, sorry. 

What's going on is that in chromium src's //BUILD.gn //tools/binary_size:binary_size_trybot_py is being added to //:gn_all, which causes //tools/binary_size/BUILD.gn to get read in. At that point, gn will learn about all of the targets defined in that file, even if they're not actually dependencies of //:gn_all. So, since `is_wasm` is now defined in BUILDCONFIG.gn from your patch, the //tools/binary_size:caspian_web target gets defined. On some builders this may be sufficient for `gn gen` to fail (e.g., ios-simulator). On others, `gn gen` succeeds, but if the builder tries to build `all` (which some builders do), it'll then try to build that target, which isn't working since you don't have the right stuff pulled into a Chromium build yet.

So you need to work out with Andrew how you want to handle the //tools/binary_size/BUILD.gn file :).

-- Dirk

[Andrew Grieve]

Feel free to just rename the "is_wasm" in that file for now. I can work towards making it work on top of your work, at some later point.