Bundling dawn into a monolithic shared lib

[Jaswant Panchumarti]

Hello everyone,

Over the last weekend, I've been wrestling to get the C entry points to dawn's webgpu implementation loadable from a single shared library on mac-arm64, linux64 and win64. It's been difficult trying to get everything to cooperate, but I'm excited to share that it is possible and you can try the proof-of-concept at https://github.com/jspanchu/webgpu-dawn-binaries/releases/tag/v127.0.6531.1!  Please don't use in production yet.

The single dawn library exposes pure C functions of dawn's webgpu impl and tests it in CI across the three platforms. The POC is interesting in some ways:

1. it pulls in `depot_tools` uses `gclient sync` from cmake, so you don't even need to manually install depot_tools and what not. All you need is git, CMake, a recent C++ compiler, hope and patience. Reason to prefer `gclient sync` is its stability over `fetch_dawn_dependencies.py`, I hope this changes in the future.

2. final library size is more conservative. it's smaller than that generated by dawn's existing BundleLibraries.cmake by 3MB

Here's the bundle sizes across different OS:

du -bsh webgpu-dawn-127.0.6531-*/*/*dawn.{dll,so,dylib}

8.0M webgpu-dawn-127.0.6531-win64/bin/dawn.dll
15M webgpu-dawn-127.0.6531-Linux/lib/libdawn.so
7.6M webgpu-dawn-127.0.6531-Darwin/lib/libdawn.dylib

Over the next couple weeks, I plan to slowly adapt/overhaul the CMake build system in dawn so that one may obtain these libraries directly by building from source. Ideally, I wish to see webgpu_dawn be a thing people download using apt/dpkg/dnf/choco/vcpkg/etc. There will be lots of issues but I think we can figure them along the way :)

[Corentin Wallez]

Hey Jaswant,

That's pretty exciting! Adding prebuilt has been on our TODO list for a while but we didn't want to tackle it before we declare webgpu.h ABI stable, but if you have it working already I think we could start providing prebuilt with a large disclaimer instead :) ABI (in)stability and the lack of LTS version which will have bugfixes for an extended period of time mean that we don't think that Dawn would be a good library to bundle in apt and friends though. We have a github repo at google/dawn where we could add the actions to do the build. I think a lot of folks would be interested in linking directly to the libs instead of dynamically loading symbols, but that would be an easy addition to the build actions.

Do you know why `gclient sync` is more stable than `fetch_dawn_dependencies.py` (aside from the latter not being tested on CI so it tends to require small fixups from time to time)? We'd want folks using the CMake build to not have to care about depot_tools. If we add Github actions, changes upstream that break actions or fetch_dawn_depdendencies.py would trigger emails / redness that would get fixed quickly.

The final binaries seem kinda big (even though we know Dawn is not exactly tiny ^^). Are all functions / data section-GCed by the linker and binaries stripped?

Looking forward to the changes upstream, the contributions you've done so far to the CMake files have been great! (though as noted on Matrix we are stuck with CMake 3,13 for another couple weeks and then maybe can only use 3.18).

Cheers,

Corentin

--
You received this message because you are subscribed to the Google Groups "Dawn Graphics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dawn-graphic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dawn-graphics/d585ed9f-6ab6-47f6-876c-ce58747aef2cn%40googlegroups.com.

[Jaswant Panchumarti]

> We have a github repo at google/dawn where we could add the actions to do the build.

got it, i'll take a look and see if i can upstream my github workflow when ready.

> I think a lot of folks would be interested in linking directly to the libs instead of dynamically loading symbols, but that would be an easy addition to the build actions.

the POC also installs the cmake plumbing required for one to discover dawn library and headers via find_package(dawn).

> Do you know why `gclient sync` is more stable than `fetch_dawn_dependencies.py` (aside from the latter not being tested on CI so it tends to require small fixups from time to time)? We'd want folks using the CMake build to not have to care about depot_tools. If we add Github actions, changes upstream that break actions or fetch_dawn_depdendencies.py would trigger emails / redness that would get fixed quickly.

That code path needs to be exercised in CI. After being affected by the recent protobuf breakage, I've tried to stay away from fetch_dawn_dependencies

> The final binaries seem kinda big (even though we know Dawn is not exactly tiny ^^). Are all functions / data section-GCed by the linker and binaries stripped?

I haven't analyzed the binaries in detail, i'll eventually get to it.

thanks!

- jaswant

[Jaswant Panchumarti]

I plan to reorganize the BundleLibraries.cmake functionality by lifting it from "tools/android/webgpu" up into "dawn/cmake" after landing https://dawn-review.googlesource.com/c/dawn/+/192641. That's likely better for reusability.

[Austin Eng]

Thanks for working on this! Super exciting to see improvements here.

FYI we do already have Github workflows though they are partially broken at this time. Please feel free to modify what we already have at https://github.com/google/dawn/blob/main/.github/workflows/ci.yml

To view this discussion on the web visit https://groups.google.com/d/msgid/dawn-graphics/3c9d5e5b-a991-456a-9587-6eea18bdfa18n%40googlegroups.com.

[William Candillon]

A big thank you to Jaswant for starting this thread; it helped me solve some of my build issues, and I am interested in using a build option that creates a monolithic lib file.
To set some context, I am looking to distribute pre-built binaries for Dawn on the iOS and Android platforms via GitHub Actions.

Originally, I distributed "libwebgpu_dawn," "libdawn_native," "libdawn_common," and "libdawn_proc," but that didn't work because I would end up with a null proc table at runtime.

This discussion thread made me realize that I needed to do something similar to what is done in tools/android and that I would need to set up my own CMakeLists.txt just for the prebuilt binaries.

It is working quite well. This is my CMakeLists: [https://gist.github.com/wcandillon/d22b3c5852accc3f69571d6607a57e6f](https://gist.github.com/wcandillon/d22b3c5852accc3f69571d6607a57e6f).

Like Jaswant, I had to use depot_tools, and fetch_dawn_dependencies didn't work for me either.

The lib size on iOS looks very decent (on par with what Jaswant reported), but on Android, I am getting enormous lib sizes (>150MB). I will get the same sizes if I use the Android build pipeline from tools/android. Could it be something related to my NDK? It may be too old. Or is there another issue there?

I am definitely interested in any future build options that Dawn might offer. I am also experimenting with using a static lib file instead of a dynamic lib.

Thank you for the great work. I would love to test new build options once they are available. I could also contribute some small documentation to help other implementors, but I need some leads on what implementors should know about this topic.

Kind regards,

William