Adding a destination path for loadable_modules

[John Bauman]

Hi all,

The loadable_modules SDK atom is designed to take a list of binaries and other resources, so that they can be included in a package and loaded at runtime. We're currently using this to expose some vulkan layers in the SDK, and it sounds like devicetree will be using this in the future.

We're running into the problem that the schema doesn't allow us to specify a destination path in the package where each file should be located. For the vulkan layers, the binaries must go under lib/ and the resources must go in data/vulkan/explicit_layer.d. This knowledge is hardcoded into the GN SDK generator, but not reflected elsewhere. I don't know what the layout requirements are for devicetree visitors; can someone else chime in on what they are?

I'm thinking we should modify the loadable_module schema to have a destination path for each file, so the Bazel SDK won't have to make assumptions about the layout needed. Any objections?

[David Dorwin]

Some related observations:

• Unlike static and shared libraries, there is not a GN template (e.g., //build/cpp/sdk_shared_library.gni) for loadable modules.
• Instead, //src/lib/vulkan/BUILD.gn does lower level work than possibly any other SDK component.
• Even though there is currently only one instance, it may be better to have everything in one place.
• Even with those templates, the IDK build is currently fragile with respect to naming and the checks are insufficient (https://fxbug.dev/323441662), so checking the resulting IDK and LSC will be important if changing anything other than metadata.
• Chromium's GN SDK may also need to be updated if the schema is changed.

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "sdk-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sdk-dev+u...@fuchsia.dev.
To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/sdk-dev/CAAr6sCcTHYr66-ktm-cxXB%3DB2Pwf4RO2iM95t8H56_GfgnjXvw%40mail.gmail.com.

[John Bauman]

On Wed, Mar 6, 2024 at 4:05 PM David Dorwin <ddo...@google.com> wrote:

Some related observations:

• Unlike static and shared libraries, there is not a GN template (e.g., //build/cpp/sdk_shared_library.gni) for loadable modules.
• Instead, //src/lib/vulkan/BUILD.gn does lower level work than possibly any other SDK component.
• Even though there is currently only one instance, it may be better to have everything in one place.
• Even with those templates, the IDK build is currently fragile with respect to naming and the checks are insufficient (https://fxbug.dev/323441662), so checking the resulting IDK and LSC will be important if changing anything other than metadata.
• Chromium's GN SDK may also need to be updated if the schema is changed.

Does Chromium use the GN SDK generated by scripts/sdk/gn/generate.py in the Fuchsia tree, or does it have its own generator?

[John Bauman]

That could make sense; I'd still like for there to be support for binaries (so selecting the right artifact based on the target platform) and multiple files per atom (so clients don't need to know the exact structure of the files that need to be included), but renaming this whole concept away from "loadable_module" would work.

On Wed, Mar 6, 2024 at 4:25 PM Chase Latta <chase...@google.com> wrote:

Would it make more sense for us to add a new IDK type for an artifact that is basically a type erased resource that can be put into a package? IIRC, the vulkan layers contain both json files and shared objects but all the end user needs to care about is where to install them. If we create this new type we don't need to worry about breaking down stream users and we get the benefit of being able to distribute these modules in a way that doesn't make assumptions about how they should be used.

[Chase Latta]

Would it make more sense for us to add a new IDK type for an artifact that is basically a type erased resource that can be put into a package? IIRC, the vulkan layers contain both json files and shared objects but all the end user needs to care about is where to install them. If we create this new type we don't need to worry about breaking down stream users and we get the benefit of being able to distribute these modules in a way that doesn't make assumptions about how they should be used.

On Wed, Mar 6, 2024 at 1:05 PM David Dorwin <ddo...@google.com> wrote:

[David Dorwin]

On Wed, Mar 6, 2024 at 1:24 PM John Bauman <jba...@google.com> wrote:

On Wed, Mar 6, 2024 at 4:05 PM David Dorwin <ddo...@google.com> wrote:

Some related observations:

• Unlike static and shared libraries, there is not a GN template (e.g., //build/cpp/sdk_shared_library.gni) for loadable modules.
• Instead, //src/lib/vulkan/BUILD.gn does lower level work than possibly any other SDK component.
• Even though there is currently only one instance, it may be better to have everything in one place.
• Even with those templates, the IDK build is currently fragile with respect to naming and the checks are insufficient (https://fxbug.dev/323441662), so checking the resulting IDK and LSC will be important if changing anything other than metadata.
• Chromium's GN SDK may also need to be updated if the schema is changed.

Does Chromium use the GN SDK generated by scripts/sdk/gn/generate.py in the Fuchsia tree, or does it have its own generator?

Last I heard:

• The GN SDK in fuchsia.git is deprecated but can't yet be deleted. See https://fxbug.dev/42080529 and its dependencies.
• Chromium forked the fuchsia.git version and is maintaining it. In the future, this will be the only GN SDK
• Chromium-based repos are also using Chromium's version (via DEPS I believe).

[Puneetha Ramachandra]

As for devicetree, there are a bunch of visitor shared objects which need to be placed under `lib/visitors/` in the driver package. So yes, updating/adding a schema to specify path in package would be useful.

[John Bauman]

Ok, I've put up https://fuchsia-review.git.corp.google.com/c/fuchsia/+/1003732 for review, which adds a new "resource" type (name is, of course, subject to change) that has destinations and might be generic beyond "loadable module" usage.

[Hunter Freyer]

Replied on the CL, but I think I'd prefer something more specific. "loadable_modules" was very difficult to understand because it was so generic, when actually there was only one use of it, which was quite specific, and SDKs made use of that knowledge.

Can we just call it `vulkan_layer` or something?

[Chase Latta]

On Fri, Mar 8, 2024 at 9:38 AM Hunter Freyer <hjfr...@google.com> wrote:

Replied on the CL, but I think I'd prefer something more specific. "loadable_modules" was very difficult to understand because it was so generic, when actually there was only one use of it, which was quite specific, and SDKs made use of that knowledge.

Can we just call it `vulkan_layer` or something?

I don't think we should call this `vulkan_layer` because we need something exactly like this for devicetree visitors. Each time we add a new type it adds work for each of our SDKs so if we are going to introduce a new type that describes how to put bundle something in a package that needs to be installed at a specific location lets just do that. 

[David Dorwin]

What are the next steps here? loadable_modules is one of the remaining exceptions to Platform Versioning in the SDK, and it doesn't make sense to work on that until we've settled on a path forward here. For example, if we go in another direction, maybe we can just remove loadable_modules.