Realm-scoped resolvers

[Justin Mattson]

A recent discussion hit on thoughts I've been having about whether it would be beneficial to have what I'll call "realm-scoped resolvers". The idea is that these resolvers would use a package index available in the Realm itself. The benefit of such a thing is that we could then have Realms that include certain packages at certain versions. This resolves some tension we have, particularly in testing, about whether to depend on package URLs or bundle components inside the test packages themselves.

If something like a test can express a dependency on another package, but not include the dependency's components and other files in its own package then we maintain a separate package-internal namespace, eliminating naming conflicts. It also means that both the test package and dependency package can carry their own, appropriate, versions of libraries or other things.

This idea also provides some interesting optimization opportunities. If a Realm carried this index and we could read it up front we could make choice about pre-fetching things. One could imagine, for example, prefetching the package meta FARs so that we'd be more quickly able to resolve manifests.

I am eager for feedback on this idea.

Cheers,

Justin

[Aaron Wood]

This seems pretty similar to what I proposed here:  fxr/557361.

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "component-framework-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to component-framewo...@fuchsia.dev.
To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CA%2BX8dX86B%3D-q%3DLyiOQVQfgxBANCxwAOkXUQow1v4uaGLNTHQpw%40mail.gmail.com.

[Justin Mattson]

I agree, that seems like an implementation that fits the fuzzy shape I'm describing.

[Hunter Freyer]

On Tue, Sep 7, 2021 at 2:33 PM 'Justin Mattson' via component-framework-dev <component-f...@fuchsia.dev> wrote:

I agree, that seems like an implementation that fits the fuzzy shape I'm describing.

On Tue, Sep 7, 2021 at 11:29 AM Aaron Wood <aaro...@google.com> wrote:

This seems pretty similar to what I proposed here:  fxr/557361.

On Tue, Sep 7, 2021 at 10:35 AM 'Justin Mattson' via component-framework-dev <component-f...@fuchsia.dev> wrote:

A recent discussion hit on thoughts I've been having about whether it would be beneficial to have what I'll call "realm-scoped resolvers".

This already exists, doesn't it? Rather than "realm-scoped" it's environment-scoped, but I think this use case is exactly what environments were meant for.

 

The idea is that these resolvers would use a package index available in the Realm itself. The benefit of such a thing is that we could then have Realms that include certain packages at certain versions. This resolves some tension we have, particularly in testing, about whether to depend on package URLs or bundle components inside the test packages themselves. 

 

If something like a test can express a dependency on another package, but not include the dependency's components and other files in its own package then we maintain a separate package-internal namespace, eliminating naming conflicts. It also means that both the test package and dependency package can carry their own, appropriate, versions of libraries or other things.

This idea also provides some interesting optimization opportunities. If a Realm carried this index and we could read it up front we could make choice about pre-fetching things. One could imagine, for example, prefetching the package meta FARs so that we'd be more quickly able to resolve manifests.

I am eager for feedback on this idea.

Cheers,

Justin

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "component-framework-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to component-framewo...@fuchsia.dev.
To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CA%2BX8dX86B%3D-q%3DLyiOQVQfgxBANCxwAOkXUQow1v4uaGLNTHQpw%40mail.gmail.com.

--
All posts must follow the Fuchsia Code of Conduct https://fuchsia.dev/fuchsia-src/CODE_OF_CONDUCT or may be removed.
---
You received this message because you are subscribed to the Google Groups "component-framework-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to component-framewo...@fuchsia.dev.

To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CA%2BX8dX9LUoNr9hTovadBv%2Bzbpvgs%3DA4NH7Z%2B8L6eqkKNe3NAog%40mail.gmail.com.

[Justin Mattson]

On Tue, Sep 7, 2021 at 12:36 PM Hunter Freyer <hjfr...@google.com> wrote:

On Tue, Sep 7, 2021 at 2:33 PM 'Justin Mattson' via component-framework-dev <component-f...@fuchsia.dev> wrote:

I agree, that seems like an implementation that fits the fuzzy shape I'm describing.

On Tue, Sep 7, 2021 at 11:29 AM Aaron Wood <aaro...@google.com> wrote:

This seems pretty similar to what I proposed here:  fxr/557361.

On Tue, Sep 7, 2021 at 10:35 AM 'Justin Mattson' via component-framework-dev <component-f...@fuchsia.dev> wrote:

A recent discussion hit on thoughts I've been having about whether it would be beneficial to have what I'll call "realm-scoped resolvers".

This already exists, doesn't it? Rather than "realm-scoped" it's environment-scoped, but I think this use case is exactly what environments were meant for.

I would say yes, but also no. Resolvers exist and are passed via environments, it is true. However, we currently don't have a reasonable way to construct and provider a resolver that meets the idea I'm trying to express. Specifically around the use case of constructing a realm and knowing at that time what packages you would like to be available in that realm at particular versions and creating a resolver that can resolve exactly those.

This type of resolver would be in contrast to general-purpose resolvers, typically meant to resolve any number of packages, typically at the most recent version.

[Hunter Freyer]

On Tue, Sep 7, 2021 at 4:05 PM Justin Mattson <jm...@google.com> wrote:

On Tue, Sep 7, 2021 at 12:36 PM Hunter Freyer <hjfr...@google.com> wrote:

On Tue, Sep 7, 2021 at 2:33 PM 'Justin Mattson' via component-framework-dev <component-f...@fuchsia.dev> wrote:

I agree, that seems like an implementation that fits the fuzzy shape I'm describing.

On Tue, Sep 7, 2021 at 11:29 AM Aaron Wood <aaro...@google.com> wrote:

This seems pretty similar to what I proposed here:  fxr/557361.

On Tue, Sep 7, 2021 at 10:35 AM 'Justin Mattson' via component-framework-dev <component-f...@fuchsia.dev> wrote:

A recent discussion hit on thoughts I've been having about whether it would be beneficial to have what I'll call "realm-scoped resolvers".

This already exists, doesn't it? Rather than "realm-scoped" it's environment-scoped, but I think this use case is exactly what environments were meant for.

I would say yes, but also no. Resolvers exist and are passed via environments, it is true. However, we currently don't have a reasonable way to construct and provider a resolver that meets the idea I'm trying to express. Specifically around the use case of constructing a realm and knowing at that time what packages you would like to be available in that realm at particular versions and creating a resolver that can resolve exactly those.

Agreed, we certainly don't have workflows or documented design patterns for this, and perhaps we should change that.

A thought that has come up in multiple conversations: things that "feel global" should really be "per-environment". For instance, we've implemented the static package index in order to do version locking globally, and I think what you're describing is essentially a "per-environment static package index".

Other candidates in this space: config overrides, component update policy, trusted software certs, allowlists, diagnostics, debugger attachment. 

 

This type of resolver would be in contrast to general-purpose resolvers, typically meant to resolve any number of packages, typically at the most recent version.

Amusingly, the "general-purpose resolvers" you describe are really only used in the dev flows. The way we do it in prod is more like what you describe above. Perhaps that's a sign that resolvers should behave more like what you're describing, and less like how they are today.

Hunter

[Aaron Wood]

I think this, and the configurable authorities idea that I linked are complimentary:

This is about when to use a separate, configurable, resolver.
That idea is about how to build the configurable resolver itself.

And the "global" resolver behaviors are, perhaps, just the "initial"/"root" ones, and other environments can override as needed.

[Justin Mattson]

 Perhaps that's a sign that resolvers should behave more like what you're describing, and less like how they are today.

It depends on the goals. In one vision of composable components where package A depends on package B to provide something we might imagine that A wants to use B an exact version because that is the version A tested with and the thing it trusts. In another vision we might imagine that A trusts B a lot and maybe the authors of A and B coordinate on release and testing and so at runtime A just wants to take the most recent version of B available.

For things that implement "the platform" or the "operating system" I do imagine we want all the components at a certain version because we test the system as a collection of components at a certain version and the stakes for misbehavior due to misalignment are high.

Cheers,

Justin 

[Gary Bressler]

Are there problems this feature could solve that package-relative components couldn't? If you're talking about today, I think the answer is 'yes':

• There's no way to publish a package or component out of tree. OOT tests have to rely on absolute URLs. We could use such a feature to restrict what package URLs a test may include.
• Packages are not partitioned by component. So we could run into the problem you mentioned where if two components depend on a different version of the same library, we can't package them together (unless we name those libraries differently, which requires some coordination).

However, when looking at the future, package-relative components may become more useful. We've discussed the idea of partitioning packages by component many times (ex, see bryanhenry's RFC here), which would solve the resource conflict issue. Regarding making packages/components publishable in the SDK, crjohns is kicking off a project there.

To view this discussion on the web visit https://groups.google.com/a/fuchsia.dev/d/msgid/component-framework-dev/CA%2BX8dX917fj%3DtG5O6Bfs2r_1v%2BAKdi--gx5XYz-%2BQ2MZx_c1NQ%40mail.gmail.com.