Re: [RFC PATCH] xen/arm: Rebranding dom0less feature

[Rich Persaud]

Hi Luca,

On Jun 30, 2023, at 05:12, Luca Fancellu <luca.f...@arm.com> wrote:

The "dom0less" feature was intended to be the feature where a domU
domain could be launched without the control domain (Dom0)
intervention, however the name seems to suggest that Dom0 cannot
be part of the configuration, while instead it's a possible use case.

Thanks for your interest in Xen boot integrity. Please see the 2018 domB RFC:

https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01306.html

At Xen Summit 2018 (Nanjing) and Xen Summit 2019 (Chicago), OpenXT contributors made a case to Xen-on-Arm contributors for the architectural unification of incumbent dom0less (Arm) and the domB (x86) proposal for improving Xen boot integrity.

To avoid that, rename the "dom0less" configuration with the name
"hyperlaunch", that is less misleading.

2018-2022 work on Xen launch integrity, thanks to Apertus and Star Lab: 

https://wiki.xenproject.org/wiki/Hyperlaunch

https://www.theregister.com/2022/12/16/xen_4_17_hyperlaunch/

2023 Hyperlaunch design session last week, thanks to Apertus and AMD:

https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01870.html

Signed-off-by: Luca Fancellu <luca.f...@arm.com>

If Arm is now ready to invest engineering resources into new Xen launch integrity features for security and safety-critical use cases, that is exciting news, 5 years into the on-again-off-again bootstrapped Hyperlaunch project! The roadmap would benefit from new funding.

Would you like to attend the next Xen working group call for Hyperlaunch?

Rich

[Christopher Clark]

+CC openxt, Jason, Marek

On Fri, Jul 7, 2023 at 2:06 PM Christopher Clark <christoph...@gmail.com> wrote:

+CC members of the Hyperlaunch Working Group + participants on earlier Hyperlaunch threads

On Thu, Jul 6, 2023 at 2:39 PM Stefano Stabellini <stefano.s...@amd.com> wrote:

On Thu, 6 Jul 2023, George Dunlap wrote:
> On Wed, Jul 5, 2023 at 11:14 PM Stefano Stabellini <stefano.s...@amd.com> wrote:
>       On Wed, 5 Jul 2023, George Dunlap wrote:
>       > On Mon, Jul 3, 2023 at 9:55 PM P S <pair...@gmail.com> wrote:
>       >       > On Jul 3, 2023, at 15:45, Luca Fancellu <luca.f...@arm.com> wrote:
>       >       >
>       >       >> On 3 Jul 2023, at 18:48, Stefano Stabellini <sstab...@kernel.org> wrote:
>       >       >>
>       >       >>> On Mon, 3 Jul 2023, Daniel P. Smith wrote:
>       >       >>> On 7/1/23 11:13, Luca Fancellu wrote:
>       >       >>>>> On 1 Jul 2023, at 08:53, Andrew Cooper <andrew....@citrix.com> wrote:

>       >       >>>>>
>       >       >>>>> On 30/06/2023 10:12 am, Luca Fancellu wrote:
>       >       >>>>>> The "dom0less" feature was intended to be the feature where a domU
>       >       >>>>>> domain could be launched without the control domain (Dom0)
>       >       >>>>>> intervention, however the name seems to suggest that Dom0 cannot
>       >       >>>>>> be part of the configuration, while instead it's a possible use case.
>       >       >>>>>>

>       >       >>>>>> To avoid that, rename the "dom0less" configuration with the name
>       >       >>>>>> "hyperlaunch", that is less misleading.
>       >       >>>>>>

>       >       >>>>>> Signed-off-by: Luca Fancellu <luca.f...@arm.com>
>       >       >>>>>> ---
>       >       >>>>>> This is an RFC to get the feeling of the community about the name
>       >       >>>>>> change, for now it's everything in one patch just to see how it
>       >       >>>>>> will look like, if there is interest on proceeding into it, I can
>       >       >>>>>> split in more commit.
>       >       >>>>>
>       >       >>>>> Have you discussed this with Dan and Chris at all?  You haven't even
>       >       >>>>> CC'd them.
>       >       >>>>
>       >       >>>> No, this rename idea started from a chat during the summit, anyway Julien
>       >       >>>> promptly add them to the CC, because I forgot.
>       >       >>>
>       >       >>> No worries and thank you for considering and taking the time to do this RFC.
>       >       >>> It is greatly appreciated that there is a strong willingness to have dom0less
>       >       >>> and hyperlaunch merged.
>       >       >>>
>       >       >>>>>
>       >       >>>>> While there is a lot of end-goal in common between the dom0less and
>       >       >>>>> hyperlaunch, and that the name dom0less is deeply misleading,
>       >       >>>>> hyperlaunch is specifically not this.
>       >       >>>>
>       >       >>>> Yes Hyperlaunch is more than this, however as I said, with this RFC I would
>       >       >>>> like
>       >       >>>> to ear opinions, @Daniel @Christopher could it be a proper name for the
>       >       >>>> dom0less
>       >       >>>> feature?
>       >       >>>
>       >       >>> As Andy has alluded, hyperlaunch is meant to provide a flexible means to
>       >       >>> handle domain construction at boot to meet a wide range of possible use cases.
>       >       >>> One of those use cases is dom0less, so yes, ultimately what dom0less does
>       >       >>> today will be achievable under hyperlaunch. Our intended approach to align the
>       >       >>> two implementations is one that is meant to be minimally disruptive, since
>       >       >>> dom0less is considered a supported (SUPPORT.md) capability. As mentioned, we
>       >       >>> are greatly appreciative to the openness to adopt the name,
>       >       >>
>       >       >> Thanks Daniel!
>       >       >>
>       >       >>
>       >       >>> but a big concern
>       >       >>> I personally have is the confusion it could cause a general user. A blanket
>       >       >>> rename would end up with two documents in the docs tree that provide two
>       >       >>> different explanations of hyperlaunch and two different device tree
>       >       >>> definitions. So I think a more measured approach should be considered here.
>       >       >>>
>       >       >>>> If this patch makes things more difficult for the Hyperlunch serie, I’m ok
>       >       >>>> to drop it,
>       >       >>>> my only aim was just to find a less misleading name for the feature.
>       >       >>>
>       >       >>> What I would like to suggest as a good first step would be an update to the
>       >       >>> dom0less document. Provide a note at the beginning that points to the
>       >       >>> hyperlaunch design doc as a more general approach that will eventually subsume
>       >       >>> dom0less. This would provide a gentler transition for exist users of dom0less.
>       >       >>>
>       >       >>> If it is not too much, I would also ask, please have a look at the design for
>       >       >>> boot modules in the series Christopher just posted. The design pulls from the
>       >       >>> work done by dom0less and expanded upon it. I major step into merging the two
>       >       >>> capabilities will be to have a common set of structures. Once those are in
>       >       >>> place, we can move to a common device tree representation, and at that point
>       >       >>> we would be fairly close, if not at the point of a formal merger of between
>       >       >>> the two.
>       >       >>
>       >       >> At the moment we have a concrete problem with explaining dom0less and
>       >       >> hyperlaunch to potential new users. Using two different names for a
>       >       >> similar feature on arm and x86 causes confusion. It is hurting Xen as a
>       >       >> solution. Personally I already had to switch to use the word
>       >       >> "hyperlaunch" for everything in my users-facing presentations.
>       >       >>
>       >       >> At the summit, we discussed that it would be a good idea to use a single
>       >       >> name to refer to both features on arm and x86. Given that "dom0less"
>       >       >> causes additional issues because it makes people think that there is no
>       >       >> Dom0, the suggestion was to use "hyperlaunch" to refer to both features.
>       >       >>
>       >       >> We don't need to 100% align the two implementations and data structures.
>       >       >> This is not for engineers that are going to look at the specifications
>       >       >> and improve them. This is for users/customers of Xen that are trying to
>       >       >> understand what the hypervisor enables them to do. We need to be able to
>       >       >> show users architecture slides with the same name and explanation on
>       >       >> both ARM and x86.
>       >       >>
>       >       >> I am sure that Daniel and Christopher remember, but for the others on
>       >       >> this email thread, the name "hyperlaunch" was born exactly to be that:
>       >       >> the one name to cover both features on ARM and x86 even if they have a
>       >       >> different implementation. Appended an old email for reference.
>       >       >>
>       >       >> Also I agree with Daniel that we need to be careful about the two docs
>       >       >> under docs/. I think he is right we need to add a paragraph explaining
>       >       >> the history and a pointer to the other document. Something like:
>       >       >>
>       >       >> "Dom0less is the name that was used when initially introducing the
>       >       >> feature on ARM. Then, the "dom0less" name was retired in favor of
>       >       >> "hyperlaunch" to avoid confusion (a Dom0 might still be present) and to
>       >       >> align with x86 (where a similar feature was called hyperlaunch from the
>       >       >> start)."
>       >       >
>       >       > I’m fully ok to add a section like this pointing to the Hyperlaunch design.
>       >
>       >       _If_ this text is added, please include links/references to the Hyperlaunch wiki page and Hyperlaunch design docs.
>       >
>       >       > @Daniel and @Christopher would it be ok for you or the changes in the serie
>       >       > are going to be problematic for your future work? In the end it’s just a mechanical
>       >       > rename, so I guess we just need to agree on naming conventions.
>       >
>       >       Please see the history of trademark litigation about the use of symbolic names to reference similar-but-different
>       artifacts. 
>       >       It is much easier to use the same name to refer to entirely different objects. Historically, confusion arises when a
>       name is
>       >       used in similar contexts.
>       >
>       >       There is also versioning.  Could we refer to dom0less as "Hyperlaunch Version -1"?
>       >
>       >       How about renaming dom0less to "Hyperlaunch Lite"?
>       >
>       >
>       > Perhaps it would be helpful if you could explain more clearly your concerns.  I take it that you want a name which can be
>       used specifically
>       > to indicate the full "domB measured boot" functionality that was Daniel and Christopher's original goal, and that you're
>       afraid that using
>       > plain "Hyperlaunch" for only the "start VMs from Xen on boot" functionality will dilute that?
>       >
>       > The "start VMs from Xen on boot" functionality is the *only* thing that a big chunk of the users of this functionality want; 
>       referring to
>       > it as "Hyperlaunch Lite" or "Hyperlaunch -1" will undermine the value of the functionality.
>       >
>       > What if we use "Measured Hyperlaunch", or "Hyperlaunch Measured Boot" to refer to the full measured boot functionality?
>
>       I think this is the best way.
>
>
>       > Or, "Hyperlaunch DT" for "Booting VMs from Xen using Device Tree" (without the involvement of a domB), "Hyperlaunch Boot
>       Domain /
>       > Hyperlaunch domB" for a more general "domB" functionality, and "Hyperlaunch Measured Boot" for the full functionality
>       (assuming there's
>       > more to this than simply having a domB involved)?
>
>
>       We need an overarching name to cover the feature "start VMs from Xen on
>       boot" on both ARM and x86. From my understanding and from the original
>       emails on the subject, the name "hyperlaunch" was it.
>
>
> Sure; but think "guitar" vs "acoustic guitar" vs "electric guitar".  "Electric guitar" is new, "guitar" covers them both, but you sometimes
> need a way to specify "acoustic".  Right now target configurations we're talking about include:
>
> 1. Booting all your domains directly from Xen using DT configurations
> 2. Booting a domB, which then executes some more complicated programmatic configuration to launch VMs before disappearing
> 3. Doing full measured boot on the whole system using a domB.
>
> If "Hyperlaunch" means 1-3, we not only need a way to specify that you're talking about 3, but *also* a way to specify that you're talking
> about 1.  In the vast majority of cases for the foreseeable future are going to be 1.  Additionally, we want to make sure that
> "Hyperlaunch" *actually* turns out to mean 1-3, and not just 1.
>
> The thing I like about "Hyperlaunch DT" is that to me it sounds pretty cool but also is very descriptive: I haven't talked to people
> building these systems, but it seems like saying, "The hypervisor launches VMs based on a Device Tree passed to it at boot" will be
> immediately understood, and stick in people's minds.
>
> So maybe informally, or in "short usage" use "Hyperlaunch", but in documentation or reference systems, when talking specifically about #1,
> try to use "Hyperlaunch DT", just to reinforce the idea that there's more to Hyperlaunch that's coming down the road?

"Hyperlaunch DT" would refer to both the x86 and ARM implementations
because they are both based on DT.

I think it is better than "Hyperlaunch Lite" but I am not a fan of
either of these two names because "DT" and "Lite" get easily lost in
presentations and discussions. For the next few years many will talk
about HyperLaunch just to refer to what is Dom0less today. So if the
goal when we say "HyperLaunch" is to bring Measure Boot or XSM to
people's minds, I don't think this will work well.

If we want to keep "Hyperlaunch" to mean 1-3 above, highlighting Measure
Boot or XSM, then I think we should consider using "Dom0less" for 1.
Yes, it is misleding, but at least it is unique, and a google search for
"dom0less" leads the user to the right results.

If we do that, we should rename "Hyperlaunch" with "Dom0less" in
docs/designs/launch/hyperlaunch.rst. That's because "Hyperlaunch" is
defined as "the ability of a hypervisor to construct and start one or
more virtual machines at system launch in a specific way", which falls
under Dom0less in this discussion.

In my opinion, it is better to have specific names for specific
features. So I would use "HyperLaunch" to mean "the ability of a
hypervisor to construct and start one or more virtual machines at system
launch in a specific way" as it is defined today.

When we add Measure Boot or XSM or other security/safety features, I
would call them out specifically. For instance, "HyperLaunch with XSM".
It is more descriptive and highlights each feature.

Note that at AMD we'll need HyperLaunch without an all-powerful Dom0,
probably with XSM. So I am not writing this because I don't think the
other features 2-3 are not important. They definitely are important.

Thanks to all the participants in this thread for the interest in Hyperlaunch and
the support for enabling common advanced boot functionality for Xen across
architectures.

I'm aiming to provide here a hopefully-fairly-objective overview of the issues
being raised so that we can ensure that these are covered, and then I will also
give my views afterwards.

------------------------------------------------------------
= Naming and communication

- Ensuring expectations are set correctly for the Hyperlaunch name
    - communicating the value of it, differentiation for Xen
    - avoiding sub-branding it for feature subsets, use cases, technologies

- Retiring the term 'dom0less'

- How to describe the "starting multiple VMs at host boot" functionality

    - How to describe further Hyperlaunch functionality beyond this
        - eg. isolation properties and relevance to critical systems
        - eg. running without a classic dom0

    - How Hyperlaunch relates to other boot functionalities and technologies,
      including:
        - architecture-specific aspects and architecture-neutral aspects
        - Device Tree
        - boot domain
        - control domain, hardware domain, dom0
        - domain builder
        - system measurement
        - XSM
        - DRTM
        - etc.


= Migration

- Providing a forward path for existing users of dom0less functionality
across the technical changes for Hyperlaunch cross-architecture implementation
    - document compatibility
    - support a "dom0less mode" with existing Device Tree structure
    - documentation updates to be paired with progress on code implementation


= Community resourcing

- Supporting code review and merge of Hyperlaunch changes into Xen
    - transitioning existing Arm logic into common, including FDT

- Provision of accurate, consistent documentation materials to support
effective communication to existing and prospective users, developers and other
stakeholders
     - Ensuring that the document structure supports ongoing maintenance:
        - Multiple use cases, structure docs accordingly: eg.
            - use case: static partitioning, critical + non-critical VMs
            - use case: measured launch with a boot domain
            - use case: fast VM start for embedded systems
        - Architecture of Hyperlaunch, relevance to other hypervisors
            - nested systems

- Design, review and agreement for common cross-architecture and
  arch-extensible interfaces, including:
    - common boot data structures
    - Device Tree structure
    - hypfs entries
    - introspection to determine hyperlaunched system status

- Development of test cases

- CI of Hyperlaunch interfaces, to ensure that it stays working
 
------------------------------------------------------------

Views arrived at in discussion with Rich and Daniel, and after reading the
thread contributions, follow this point.

"Hyperlaunch" is to be the common name across architectures for a flexible,
configurable boot system for coordinating the launch of multiple VMs by the
hypervisor, with a common implementation, pooling resources and development
risk across the Xen community. The interfaces are core to it.

From the Hyperlaunch design document (reviewed + committed to the tree):

"The design enables seamless transition for existing systems that require a dom0, and provides a new general capability to build and launch alternative configurations of virtual machines, including support for static partitioning and accelerated start of VMs during host boot, while adhering to the principles of least privilege. It incorporates the existing dom0less functionality, extended to fold in the new developments from the Hyperlaunch project, with support for both x86 and Arm platform architectures, building upon and replacing the earlier 'late hardware domain' feature for disaggregation of dom0.

Hyperlaunch is designed to be flexible and reusable across multiple use cases, and our aim is to ensure that it is capable, widely exercised, comprehensively tested, and well understood by the Xen community."

https://xenbits.xen.org/docs/4.17-testing/designs/launch/hyperlaunch.html

ie.  Hyperlaunch was created to move away from point solutions that hard-code
specific launch configurations, isolation properties and threat models. It
isn't just about starting domains -- it is about enabling the construction of
complex use cases runtimes for Xen. It is the result of iterative design
starting with the disaggregation for the late hardware domain, through
dom0less development and then with the comprehensive hyperlaunch design and
implementation that builds upon them both.

The current interest and investment in Hyperlaunch is driven by its relevance
to Safety Critical systems due to the isolation properties and improvement
in the ability to certify the software -- this is closely related to but
slightly different from starting multiple VMs at host boot.

To encourage commonality and allow for future development, we should not
use architecture-specific or vendor-specific name variations, and also avoid
technology-specific name variations (eg. Device Tree or "DT").

Instead, the use case configurations should themselves be describable.

Thanks again,

Christopher

 

[Rich Persaud]

On Jul 7, 2023, at 21:17, Stefano Stabellini <stefano.s...@amd.com> wrote:

> Thanks Christopher, Daniel and all!
>
> So if I understand correctly, you are in favor if renaming Dom0less to
> Hyperlaunch throughout the Xen codebase? And we need a clarification of
> the docs/, especially docs/features/dom0less.pandoc?

Christopher wrote:
>> = Community resourcing

Note the pre-requisite work items for upstream Xen, listed under "Community Resourcing", to merge code for Hyperlaunch common interfaces and test cases, with docs on configuration of Hyperlaunch to deliver functionality for dom0less use cases.

Thanks,
Rich

[Rich Persaud]

On Jul 8, 2023, at 03:29, Luca Fancellu <luca.f...@arm.com> wrote:
> 

>>>>
>>>> Instead, the use case configurations should themselves be describable.
>>>
>>> Thanks Christopher, Daniel and all!
>>>
>>> So if I understand correctly, you are in favor if renaming Dom0less to
>>> Hyperlaunch throughout the Xen codebase? And we need a clarification of
>>> the docs/, especially docs/features/dom0less.pandoc?
>>
>> Christopher wrote:
>>>> = Community resourcing
>>
>> Note the pre-requisite work items for upstream Xen, listed under "Community Resourcing", to merge code for Hyperlaunch common interfaces and test cases, with docs on configuration of Hyperlaunch to deliver functionality for dom0less use cases.
>

> Are you saying that before renaming the “dom0less” feature, we should wait for it to be ported to the common code?

Why "wait"? In what timeframe do you expect dom0less to use Hyperlaunch code?

Can kernel component foo adopt the name of kernel component bar without code change?

Can dom0less stakeholders derive Hyperlaunch benefits without using Hyperlaunch code?

Rich


P.S. An intellectual property story from Cambridge, UK
https://en.wikipedia.org/wiki/Leibniz–Newton_calculus_controversy
https://www.smithsonianmag.com/history/ten-famous-intellectual-property-disputes-18521880/

"By the early 18th century, many credited the German mathematician and philosopher Gottfried Wilhelm Leibniz with inventing the study of calculus. Leibniz had, after all, been the first to publish papers on the topic in 1684 and 1686. But when Englishman Isaac Newton published a book called Opticks in 1704, in which he asserted himself as the father of calculus, a debate arose. Each of the thinkers’ respective countries wanted to stake a claim in what was one of the biggest advances in mathematics."

[Daniel P. Smith]

On 7/8/23 14:08, Stefano Stabellini wrote:

> On Sat, 8 Jul 2023, Rich Persaud wrote:
>> On Jul 8, 2023, at 03:29, Luca Fancellu <luca.f...@arm.com> wrote:
>>> 
>>>>>>
>>>>>> Instead, the use case configurations should themselves be
describable.
>>>>>
>>>>> Thanks Christopher, Daniel and all!
>>>>>
>>>>> So if I understand correctly, you are in favor if renaming
Dom0less to
>>>>> Hyperlaunch throughout the Xen codebase? And we need a
clarification of
>>>>> the docs/, especially docs/features/dom0less.pandoc?
>>>>
>>>> Christopher wrote:
>>>>>> = Community resourcing
>>>>
>>>> Note the pre-requisite work items for upstream Xen, listed under
"Community Resourcing", to merge code for Hyperlaunch common interfaces
and test cases, with docs on configuration of Hyperlaunch to deliver
functionality for dom0less use cases.
>>>
>>> Are you saying that before renaming the “dom0less” feature, we
should wait for it to be ported to the common code?
>>
>> Why "wait"? In what timeframe do you expect dom0less to use
Hyperlaunch code?
>>
>> Can kernel component foo adopt the name of kernel component bar
without code change?
>>
>> Can dom0less stakeholders derive Hyperlaunch benefits without using
Hyperlaunch code?
>
>

> I think Rich is saying that before using the same name we should make
> sure that the interfaces and features are actually comparable and maybe
> even "compatible". I think that is very reasonable. Rich, did I
> understand correctly?

Essentially, yes this is what is being sought here. This does not mean
that the full capability has to be present for the adoption of the
common name, but that it can be accomplished through a path of integration.

> The Hyperlaunch (x86) code is not yet upstream, but the design document
> that describes the device tree interface shows an interface that is very
> similar, almost compatible, with today's dom0less (ARM) device tree
> interface.

I would caution the use of the current in-tree document as it is today,
it was posted under the design folder as it was only the design and not
burdened with the realities of implementation. Along the path of v1
implementation, recent PVH expansion, and roles update, each have
required updates to the design which are not yet included in the in-tree
docs.

To address, this the plan below starts with the documentation patch
posted in v1 of the hyperlaunch series:


https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00353.html

and will contain updates for community feedback received:


https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg01015.html

and development work that has been done for PVH and roles.

> The structure of the device tree information is the same. Going through
> it I could only spot only tiny differences:
> - top level node is "hypervisor" instead of "chosen"
> - "module-addr" instead of "reg"
> - "module,kernel" instead of "multiboot,kernel"
> - "module,ramdisk" instead of "multiboot,ramdisk"
>
> The rest is the same. If we sort out these small differences one way or
> the other then the resulting interface should actually be fully
> compatible and we could reuse the existing Dom0less (ARM) code to parse
> an HyperLaunch (x86) configuration.
>
> The top level node is not a problem. We could easily deal with both
> "hypervisor" and also "chosen". Or we could pick a third different name
> for both: "domains" which is the one used by System Device Tree.
>
> I think we should rename "module-addr" to "reg" in the hyperlaunch
> design document. I don't think it would have any effect on the existing
> hyperlaunch (x86) code and usage because direct addresses are typically
> not used on x86.
>
> "module,kernel" and "module,ramdisk": we could either get rid of them in
> favor of "multiboot,kernel" and "multiboot,ramdisk", or we could add
> "module,kernel" and "module,ramdisk" as alternative aliases in the
> existing dom0less (ARM) code. We already have "xen,linux-zimage" and
> "xen,linux-initrd" as aliases so it is not a problem.
>
>
> Also, I do think that Dom0less stakeholders would benefit from
> Hyperlaunch code such as Dom0's reduction of privilege. Things like
> "permissions" and "functions" of the Hyperlauch device tree interface
> design document.

The roles work takes that to its final conclusion, from which everyone
will benefit.

> So, my opinion is that we should go ahead with dom0less->hyperlaunch
> rename but we should also try to make the two device tree interfaces
> compatible, sorting out the small differences above. That would help a
> lot in terms of documentation and tooling. It would be ideal if things
> like ImageBuilder worked equally well for Hyperlaunch (x86) and Dom0less
> (ARM).

Let me build on the plan laid out by Christopher, that detailed arriving
at the completion for hyperlaunch, and provide a set of steps to arrive
at a new short-term milestone to a larger roadmap. The intent being to
arrive at the immediate desire of the community to see dom0less renamed.
For the longer term, as Christopher was alluding, there is still a
significant amount of work to be done to deliver one of the biggest
market differentiating capabilities for Xen in some time.

As Stefano has acknowledged, hyperlaunch is a concept that is beyond
domain construction and that is, or will be, embodied by a set of
interfaces and capabilities. Considering dom0less as it is implemented
today, does not meet the former and in spirit meets the latter.
Compounding this is that dom0less is a supported feature today, with its
own defined interface, and a user base that is using that interface, at
least to some degree AIUI.

We have a responsibility to the dom0less user base and future
hyperlaunch users with requirements based on hyperlaunch design docs and
presentations. As such, any action taken should be done so under a
larger roadmap of adding the complete hyperlaunch capability to Xen.

With the initial funding by AMD, the first milestone was to be moving
Xen on to a common representation of boot material provided to the
hypervisor. As a result of this discussion, I would like to put forth a
new nearer term milestone of incorporating dom0less under hyperlaunch.

The naming suggestions by the community are greatly appreciated, and I
do not want to seem dismissive of clear offers of help and assistance.
This area is something Christopher and I discussed at length during the
drafting of the hyperlaunch design. Something we arrived at is that
there is only a single top level feature, which is hyperlaunch. The
hyperlaunch feature itself will provide multiple means to configure how
a launch will occur, one could even consider them modes of operation.

Reviewing the design doc, you will see that an initial attempt to
categorize them was done, splitting them into a dynamic mode and static
mode. Under these modes were multiple use cases. With that, we did not
mean to limit the hyperlaunch modes to strictly those two, and thus
other modes are more than reasonable to add. As such, my suggestion
would be the introduction of the dom0less compatibility mode for
hyperlaunch. Its very definition is a mode of hyperlaunch that supports
the "legacy" dom0less configuration interface. This approach allows
dom0less to effectively become hyperlaunch, deconflicts the fact that
hyperlaunch proper has a different interface than dom0less, and provides
a clean roadmap for migration to existing dom0less users.

To provide a concrete, measurable set of steps to achieve the dom0less
merging milestone, I will lay out the approach as three patch series
that will be a collaboration by the community. Each will need to be
submitted, reviewed and merged into the tree, with the end being the
existing dom0less becoming hyperlaunch's dom0less compatibility mode.

== Patch Series 1 (Resourced by Apertus)

The goal of this series is to properly introduce the hyperlaunch
"feature" in to Xen. This series would be submitted by myself and will
consist of two patches derived from the original hyperlaunch v1 series.
The first patch is the docs patch that updates the hyperlaunch device
tree design to reflect review feedback and updates for PVH and roles,
mentioned above.

The second patch will start with the v1 series patch that moved the fdt
parsing helpers into common:
https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00352.html

The patch currently moves common FDT parsing to common/fdt.c, it will be
updated to add this path to DEVICE TREE in MAINTAINERS. As part of
updating MAINTAINERS, there will be the addition of HYPERLAUNCH, which
will own common/domain-builder/ and doc/design/launch paths. As such,
this will effectively be a declaration of the top level hyperlaunch
feature, with this as an interface, and establish the maintainers of the
feature.

== Patch Series 2 (Requesting resourcing by Arm)

The goal of this series would be to move the dom0less device tree
parsing under hyperlaunch. We would respectfully request a member of the
Arm community to volunteer to take ownership and steward the series
through submission and review process.

The implementation of this series would see dom0less device tree parsing
to use, and expand if necessary, the common/fdt parsing helpers. I
personally would see this logic move under common/domain-builder using a
file name that would not collide with the files from the hyperlaunch v1
series, e.g. a suggestion would be fdt-dom0less.c.

As for ownership of that file, I would suggest the addition of a
HYPERLAUNCH DOM0LESS COMPATIBILITY be added to MAINTAINERS with the
appropriate Arm community members, but understand and willing to
consider the position that it falls under HYPERLAUNCH. The purpose of
using HYPERLAUNCH DOM0LESS COMPATIBILITY will be to provide a means to
signal the retirement of dom0less compatibility mode at some future point.

== Patch Series 3 (Resourced by Apertus)

The goal of this series will be to formalize hyperlaunch dom0less
compatibility mode. The series would be submitted by myself and/or
Christopher. It will consist of documentation patches that will add a
doc/features/hyperlaunch.rst and a
doc/features/hyperlaunch/dom0less-compatiblilty.rst. The hyperlaunch.rst
path will fall under HYPERLAUNCH, while dom0less-compatiblilty.rst will
fall under HYPERLAUNCH DOM0LESS COMPATIBILITY. This will also see
DOM0LESS in SUPPORTED.md be renamed to HYPERLAUNCH.

In summary, I would like to reiterate points made by Rich and
Christopher. The motivation and concept of the hypervisor
differentiating capability that is hyperlaunch goes back to the 2012
domain builder work that was a companion to the hardware domain work.

Since then, there have been a few of us in the OpenXT community that
desired to make this an integral part of Xen. Internal OpenXT
discussions in 2017-2018 along with the announcement of dom0less
inspired confidence there were in fact other uses cases and interest in
Xen gaining such an integrated capability.

A fact that should not go overlooked or undervalued is that hyperlauch
would not exist today without the extremely generous support StarLab
provided to sponsor Christopher and I to do the R&D, proof of viability,
and a full working prototype that provided the basis for the v1 series.
This was not a minor investment on their part, and taking the design to
completion requires further substantial investment. Due to an
acquisition imposed shift in market focus, StarLab has since stepped away.

Fortunately, and for which we are immensely appreciative, AMD has
recently stepped up to fund some hyperlaunch work items. With a growing
number of hyperlaunch use cases, I would be happy to help those with
budgetary influence to communicate the business benefits of investment
in open work items, targeting specific launch configurations, safety and
security properties.

The amount of work to be done goes beyond just parallel domain
construction, there are tangential capabilities that need updating and
incorporation. For instance, a topic discussed during the summit, VPCI
and device assignment at/during hypervisor startup. Anyone looking or
willing to provide purely financial support, feel free to reach out
directly to Christopher and me, as we have a few avenues to enable the
flow of funds.

Attached are candidate work items for funding and resourcing of
hyperlaunch integration, building upon the proposed Apertus and Arm
patch series for dom0less rename and migration to hyperlaunch common
interfaces.

Lastly, thank you to everyone who has taken the time to engage and
collaborate on how to resolve the task at hand!

V/r,
Daniel P. Smith

===

Here are outlines for the next development items appropriate for funding
for progress on hyperlaunch integration, following on from the three
series described above.

== Work Item: initial x86 Hyperlaunch SUPPORT and launch of dom0
(Resourcing TBD)

The goal is to enable initial hyperlaunch SUPPORT on x86, building upon
from the work in the posted v3 series of hyperlaunch.

Development proceeds from patches 9-12 of the v1 hyperlaunch series, to
add boot with minimal construction of a classic dom0 from a device tree
configuration. The Hyperlaunch SUPPORT statement would add x86 and Xen
nightly CI and release-gating test configurations would be extended to
include hyperlaunch on x86.


== Work Item: Hyperlaunch XSM policy for guests (Resourcing TBD)

The goal is to ensure that security policies of guests started via
hyperlaunch are accurately aligned with the configured functions of each
guest, which would allow for reduction of privilege of the dom0 in
dom0less systems.

A new XSM policy with granular permissions for domain functions, aligned
with the roles work, would be integrated into the hypervisor. It would
define new domain security labels appropriate for assigning to domUs,
matching privileges for the domain functions that can be assigned with
hyperlaunch, and allowing for more expressive policy control than is
expressible with dom0less.


== Work Item: Hyperlaunch of Arm guests SUPPORT (Resourcing TBD)

The goal of this effort is to enable hyperlaunch of Arm guests.

It would enable use of the architecture-common Hyperlaunch Device Tree
format on Arm, providing the forward migration path from the Dom0less
format, to the new format with additional flexibility for guest
construction, including control over console privilege assignment and
XSM label control.

It would achieve the objective of using common cross-architecture boot
structures and community-maintained common code.

Development would proceed from patches 9-10 of the v1 hyperlaunch
series, to add boot with construction of dom0 and domU guests from a
hyperlaunch device tree configuration.

Xen nightly CI and release-gating test configurations would be extended
to include hyperlaunch with guests on Arm.


== Work Item: Hyperlaunch of x86 PV and PVH guests (Resourcing TBD)

The goal of this effort is to enable Hyperlaunch of x86 PV and PVH guests.

This builds upon the initial x86 Hyperlaunch support to add domain
construction of PV and PVH guests.

Xen nightly CI and release-gating test configurations would be extended
to include hyperlaunch with guests on x86.


== Work Item: XSM-enforced FuSA (Resourcing TBD)

Hyperlaunch enables disaggregation, and XSM enables granular policy.

The goal of this work is to design and implement new modular,
fine-grained policy to integrate into Xen for control for
safety-critical VMs and isolation from non-safety-critical VMs.


== Work Item: Design for Hyperlaunch of x86 HVM guests (Resourcing TBD)

The goal of this effort is to research, prototype and produce design
documentation for enabling hyperlaunch of x86 HVM guests, ie. with
support for a device emulator.

This would build upon the x86 Hyperlaunch support for PVH guests.

Investigation to consider:
- launch alongside a classic dom0 domain
- launch on a static partitioned system without a dom0
- stub domains for device emulator isolation
- boot domain integration
- device assignment


-- Thanks for your consideration!