Gramine on a microkernel OS

[Sid Hussmann]

Dear Gramine Developers!

I came across the exciting FOSDEM talk from Dmitrii Kavaiskii about the
current state of Gramine. We at Gapfruit developed a microkernel
operating system based on the Genode Framework [0].

I want to explore writing a Gramine backend for executing unmodified
binaries built for a POSIX/Linux system on our platform.

For a first PoC, I'm in the process of identifying the required steps to
run a simple "Hello world!" Linux binary on our OS using Gramine.

Our system does not rely on POSIX/Linux, but we can create POSIX
components linked to the libc ported from FreeBSD. The target
architecture to run Gramine is x86_64 and eventually arm_v8.

1. To run a Linux binary with Gramine on top of a new platform/OS, is
PAL the only Gramine component that I need to extend?

2. I don't fully understand the mechanism on how you intercept system
calls. I had a look at "Implementing new system call" from the wiki and
`LibOS/shim/src/arch/x86_64/syscallas.S`,
`gramine/LibOS/shim/src/arch/x86_64/shim_table.c`,
`LibOS/shim/src/arch/x86_64/*` but I seem to be missing stuff. Could you
please point me to source files or general concepts to adapt PAL to a
new TEE/OS?

3. I suppose it is possible to implement only a minimal set of PAL
functionalities for the "Hello world!"? What would they be?

Cheers,
Sid

[0] https://genode.org/

[Michał Kowalczyk]

Hey,

That's a nice use case for Gramine!

> 1. To run a Linux binary with Gramine on top of a new platform/OS, is
> PAL the only Gramine component that I need to extend?

In theory yes, but in practice you may find out that you need some
changes to the PAL API or LibOS also (e.g. if you find a bug).

> 2. I don't fully understand the mechanism on how you intercept system
> calls. I had a look at "Implementing new system call" from the wiki
> and `LibOS/shim/src/arch/x86_64/syscallas.S`,
> `gramine/LibOS/shim/src/arch/x86_64/shim_table.c`,
> `LibOS/shim/src/arch/x86_64/*` but I seem to be missing stuff. Could
> you please point me to source files or general concepts to adapt PAL
> to a new TEE/OS?

We have a few ways, depending on the backend. The recommended way
(because of zero perf impact) is using our patched glibc/musl version
which has all `syscall` instructions replaced with calls to Gramine. If
you don't want to modify the libc or need to run a statically compiled
app then we have a fallback, which is based on intercepting exceptions
from `syscall` instructions executed in the app and then emulating them.
This part has to differ between Linux PAL and Linux-SGX PAL: on Linux
you have to use either seccomp (which is what we do) or ptrace to trap
`syscall` instructions, and on SGX `syscall` always traps by default.

> 3. I suppose it is possible to implement only a minimal set of PAL
> functionalities for the "Hello world!"? What would they be?

Depends on what you mean by a hello world example. Different language
runtimes have different requirements.

Best,
Michał

[Dmitrii Kuvaiskii]

Dear Sid,

Thanks for the interest in the Gramine project! We'll be happy to
welcome your contributions!

Michal already replied, I'll just add some links to the relevant code.

> > 1. To run a Linux binary with Gramine on top of a new platform/OS, is
> > PAL the only Gramine component that I need to extend?
> In theory yes, but in practice you may find out that you need some
> changes to the PAL API or LibOS also (e.g. if you find a bug).

As Michal says, ideally you only need to change the PAL component, and
you don't need to touch the LibOS component.

However, in my private experiments, I also had to modify some parts of
the LibOS. In particular:
- You'll obviously need to make LibOS aware of your new PAL
implementation. It is enough to add the new PAL dependency here:
https://github.com/gramineproject/gramine/blob/a775d2748c417d8c288e26a8fee734c927590d7e/LibOS/shim/src/meson.build#L7
-

> > 2. I don't fully understand the mechanism on how you intercept system
> > calls. I had a look at "Implementing new system call" from the wiki
> > and `LibOS/shim/src/arch/x86_64/syscallas.S`,
> > `gramine/LibOS/shim/src/arch/x86_64/shim_table.c`,
> > `LibOS/shim/src/arch/x86_64/*` but I seem to be missing stuff. Could
> > you please point me to source files or general concepts to adapt PAL
> > to a new TEE/OS?
> We have a few ways, depending on the backend. The recommended way
> (because of zero perf impact) is using our patched glibc/musl version
> which has all `syscall` instructions replaced with calls to Gramine. If
> you don't want to modify the libc or need to run a statically compiled
> app then we have a fallback, which is based on intercepting exceptions
> from `syscall` instructions executed in the app and then emulating them.
> This part has to differ between Linux PAL and Linux-SGX PAL: on Linux
> you have to use either seccomp (which is what we do) or ptrace to trap
> `syscall` instructions, and on SGX `syscall` always traps by default.

[I'm describing below only the x86-64 arch.]

Please note that Michal's reply assumes that Gramine itself runs in
ring-3 (same as the app). However, it is definitely possible to run
Gramine in ring-0. In this case, your app runs in ring-3 and uses
*completely unmodified* binaries, so there is no need for our patched
Glibc/Musl and indeed no need to use anything special at all.
Gramine's LibOS and PAL components run in ring-0.

To achieve this, you'll need to set up an ISR 128 (for legacy `int
0x80` syscalls) and an MSR_LSTAR family of registers (for new
`syscall` instructions). And then you'll need a trampoline to and from
the LibOS "syscall emulation" code like this:
```
syscall_asm:
mov %rcx, %gs:PAL_TCB_VM_USER_RIP
lea sysret_asm(%rip), %rcx
jmp *%gs:(PAL_TCB_LIBOS_SYSCALLDB + 0x8) // `syscalldb`
addr is at offset 0x8

sysret_asm:
mov %gs:PAL_TCB_VM_USER_RIP, %rcx
or $0x200, %r11 // userland
RFLAGS will contain IF
sysretq
```

Note that `syscalldb()` is an entrypoint of the LibOS to emulate the
syscall: https://github.com/gramineproject/gramine/blob/a498b73657bf6978d7ee9df208eb018f60919e8f/LibOS/shim/src/arch/x86_64/syscallas.S#L30

This is a very rough idea. Here `syscall_asm` is the entry point of
your MSR_LSTAR and ISR128, and the syscall-exit path is `sysret_asm`.
With this, you'll be able to run Gramine as a proper OS kernel in
ring-0, and with *no* modifications to the ring-3 software. Hopefully
it helps.

However, if you're Ok with running Gramine in ring-3 (just like we do
it now with our Linux and Linux-SGX PALs), the relevant code is:
- Our patched Glibc (raw `syscall` instructions replaced with function
calls into Gramine):
https://github.com/gramineproject/gramine/blob/master/subprojects/packagefiles/glibc-2.34/glibc-2.34.patch
- Our patched Musl (same idea as Glibc):
https://github.com/gramineproject/gramine/blob/master/subprojects/packagefiles/musl-1.2.2/musl-1.2.2.patch
- Patched Glibc/Musl ends up calling `syscalldb`:
https://github.com/gramineproject/gramine/blob/a498b73657bf6978d7ee9df208eb018f60919e8f/LibOS/shim/src/arch/x86_64/syscallas.S#L30
- `syscalldb` jumps into Gramine LibOS code:
https://github.com/gramineproject/gramine/blob/e6b5065892099e7675e5f6f39a14635ad0302e52/LibOS/shim/src/shim_syscalls.c#L24
- This header has some useful comments:
https://github.com/gramineproject/gramine/blob/e6b5065892099e7675e5f6f39a14635ad0302e52/LibOS/shim/include/shim_entry.h

... and then Gramine LibOS component emulates the syscall, and when
needed, calls into PAL functions (described in `pal.h`).

> > 3. I suppose it is possible to implement only a minimal set of PAL
> > functionalities for the "Hello world!"? What would they be?
> Depends on what you mean by a hello world example. Different language
> runtimes have different requirements.

If you're talking about a `printf("helloworld")` C code (linked
against Glibc), then from my experience it's a lot of PAL stuff to
implement:
- Pipes (`db_pipes.c`) -- used by internal LibOS threads, so you need
to implement them even if your app doesn't use them
- Ppoll-like waiting (`db_object.c`) -- used by internal LibOS
threads, so same as above
- Threading and scheduling (`db_threading.c`) -- again, since we have
internal helper threads in LibOS, you'll need this as well even if
your app doesn't use threads
- Events (`db_events.c`) -- same as all above, used by internal LibOS threads
- Files (`db_files.c`) -- because your app will load at least libc.so
- Memory management (`db_memory.c`) -- obviously
- Misc stuff (`db_misc.c`) -- because internally LibOS needs stuff
like time, setting FSGS regs, etc
- RTLD (`db_rtld.c`) -- may be absolutely minimal, see Linux-SGX example
- Main load and init phase (`db_main.c`)

Some things that you do NOT need to implement straight away:
- `db_devices.c`
- `db_eventfd.c`
- `db_exception.c`
- `db_process.c`
- `db_sockets.c` (but this will be required as soon as you want network sockets)
- `db_streams.c`

--
Yours sincerely,
Dmitrii Kuvaiskii

[Sid Hussmann]

Thank you, Michał and Dmitrii, for your answers.

Your responses made me realize that I have some catching up to do.

I have read the original paper [0], the current Gramine docs [1], and the CC presentation from 2021 [2].

[0] http://www.cs.unc.edu/~porter/pubs/tsai14graphene.pdf
[1] https://gramine.readthedocs.io/en/latest/
[2] https://confidentialcomputing.io/wp-content/uploads/sites/85/2021/02/CCC-webinar-Graphene-Technical-Introduction-02-2020.pdf

> Ifyou don't want to modify the libc or need to run a statically compiled

> app then we have a fallback, which is based on intercepting exceptions
> from `syscall` instructions executed in the app and then emulating them.
> This part has to differ between Linux PAL and Linux-SGX PAL: on Linux
> you have to use either seccomp (which is what we do) or ptrace to trap
> `syscall` instructions, and on SGX `syscall` always traps by default.

Thank you, that is a crucial part of understanding and what initially caused some confusion. With the statically linked scenario, I'm not sure what the best solution would be for trapping syscalls in our case.

- Running Gramine and its userland app within a VM
- Binary patching the app
- Trapping the syscall in the microkernel

> Please note that Michal's reply assumes that Gramine itself runs in
> ring-3 (same as the app). However, it is definitely possible to run
> Gramine in ring-0. In this case, your app runs in ring-3 and uses

> _completely unmodified_ binaries, so there is no need for our patched

> Glibc/Musl and indeed no need to use anything special at all.
> Gramine's LibOS and PAL components run in ring-0.

If I understand correctly, in practice, this means having Gramine+App run inside a VM, right? But this raises the question of how scheduling and memory management etc., would be handled.

> With this, you'll be able to run Gramine as a proper OS kernel in

> ring-0, and with _no_ modifications to the ring-3 software.

This statement seems to answer that question. However, I can't wrap my head around how this would work.

Is this all being taken care of by Gramine when it's executed in ring-0? If this is the case, this opens the question of how to interact with that VM from our userland. I can imagine needing something similar to the "Untrusted Linux SGX PAL" as seen in slide 15 of [2] (above).

Further, the docs [1] mentioned that on SGX each Linux application process would run in their own SGX enclave. E.g., executing `bash -c ls` would end up spawning two SGX enclaves. In the VM approach, would you also implement that scenario so that it spawns two VMs?

Another interpretation of your response could also suggest that we'd somehow run Gramine next to the kernel, which seems like a wrong approach from a microkernel mindset.

> - Our patched Glibc (raw`syscall` instructions replaced with function
> calls into Gramine):
> https://github.com/gramineproject/gramine/blob/master/subprojects/packagefiles/glibc-2.34/glibc-2.34.patch
> - Our patched Musl (same idea as Glibc):
> https://github.com/gramineproject/gramine/blob/master/subprojects/packagefiles/musl-1.2.2/musl-1.2.2.patch

I did not expect that patch to be as straightforward as this, TBH.

> If you're talking about a `printf("helloworld")` C code (linked
> against Glibc), then from my experience it's a lot of PAL stuff to
> implement:
> - Pipes (`db_pipes.c`) -- used by internal LibOS threads, so you need
> to implement them even if your app doesn't use them
> - Ppoll-like waiting (`db_object.c`) -- used by internal LibOS
> threads, so same as above
> - Threading and scheduling (`db_threading.c`) -- again, since we have
> internal helper threads in LibOS, you'll need this as well even if
> your app doesn't use threads
> - Events (`db_events.c`) -- same as all above, used by internal LibOS threads
> - Files (`db_files.c`) -- because your app will load at least libc.so
> - Memory management (`db_memory.c`) -- obviously
> - Misc stuff (`db_misc.c`) -- because internally LibOS needs stuff
> like time, setting FSGS regs, etc
> - RTLD (`db_rtld.c`) -- may be absolutely minimal, see Linux-SGX example
> - Main load and init phase (`db_main.c`)

Thank you. Yes, I was referring to precisely this use-case. So the way I understand, I would copy the directory `gramine/Pal/src/host/Skeleton` to something like `gramine/Pal/src/host/GapfruitOS`. This will provide the Gramine ABI. From that point on, I would then implement the function stubs. For that implementation, I would link against the FreeBSD libc port (that we already support) or use the Genode API.

Are my assumptions correct?

And last, has anyone successfully used `gcc` instead of `clang` to compile Gramine?

Best,
Sid

[Dmitrii Kuvaiskii]

> Thank you, that is a crucial part of understanding and what initially caused some confusion. With the statically linked scenario, I'm not sure what the best solution would be for trapping syscalls in our case.

Sorry, I don't know anything about Gapfruit and Genode, so I'm missing
some context. If you want to "sink" Gramine to ring-0, then you
shouldn't care about trapping syscalls -- the syscall instruction (or
`int 0x80`, whatever your apps use) will jump to the ring-0's Gramine
syscall entrypoint. Of course, you'll need to modify Gramine for this
slightly, but it's doable.

> - Running Gramine and its userland app within a VM
> - Binary patching the app
> - Trapping the syscall in the microkernel
> > Please note that Michal's reply assumes that Gramine itself runs in
> > ring-3 (same as the app). However, it is definitely possible to run
> > Gramine in ring-0. In this case, your app runs in ring-3 and uses
> > _completely unmodified_ binaries, so there is no need for our patched
> > Glibc/Musl and indeed no need to use anything special at all.
> > Gramine's LibOS and PAL components run in ring-0.
>
> If I understand correctly, in practice, this means having Gramine+App run inside a VM, right? But this raises the question of how scheduling and memory management etc., would be handled.

I thought that this is what Gapfruit and Genode provide -- the
primitives to link against and call into, including the primitives for
scheduling and memory management. So this is not the case? Then you'll
need to implement these primitives in Gramine itself, yes. The current
Gramine code in Linux and Linux-SGX PALs does not contain scheduling
and contains only minimal memory management (in Linux-SGX PAL),
because Gramine relies on the host OS to do that.

And yes, "sinking" Gramine to ring-0 necessitates the use of a VM.
Again, I thought this is what you intended to do anyway with Gapfruit
:) It is possible to have rather minimal mods to Gramine to enable it
as a minimalistic VM kernel; especially if you use QEMU with "microvm"
machine type or Firecracker.

> > With this, you'll be able to run Gramine as a proper OS kernel in
> > ring-0, and with _no_ modifications to the ring-3 software.
>
> This statement seems to answer that question. However, I can't wrap my head around how this would work.
>
> Is this all being taken care of by Gramine when it's executed in ring-0? If this is the case, this opens the question of how to interact with that VM from our userland. I can imagine needing something similar to the "Untrusted Linux SGX PAL" as seen in slide 15 of [2] (above).

No, you'll need to add additional code to Gramine. Hm, sorry, I seem
to completely misunderstand how and why you want to use Gramine with
your GapFruit. I guess you actually want Gramine to run in ring-3 as a
service, and then other ring-3 apps will contact Gramine for
something? Why would ring-3 apps go to Gramine? What's the purpose?

> Further, the docs [1] mentioned that on SGX each Linux application process would run in their own SGX enclave. E.g., executing `bash -c ls` would end up spawning two SGX enclaves. In the VM approach, would you also implement that scenario so that it spawns two VMs?

Good question. In my personal experiments, I made a decision to
support only single-process applications. So no need to spawn a second
VM, because I simply didn't support fork/clone.

But in general yes, the Gramine philosophy is to spawn a new isolate
for each new child process. So you would end up with two VMs.

> Another interpretation of your response could also suggest that we'd somehow run Gramine next to the kernel, which seems like a wrong approach from a microkernel mindset.

No, I thought of Gramine running in ring-0 as a tiny OS kernel in a VM.

> Thank you. Yes, I was referring to precisely this use-case. So the way I understand, I would copy the directory `gramine/Pal/src/host/Skeleton` to something like `gramine/Pal/src/host/GapfruitOS`. This will provide the Gramine ABI. From that point on, I would then implement the function stubs. For that implementation, I would link against the FreeBSD libc port (that we already support) or use the Genode API.

That's correct. That's how a new PAL is written.

> And last, has anyone successfully used `gcc` instead of `clang` to compile Gramine?

Yes, of course. We actually build Gramine with GCC most of the time.
We use Clang only for AddressSanitizer builds.

[Michał Kowalczyk]

On 2/23/22 17:50, Sid Hussmann wrote:
>
>> Ifyou don't want to modify the libc or need to run a statically compiled
>> app then we have a fallback, which is based on intercepting exceptions
>> from `syscall` instructions executed in the app and then emulating them.
>> This part has to differ between Linux PAL and Linux-SGX PAL: on Linux
>> you have to use either seccomp (which is what we do) or ptrace to trap
>> `syscall` instructions, and on SGX `syscall` always traps by default.
>

> Thank you, that is a crucial part of understanding and what initially
> caused some confusion. With the statically linked scenario, I'm not
> sure what the best solution would be for trapping syscalls in our case.
>

> - Running Gramine and its userland app within a VM
> - Binary patching the app
> - Trapping the syscall in the microkernel

I'd suggest just doing the same as Linux PAL does: trap `syscall`
instruction (you can do this, as it's your own kernel where you can add
such functionality) and additionally (for performance) provide patched
versions of some popular libcs with `syscall` instructions replaced with
calls to Gramine.
I don't really see any justification to put Gramine into ring0 in this
case, it just makes everything much more complex.

>> Please note that Michal's reply assumes that Gramine itself runs in
>> ring-3 (same as the app). However, it is definitely possible to run
>> Gramine in ring-0. In this case, your app runs in ring-3 and uses
>> _completely unmodified_ binaries, so there is no need for our patched
>> Glibc/Musl and indeed no need to use anything special at all.
>> Gramine's LibOS and PAL components run in ring-0.
>
> If I understand correctly, in practice, this means having Gramine+App
> run inside a VM, right? But this raises the question of how scheduling
> and memory management etc., would be handled.
>

Yes, that's why I wouldn't recommend going this route :)

>> With this, you'll be able to run Gramine as a proper OS kernel in
>> ring-0, and with _no_ modifications to the ring-3 software.
>
> This statement seems to answer that question. However, I can't wrap my
> head around how this would work.
>

You can also run software without any modification when Gramine is in
ring3, I don't understand Dmitrii's point here. Both ring0 and ring3
solutions are the same in this manner.

>
> Another interpretation of your response could also suggest that we'd
> somehow run Gramine next to the kernel, which seems like a wrong
> approach from a microkernel mindset.
>

I agree, this doesn't fit nicely into a microkernel, but ring3 solutions
doesn't have this issue.

>
> Thank you. Yes, I was referring to precisely this use-case. So the way
> I understand, I would copy the directory
> `gramine/Pal/src/host/Skeleton` to something like
> `gramine/Pal/src/host/GapfruitOS`. This will provide the Gramine ABI.
> From that point on, I would then implement the function stubs. For
> that implementation, I would link against the FreeBSD libc port (that
> we already support) or use the Genode API.
>

> Are my assumptions correct?
Yes, this sounds right.

Dmitrii:

> Please note that Michal's reply assumes that Gramine itself runs in
> ring-3 (same as the app). However, it is definitely possible to run
> Gramine in ring-0.

Yes, but why would you do it? I see no advantage of running Gramine in
ring0 in this case, it only makes everything *much* more complex.

>> If I understand correctly, in practice, this means having Gramine+App run inside a VM, right? But this raises the question of how scheduling and memory management etc., would be handled.
> I thought that this is what Gapfruit and Genode provide -- the
> primitives to link against and call into, including the primitives for
> scheduling and memory management. So this is not the case? Then you'll
> need to implement these primitives in Gramine itself, yes. The current
> Gramine code in Linux and Linux-SGX PALs does not contain scheduling
> and contains only minimal memory management (in Linux-SGX PAL),
> because Gramine relies on the host OS to do that.

But Sid asked here about the kernel running inside the VM, which would
be Gramine, not Gapfruit. And Gramine doesn't have such functionality.

>
> No, you'll need to add additional code to Gramine. Hm, sorry, I seem
> to completely misunderstand how and why you want to use Gramine with
> your GapFruit. I guess you actually want Gramine to run in ring-3 as a
> service, and then other ring-3 apps will contact Gramine for
> something? Why would ring-3 apps go to Gramine? What's the purpose?

Not as a service, but as a shim to be able to run binaries based on
Linux kernel ABI. Same as we currently do in Linux PAL and in the past
did with FreeBSD PAL.

[Michał Kowalczyk]

On 3/7/22 13:31, Sid Hussmann wrote:
> From that perspective, this multi-kernel support makes me
> hesitant to implement this syscall-trap feature within the various
> microkernels. However, I would not completely rule out this path. Maybe
> this could be a use case that only one microkernel would support.
Sure, but then you'll be able to support only dynamic binaries. For this
you just need to provide a patched libc binaries, as we do ourselves in
Gramine for performance.

>
>> Same as we currently do in Linux PAL and in the past
>> did with FreeBSD PAL.

> Huh. I did not know that Gramine had a FreeBSD PAL. I assume the FreeBSD
> support was dropped because of maintenance reasons and not technical
> limitations. Since we support the libc from FreeBSD, it might be more
> straightforward to pick up where you left.
We had, but no one maintained, nor tested it. You can find some
(probably broken) version here:
https://github.com/gramineproject/graphene/tree/don/freebsd. I have no
idea whether and to what extent it ever worked, when ITL came to this
project FreeBSD PAL was already broken and covered with a thick layer of
dust :)

[Sid Hussmann]

On 2/24/22 11:32, Dmitrii Kuvaiskii wrote:

> Sorry, I don't know anything about Gapfruit and Genode, so I'm missing
> some context.

Of course. My apologies for not sharing more details from the beginning.
Let me try to give a short overview. Genode is a framework to design
component-based scenarios or whole operating systems on top of various
microkernels. Every component is strongly isolated and only gets access
to resources and services in an explicit manner. The microkernel
guarantees strong isolation and maintains access control to various
resources via capability objects. This mechanism is often referred to as
capability-based security.

The system designer defines the component topology in a declarative and
nested manner - making it possible to abstract sub-systems while
maintaining the possibility to define fine-grained trust relationships
between the components. This approach lets you identify the critical
components make them as simple and robust as possible, which ultimately
leads to an overall resiliency of the system.

Gapfruit OS is an operating system built with the Genode framework. We
currently focus on headless appliances on x86_64 and arm_v8. We support
various runtimes: E.g., FreeBSD libc, WASM/WASI, ADA/SPARK, Solo5
Unikernel Runtime, JVM, etc., and full-blown VMs. For some workloads, we
would like to run unmodified Linux binaries. Obviously, this would work
with running Linux within a VM. However, we would like a more
lightweight solution. Hence, my approach to talking to y'all about
Gramine :)

>> If I understand correctly, in practice, this means having Gramine+App
>> run inside a VM, right? But this raises the question of how
>> scheduling and memory management etc., would be handled.
>
> I thought that this is what Gapfruit and Genode provide -- the
> primitives to link against and call into, including the primitives for
> scheduling and memory management. So this is not the case?

That's correct. However, everything but the microkernel runs in ring-3.
The microkernel is the only component that runs in ring-0.

> I guess you actually want Gramine to run in ring-3 as a
> service, and then other ring-3 apps will contact Gramine for
> something? Why would ring-3 apps go to Gramine? What's the purpose?

To have a lightweight runtime for unmodified Linux apps on Gapfruit OS.

>> Further, the docs [1] mentioned that on SGX each Linux application
>> process would run in their own SGX enclave. E.g., executing `bash -c
>> ls` would end up spawning two SGX enclaves. In the VM approach, would
>> you also implement that scenario so that it spawns two VMs?
>
> Good question. In my personal experiments, I made a decision to
> support only single-process applications. So no need to spawn a second
> VM, because I simply didn't support fork/clone.
>
> But in general yes, the Gramine philosophy is to spawn a new isolate
> for each new child process. So you would end up with two VMs.

Eventually, I would like to have the ability for Linux processes to
spawn children. From my perspective, the children of "Linux" processes
that run on top of Gramine would belong to the same trust boundary. I
sense that it would be better for performance reasons to share the same
instance of Gramine for the different processes. But since the VM
approach does not seem reasonable anymore, there might be other factors
to consider.

On 2/28/22 02:29, Michał Kowalczyk wrote:

> I'd suggest just doing the same as Linux PAL does: trap `syscall`
> instruction (you can do this, as it's your own kernel where you can add
> such functionality) and additionally (for performance) provide patched
> versions of some popular libcs with `syscall` instructions replaced with
> calls to Gramine.

A little more background regarding microkernels: We consider the
microkernel as one of many building blocks (albeit an important one)
that form the operating system. We choose different microkernels for
different use-cases. We support Nova, base-hw, SeL4, and Linux (using
seccomp for isolation and fd's as capability objects). E.g., if
virtualization is needed on x86_64, the Nova hypervisor works best. In a
way, we are users of these microkernels. And I would certainly not call
myself a microkernel expert, especially when it comes to implementation
details. From that perspective, this multi-kernel support makes me

hesitant to implement this syscall-trap feature within the various
microkernels. However, I would not completely rule out this path. Maybe
this could be a use case that only one microkernel would support.

> I don't really see any justification to put Gramine into ring0 in this
> case, it just makes everything much more complex.

Agreed.

> Not as a service, but as a shim to be able to run binaries based on
> Linux kernel ABI.

That sums it up perfectly.

> Same as we currently do in Linux PAL and in the past
> did with FreeBSD PAL.

Huh. I did not know that Gramine had a FreeBSD PAL. I assume the FreeBSD
support was dropped because of maintenance reasons and not technical
limitations. Since we support the libc from FreeBSD, it might be more
straightforward to pick up where you left.

Cheers,
Sid