Run untrusted software inside SGX with Gramine

[Stanislav Iablokov]

Hello and thank you for developing such a wonderful tool for confidential computing.

In our project we intend to use Gramine for launching (almost) arbitrary applications inside a Secure Enclave. More precisely, a special launcher A is first loaded as Gramine's entrypoint. This launcher next downloads trusted software B & private data C from a designated repository. This data C (and software B) is then passed to the Secure Enclave and decrypted. The software B is then (checked for integrity and) launched using launcher A on input data C. The output D is then encrypted and uploaded to the repository.

The key ideas behind this use case is that

(i) the list of files (software B and data C) is unknown when building a Gramine manifest and running the launcher A;

(ii) we want first to copy input software B & data C from HDD to Secure Enclave's RAM and then use it there (decrypt and launch) as if they were a part of some virtual file system.

Is that even possible with Gramine?

Thank you.

[Borys]

Hi,
I'm not sure I've understood you correctly, because you have "untrusted" in topic, yet from the description I would assume you mean "trusted"? I'll assume trusted.
Anyway, there are couple of options, but the best one is to use a feature we have called "Encrypted Files" (you can read more here: https://gramine.readthedocs.io/en/stable/manifest-syntax.html#encrypted-files) - it's basically sounds like what you would want

Borys

[Stanislav Iablokov]

Well, they are untrusted in one sense (not specified in manifest) and trusted in the other (possess metadata with trusted signature)...

Anyway, using "Encrypted Files" is not suitable for me, because in this case the entire list of files should be specified in advance.

However, in my case, this list can be dynamic and depends on many parameters. Thus, the only file I would like to pass into Secure Enclave would be a configuration file (e.g., config.json) with identifiers of data archives (software & datasets) which are to be downloaded lately from a repository into Secure Enclave's RAM. And also, I would like to use my own encryption scheme, not the ones provided by Gramine.

Once again, the key point here is that I need a way to copy files from either a website/cloud or HDD into some kind of virtual file system inside an enclave, and then work with them as if they were just files stored on HDD.

[Dmitrii Kuvaiskii]

Dear Stanislav,

You probably just want a `tmpfs` mount point:
https://gramine.readthedocs.io/en/latest/manifest-syntax.html#fs-mount-points
(see the paragraph on "tmpfs").

Tmpfs files behave like normal files to the in-enclave application on
top of Gramine, but inside of Gramine they are just pseudo-files
backed by strings in enclave memory. This way, the contents of files
never leave the enclave, and so they are protected by the Intel SGX
hardware itself.

But note that tmpfs files are *not* copied to children enclaves and
*not* synchronized between enclaves. See this comment:
https://github.com/gramineproject/gramine/blob/master/libos/src/fs/tmpfs/fs.c#L8-L10

TLDR: use tmpfs, but keep the usage of these tmpfs files limited to
only one Gramine enclave.

> --
> You received this message because you are subscribed to the Google Groups "Gramine Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gramine-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/gramine-users/CAM_SVPMs2iRGHXV%3DQLqzd4oxOBne9ZiwRmAxot44K1_qqggh5Q%40mail.gmail.com.



--
Yours sincerely,
Dmitrii Kuvaiskii

[Stanislav Iablokov]

Thank you. I think this would satisfy my needs.

Once again, let me describe my algorithm.

1. I first put several data archives into a folder on HDD, e.g., "/data"

2. Second, I specify the whole "/data" folder (not a list of individual files, because files can be added to this folder in the runtime) in the manifest within sgx.allowed_files section

3. Next, inside the enclave I run "cp /data/* /tmpfs -R"

4. Finally, I run apps inside the enclave with these files being passed as inputs.

Could you please confirm that this would work?

[Borys]

> Anyway, using "Encrypted Files" is not suitable for me, because in this
> case the entire list of files should be specified in advance.
> However, in my case, this list can be dynamic and depends on many
> parameters.

It is suitable. You do not specify a list of files in manifest in case of Encrypted Files, just a directory, which content is not know at build time.

Also you can write to that directory (I believe) at runtime, creating new encrypted files.

> And also, I would like to use my own
> encryption scheme, not the ones provided by Gramine.
>

Well, you can use your own on top of this (albeit I don't see a reason why). Just decrypt files into tmpfs (like Dmitrii explained, mount tmpfs in manifest). Or into Encrypted Files, if you want doubly encrypted files lying on host OS drive.

[Dmitrii Kuvaiskii]

> Once again, let me describe my algorithm.
> 1. I first put several data archives into a folder on HDD, e.g., "/data"
> 2. Second, I specify the whole "/data" folder (not a list of individual files, because files can be added to this folder in the runtime) in the manifest within sgx.allowed_files section
> 3. Next, inside the enclave I run "cp /data/* /tmpfs -R"
> 4. Finally, I run apps inside the enclave with these files being passed as inputs.
>
> Could you please confirm that this would work?

Yes, this flow looks good to me. I'm assuming that data archives under
`/data/` are all encrypted with your secret key. I'm also assuming
that somewhere at step 3, you copy the contents of each of the
`/data/` files in enclave memory, decrypt the contents, and then
create files under `/tmp/` and write the decrypted contents into them
(in other words, a simple `cp /data/* /tmp/` makes no sense because it
copies encrypted contents to another encrypted contents).

With all these details in mind, yes, your flow makes perfect sense and
is doable in Gramine-SGX (bar some bugs, but in this case report them
and we'll fix). Also, see Borys's reply which basically tells the
same.

By the way, your Gramine manifest file must contain something like this then:
```
fs.mounts = [
...
{ type = "chroot", path = "/data/", uri = "file:/data/" },
{ type = "tmpfs", path = "/tmp" },
]

sgx.allowed_files = [
...
"file:/data/",
]
```

On Tue, Jul 5, 2022 at 1:48 PM Stanislav Iablokov

> To view this discussion on the web visit https://groups.google.com/d/msgid/gramine-users/CAM_SVPPKKo-5RUXFb%2BZrK%3DOzvFWnvZ4uN0GVc9ffE3S0cqxU2Q%40mail.gmail.com.

[Stanislav Iablokov]

Great. This sounds good. Thank you.

In case of other questions, should I contact you (or us...@gramineproject.io ) directly or is there a better way for resolving these types of issues (like github or some forum)?

I'm looking towards launching a chain (pipeline) of Docker containers inside a Secure Enclave via Gramine. This presumably would generate a lot of questions.

[Borys]

> In case of other questions, should I contact you (or us...@gramineproject.io
> ) directly or is there a better way for resolving these types of issues
> (like github or some forum)?

Either that email list or you can join our gitter chat: https://gitter.im/gramineproject

> I'm looking towards launching a chain (pipeline) of Docker containers
> inside a Secure Enclave via Gramine. This presumably would generate a lot
> of questions.

You cannot run a Docker container inside SGX enclave (only other way round, i.e. an enclave inside a docker container).


Borys

[Stanislav Iablokov]

Okay, I will join the chat soon.

If running Docker is not possible, what other options to restrict/control the execution of software inside an enclave is possible?

I want my python script to execute a sequence of software (both executables and other python scripts), preferably, each piece of software in its own environment.

To be more precise, I want to run a bioinformatic pipeline inside an enclave, whose steps are orchestrated (launched one by one) with my script (e.g., run.py) placed at the entrypoint (i.e., "python run.py").

[Borys]

> If running Docker is not possible, what other options to restrict/control
> the execution of software inside an enclave is possible?

What do you mean? Docker is not for restricting software, merely for having a same environment everywhere. If you want to restrict the software you need to use a security sandbox (you can just run whole Gramine inside it).

> I want my python script to execute a sequence of software (both executables
> and other python scripts), preferably, each piece of software in its own
> environment.

What do you mean by environment here? Does these scripts and executables come from one source and trust each other? If not, then you would have to spawn each in a dedicated enclave and attest them one by one. If they trust each other (e.g. can share encryption keys, because they come from one tenant) then you can just run it in one enclave (just a simple python/bash script running each binary is the simplest apporach).

> To be more precise, I want to run a bioinformatic pipeline inside an
> enclave, whose steps are orchestrated (launched one by one) with my script
> (e.g., run.py) placed at the entrypoint (i.e., "python run.py").
>

What's the problem with just running this in Gramine? You do not trust rest of the scripts? Then how can you give them your unencrypted data to work on?

BTW, a malicious binary can take over the enclave completely, so you cannot run both your trusted with secrets software and untrusted binary in one enclave.


Borys