Gramine Enclave size and performance

[Raghudeep Kannavara]

Hello!

I have a couple of question on the gramine. 

1. Is there a size limitation on applications that can run in gramine SGX? i.e., without application modification or application refactoring?
2. Are there are performance hits on system call, network io, anything that causes transitions outside of enclave?

Thank you for looking into these questions.

Thanks,

Raghudeep Kannavara

[Wojtek Porczyk]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Thu, Mar 21, 2024 at 05:37:33PM +0000, 'Raghudeep Kannavara' via Gramine Users wrote:
> 1.

> Is there a size limitation on applications that can run in gramine SGX?
> i.e., without application modification or application refactoring?

Not directly, but there are performance discontinuities related to the size of
the application's memory. One is, on not-very-recent silicon there's limited
Enclave Page Cache (EPC) and if you run too big app or too many enclaves at
once, this might cause excessive swapping.

> 2.

> Are there are performance hits on system call, network io, anything that
> causes transitions outside of enclave?

There is significant performance drop compared to running app outside of SGX
enclave. You need to measure it yourself for your own workload on a particular
configuration.


- --
pozdrawiam / best regards
Wojtek Porczyk
Gramine / Invisible Things Lab

I do not fear computers,
I fear lack of them.
-- Isaac Asimov
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEaO0VFfpr0tEF6hYkv2vZMhA6I1EFAmX9WJMACgkQv2vZMhA6
I1GONw/8DoIjnG5vXch7bg0BAxfIIugugmhFkAs86CV6U7om6k0t+MCuw970NLUd
YYqSANHJnoQDkZx6Z6cMvXrLmZz78ICqmf4bUHLMW2IoTYuvteyfVGHnLhZAAkNu
MOcUl0VpUR8ziXQSyx9ges22bSZNg6HyMBhacQ8qK2iejz04qGnhoU+4EPK8pbmc
8D5tQHVxZMycjv+VtQywD+ERnXymkae3EAUWk9IXHvncDjcMV3JW/itxQp2um5Mt
yF3yTerOzY1p04qAbdfKF/MI1b1GxN9hKx3UfvZT+5yX9r2ctibRn9ULduMFIRIG
O4i4Ht0z8c6LTYOgFgzpuZTTyDDcvFn+AbZqGBajqxtDmvfpN/1GaEi3yz4Ls0ja
3mpQBvtloElqypNBCvmSCdBdEZx+ccvs4MHZMz+aXW/fXIV8s8kT2wyAJi9rlEbJ
VtgBiwmBWcqUHZJH+3TNa4lmTieEiE2v+1+VETtX+PVGlJ1lU7YmtQ3ajezXfIxt
ZrX5itAi0HWbB+ZaflTzowaBrnuPAOOZvZdNYaXSbhp0crZ5PyomrzYapD/IYdvO
hqbmSSCMJX711bctNENw7XR24vpRSiaV43YXk75T3X8ZPVMeYxNZuKePqkynkL6b
HVQQtlFMo4bsMqHYPVJ8UE8SEUxzkSVFs7n9AefXt8fdW1Uw0I4=
=L4we
-----END PGP SIGNATURE-----

[Michał Kowalczyk]

On 3/22/24 11:08, Wojtek Porczyk wrote:
> On Thu, Mar 21, 2024 at 05:37:33PM +0000, 'Raghudeep Kannavara' via
> Gramine Users wrote:
> >   1.
> > Is there a size limitation on applications that can run in gramine SGX?
> > i.e., without application modification or application refactoring?
>
> Not directly, but there are performance discontinuities related to the
> size of
> the application's memory. One is, on not-very-recent silicon there's
> limited
> Enclave Page Cache (EPC) and if you run too big app or too many
> enclaves at
> once, this might cause excessive swapping.

One clarification: technically there's an upper bound on possible
enclave size, but in practice it's large enough that it's unlikely to
ever cause any problems for you. You can check it with our
`is-sgx-available` tool provided with Gramine, the relevant lines start
with "Max enclave size". On the systems I have it's 64 GB for an old,
client CPU, and 64 PB for a much newer server CPU.

[Vij, Mona]

I wouldn't say that there is significant performance drop as a general statement. It depends on your workload. If your application does not fit in EPC then you may see big overheads. But limited EPC is an issue with older hardware, most of new server parts have upto 1TB of EPC, so EPC size problem is not an issue anymore, and that is why we see little perf issue with most applications. If your application fits within the EPC then in some cases we have seen as low as 2-3% overhead.

So really depends on the workload and we have several tuning optimizations to improve your performance in Gramine.

See https://gramine.readthedocs.io/en/stable/performance.html

Thanks
Mona

--
You received this message because you are subscribed to the Google Groups "Gramine Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gramine-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gramine-users/Zf1YkwiG8Q9HnSpY%40invisiblethingslab.com.

[Michał Kowalczyk]

Woju was unclear on this, but I think he was replying to "Are there are performance hits on system call, network io, anything that causes transitions outside of enclave?" - so, it was only about transitions to the outside of the enclave.

On 3/22/24 16:30, Vij, Mona wrote:

> I wouldn't say that there is significant performance drop as a general statement. It depends on your workload. If your application does not fit in EPC then you may see big overheads. But limited EPC is an issue with older hardware, most of new server parts have upto 1TB of EPC, so EPC size problem is not an issue anymore, and that is why we see little perf issue with most applications. If your application fits within the EPC then in some cases we have seen as low as 2-3% overhead. > > So really depends on the workload and we have several tuning optimizations to improve your performance in Gramine. > > See https://gramine.readthedocs.io/en/stable/performance.html > > Thanks > Mona > > -----Original Message----- > From: gramin...@googlegroups.com <gramin...@googlegroups.com> On Behalf Of Wojtek Porczyk > Sent: Friday, March 22, 2024 3:08 AM > To: Raghudeep Kannavara <ragh...@meta.com> > Cc: us...@gramineproject.io > Subject: Re: Gramine Enclave size and performance >

On Thu, Mar 21, 2024 at 05:37:33PM +0000, 'Raghudeep Kannavara' via Gramine Users wrote:

>   1.
> Is there a size limitation on applications that can run in gramine SGX?
> i.e., without application modification or application refactoring?

Not directly, but there are performance discontinuities related to the size of the application's memory. One is, on not-very-recent silicon there's limited Enclave Page Cache (EPC) and if you run too big app or too many enclaves at once, this might cause excessive swapping.

>   2.
> Are there are performance hits on system call, network io, anything
> that causes transitions outside of enclave?

There is significant performance drop compared to running app outside of SGX enclave. You need to measure it yourself for your own workload on a particular configuration.

[Raghudeep Kannavara]

Hi,

I had to change machines to due older skylake processor. I am now using this NUC:   https://www.amazon.com/Intel-Barebones-NUC9VXQNX-Bluetooth-Thunderbolt/dp/B0CJG16Z8B

I have gramine installed on this on CentOS9. Here is the output from is-sgx-available:

[meta@localhost helloworld]$ is-sgx-available
SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): false
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): false
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x1000000000
EPC size: 0x5d80000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
#PF/#GP information in EXINFO in MISC region of SSA supported: false
#CP information in EXINFO in MISC region of SSA supported: false
[meta@localhost helloworld]$

Within CI Example, I run the make command for the Hello World example

[meta@localhost helloworld]$ make
cc -Wall -Wextra -O3   -c -o helloworld.o helloworld.c
cc   helloworld.o   -o helloworld
gramine-manifest \
-Dlog_level=error \
helloworld.manifest.template helloworld.manifest
WARNING: error in manifest (after rendering): required key not provided @ data['loader']['entrypoint']
gramine-manifest-check helloworld.manifest
error in manifest: required key not provided @ data['loader']['entrypoint']
make: *** [Makefile:26: helloworld.manifest] Error 1

It looks an error is thrown, can you please help with this?

Thanks,

Raghudeep Kannavara

---

From: Michał Kowalczyk <mk...@invisiblethingslab.com>
Sent: Friday, March 22, 2024 6:35 PM
To: Vij, Mona <mona...@intel.com>; Wojtek Porczyk <wo...@invisiblethingslab.com>; Raghudeep Kannavara <ragh...@meta.com>
Cc: us...@gramineproject.io <us...@gramineproject.io>

Subject: Re: Gramine Enclave size and performance

Woju was unclear on this, but I think he was replying to "Are there are performance hits on system call, network io, anything that causes transitions outside of enclave?" - so, it was only about transitions to the outside of the enclave. On

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

 

ZjQcmQRYFpfptBannerEnd

[Michał Kowalczyk]

I think this was caused by running mismatched examples and Gramine versions - I guess your helloworld example code comes from the current master (which already contains https://github.com/gramineproject/gramine/pull/1716), but you installed an older Gramine version which you use to run the example (probably 1.7 from the repo).

Please try to either use the example code from the repo tag 1.7, or build Gramine from master.

[Raghudeep Kannavara]

Thank you. This is resolved with pulling in 1.7 tagged CI example repo. I was able to get the hello world to work.

I am trying to build the ra-tls-secret-prov. I was able to get the SPID (linkable) from the Intel website.  I am following the quick start guide. https://github.com/gramineproject/gramine/tree/master/CI-Examples/ra-tls-secret-prov#quick-start

I am running into some errors with this. I have the output copied below for reference, was hoping you may have some insights into this please. 

[meta@localhost ra-tls-secret-prov]$ export RA_TLS_ALLOW_DEBUG_ENCLAVE_INSECURE=1
export RA_TLS_ALLOW_OUTDATED_TCB_INSECURE=1
export RA_TLS_ALLOW_HW_CONFIG_NEEDED=1
export RA_TLS_ALLOW_SW_HARDENING_NEEDED=1
[meta@localhost ra-tls-secret-prov]$
[meta@localhost ra-tls-secret-prov]$
[meta@localhost ra-tls-secret-prov]$
[meta@localhost ra-tls-secret-prov]$ make app epid RA_TYPE=epid RA_CLIENT_SPID=111111111111111111111111111      RA_CLIENT_LINKABLE=1
cd secret_prov_minimal && \
gramine-manifest \
-Dlog_level=error \
-Darch_libdir=/lib/x86_64-redhat-linux \
-Dra_type=epid \
-Dra_client_spid=974654C017326EA311C8E3A6C07E0BE9 \
-Dra_client_linkable=1 \
client.manifest.template > client.manifest
cc secret_prov_minimal/client.c -O2 -fPIE -Wall -std=c11 -I/usr/include/gramine  -pie -Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64  -o secret_prov_minimal/client
cd secret_prov_minimal && \
gramine-sgx-sign \
--manifest client.manifest \
--output client.manifest.sgx
Traceback (most recent call last):
  File "/usr/bin/gramine-sgx-sign", line 171, in <module>
    main() # pylint: disable=no-value-for-parameter
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/bin/gramine-sgx-sign", line 132, in main
    expanded = manifest.expand_all_trusted_files(chroot=chroot)
  File "/usr/lib64/python3.9/site-packages/graminelibos/manifest.py", line 415, in expand_all_trusted_files
    for tf in TrustedFile.from_manifest(data, chroot=chroot).expand_directory():
  File "/usr/lib64/python3.9/site-packages/graminelibos/manifest.py", line 260, in expand_directory
    raise ManifestError(f'URI {self.uri!r} ends with "/" but is not a directory')
graminelibos.manifest.ManifestError: URI 'file:/lib/x86_64-redhat-linux/' ends with "/" but is not a directory
make: *** [Makefile:96: sgx_sign_secret_prov_minimal_client] Error 1

---

From: Michał Kowalczyk <mk...@invisiblethingslab.com>
Sent: Thursday, May 23, 2024 3:53 PM
To: Raghudeep Kannavara <ragh...@meta.com>
Cc: us...@gramineproject.io <us...@gramineproject.io>; Vij, Mona <mona...@intel.com>; Wojtek Porczyk <wo...@invisiblethingslab.com>

Subject: Re: Gramine Enclave size and performance

I think this was caused by running mismatched examples and Gramine versions - I guess your helloworld example code comes from the current master (which already contains https: //github. com/gramineproject/gramine/pull/1716), but you installed

[Saikat]

Hi Raghudeep,

"Hello World" works in all versions.

Also, my project is on hold now. 

Thanks & Regards,

Saikat

To view this discussion on the web visit https://groups.google.com/d/msgid/gramine-users/BY3PR15MB5026E49BB24E324EDCEA46EFD5F52%40BY3PR15MB5026.namprd15.prod.outlook.com.

[Michał Kowalczyk]

Default arch library directory is for some reason wrong for you. Please try rebuilding the example with ARCH_LIBDIR argument passed to Make and set to wherever that directory is located on your system (our Makefile guessed that it's /lib/x86_64-redhat-linux, but it seems it's wrong).

[Raghudeep Kannavara]

Thank you for the quick reply and help. It works when I add the ARCH_LIBDIR=/lib.

I am working through the other commands in this quick start guide: https://github.com/gramineproject/gramine/tree/master/CI-Examples/ra-tls-secret-prov#quick-start

I got the following error message on running "gramine-sgx ./client". Please see the output below. How to resolve this please?

error: AESM service returned error 30; this may indicate that infrastructure for the EPID attestation requested by Gramine is missing on this machine
error: load_enclave() failed with error: Operation not permitted (EPERM)

[meta@localhost secret_prov_pf]$ RA_TLS_EPID_API_KEY=1111111111111111111111111111111 ./server_epid wrap_key &
[1] 197166
--- Reading the master key for encrypted files from 'wrap_key' ---
--- Starting the Secret Provisioning server on port 4433 ---
[meta@localhost secret_prov_pf]$

[meta@localhost secret_prov_pf]$ ls
client  client.c  client.manifest  client.manifest.sgx  client.manifest.template  client.sig  enc_files  plain_files  server.c  server_epid  wrap_key

[meta@localhost secret_prov_pf]$
[meta@localhost secret_prov_pf]$ sudo gramine-sgx ./client
Gramine is starting. Parsing TOML manifest file, this may take some time...
error: AESM service returned error 30; this may indicate that infrastructure for the EPID attestation requested by Gramine is missing on this machine
error: load_enclave() failed with error: Operation not permitted (EPERM)

[meta@localhost secret_prov_pf]$ is-sgx-available

SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): false
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): false
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x1000000000
EPC size: 0x5d80000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
#PF/#GP information in EXINFO in MISC region of SSA supported: false
#CP information in EXINFO in MISC region of SSA supported: false

[meta@localhost secret_prov_pf]$

---

From: Michał Kowalczyk <mk...@invisiblethingslab.com>
Sent: Friday, May 24, 2024 2:58 PM

To: Raghudeep Kannavara <ragh...@meta.com>
Cc: us...@gramineproject.io <us...@gramineproject.io>; Vij, Mona <mona...@intel.com>; Wojtek Porczyk <wo...@invisiblethingslab.com>
Subject: Re: Gramine Enclave size and performance

Default arch library directory is for some reason wrong for you. Please try rebuilding the example with ARCH_LIBDIR argument passed to Make and set to wherever that directory is located on your system (our Makefile guessed that it's /lib/x86_64-redhat-linux,

[Michał Kowalczyk]

RA_TLS_EPID_API_KEY=1111111111111111111111111111111

Isn't this the problem? Or maybe you censored the key only here and you are actually using a correct API key on your machine?

Error 30 is "AESM_SERVICE_UNAVAILABLE": https://github.com/intel/linux-sgx/blob/80a6625c497056c43e78d993e414ca99a9efed5c/psw/ae/inc/internal/aesm_error.h#L65.

I'm not sure what could have caused it, but it seems that there's something wrong with your SGX PSW installation. What does "sudo service aesmd status" print for you?

Maybe Dmitrii (CC) will know what's exactly wrong.

[Raghudeep Kannavara]

Hi, thank you again, yes I had removed the key and replaced with ones.

Here is the output from sudo service aesmd status,please review.

● aesmd.service - Intel(R) Architectural Enclave Service Manager
     Loaded: loaded (/usr/lib/systemd/system/aesmd.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-05-23 01:22:43 EDT; 2 days ago
   Main PID: 179656 (aesm_service)
      Tasks: 4 (limit: 408986)
     Memory: 4.1M
        CPU: 249ms
     CGroup: /system.slice/aesmd.service
             └─179656 /opt/intel/sgx-aesm-service/aesm/aesm_service

May 23 01:22:43 localhost.localdomain groupadd[179494]: group added to /etc/group: name=sgx_prv, GID=1001
May 23 01:22:43 localhost.localdomain groupadd[179494]: group added to /etc/gshadow: name=sgx_prv
May 23 01:22:43 localhost.localdomain groupadd[179494]: new group: name=sgx_prv, GID=1001
May 23 01:22:43 localhost.localdomain usermod[179534]: add 'aesmd' to group 'sgx_prv'
May 23 01:22:43 localhost.localdomain usermod[179534]: add 'aesmd' to shadow group 'sgx_prv'
May 23 01:22:43 localhost.localdomain usermod[179643]: add 'aesmd' to group 'sgx'
May 23 01:22:43 localhost.localdomain usermod[179643]: add 'aesmd' to shadow group 'sgx'
May 23 01:22:43 localhost.localdomain aesm_service[179655]: aesm_service: warning: Turn to daemon. Use "--no-daemon" option to execute in foreground.
May 23 01:22:43 localhost.localdomain systemd[1]: Started Intel(R) Architectural Enclave Service Manager.
May 23 01:22:43 localhost.localdomain aesm_service[179656]: The server sock is 0x602f60

-- 

Best Regards,
Raghudeep Kannavara

Get Outlook for iOS

---

From: Michał Kowalczyk <mk...@invisiblethingslab.com>
Sent: Saturday, May 25, 2024 6:05:13 AM
To: Raghudeep Kannavara <ragh...@meta.com>
Cc: us...@gramineproject.io <us...@gramineproject.io>; Vij, Mona <mona...@intel.com>; Wojtek Porczyk <wo...@invisiblethingslab.com>; Kuvaiskii, Dmitrii <dmitrii....@intel.com>

Subject: Re: Gramine Enclave size and performance

RA_TLS_EPID_API_KEY=1111111111111111111111111111111 Isn't this the problem? Or maybe you censored the key only here and you are actually using a correct API key on your machine? Error 30 is "AESM_SERVICE_UNAVAILABLE": https: //github. com/intel/linux-sgx/blob/80a6625c497056c43e78d993e414ca99a9efed5c/psw/ae/inc/internal/aesm_error. h#L65.