GSC: Large enclave requirements

[Prateek Sahu]

Hi Dmitrii, Team,

I had posted an issue regarding deployment of a kubernetes sample application using GSC.

Thanks Dmitrii for helping me navigate the errors/issues.

Upon final deployment, I noticed that I needed to provide very large enclave_size and thread_num to get the application working - specifically, the enclave_size I am using are 1GB and thread_num are 16 (it crashes with 512MB and/or <12 threads).

I am curious to know if this is known and if it is due to the fact that we have to build the gsc app container on top of a complete ubuntu image since the application is a very simple go app and doesnot require such high resource requirements.

Thanks,

Prateek

[Kuvaiskii, Dmitrii]

“Very simple go app” doesn’t imply that it doesn’t use a lot of resources :)

 

https://povilasv.me/go-memory-management/

https://medium.com/a-journey-with-go/go-goroutine-os-thread-and-cpu-management-2f5a5eaf518a

 

TLDR: Google Go reserves a lot of virtual memory (-> big enclave size) and re-uses a pool of OS threads for goroutines (-> big enclave thread num).

 

In other words, it’s not your app that eats resources, it’s the Google Go runtime.

 

--

Dmitrii

Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva  
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

[King, Gordon]

Recall that most of such garbage-collection kind of language runtime (the dev. doesn’t need to release memory resource explicitly) e.g. JVM, Go, that utilized the mechanism of mmap to reserve a range of continuous memory address space as a memory pool but not mapping a  physical page to it till really using it. It is very clever trick to manage application specific memory resource while keeping memory allocation efficient and optimizable. Thus, in Linux native mode, you simple goapp shouldn’t consume a lot of memory resources but the language runtime will reserve huge amount of continuous virtual memory address space and trying to guess it is big enough to cover the needs of memory resources through the whole lifecycle of the go app. a smart language runtime can heuristically precisely guess it at compile time and also allow your to override it by compile time or runtime options. However, when are running the go application in GSGX enclave, it doesn’t work such way, because, so far, my understand is it doesn’t support memory space reserve, and it mapping physical memory pages into the reserved memory pools, otherwise failed at very beginning. that’s why you found it requires huge amount enclave size when you are trying run even very simple of goapp. There are several ways to handle this case as workaround solution.

1. Obviously, given a large enough of enclave container, but it become very wasteful due to the enclave size must be 2^n for memory intensive apps.
2. Tune the language runtime parameters associate with the size of memory reservation
3. The EDMM feature may be merged soon, but you can try it as experimental run, never use it in production environment at this moment.

https://github.com/oscarlab/graphene/pull/2190

--
You received this message because you are subscribed to the Google Groups "Graphene Support Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graphene-suppo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/graphene-support/PH0PR11MB5675CED9184DAB8FA50AEA52FFF09%40PH0PR11MB5675.namprd11.prod.outlook.com.

[Prateek Sahu]

Thanks all for the insights.

I am running the goapp in a container with a go runtime and the memory utilization of the container is pretty small.

The application I am running is also used in marblerun using SGX but has small resource requirements too, hence my confusion around the resource requirements.

But as Gordon explained, it is probable that using gsc, the entire go-runtime's memory requirements are encompassed into the gsc-container created. I probably would need to dig deeper into marblerun's deployments to understand what is causing their containers to be small as well as protecting the application (and probably the runtime) memory.

Also, I wanted to understand if the gsc-ubuntu container running inside an enclave or is it just the application inside the container running in an enclave?

Thanks,

Prateek

[King, Gordon]

They may use the Ego that is optimized for enclave environment.

https://github.com/edgelesssys/ego

[Kuvaiskii, Dmitrii]

As @King, Gordon mentioned, Marblerun developers use their own SGX framework called “EGo”. It is somehow optimized to run Golang applications, but I don’t know the details.

 

> Also, I wanted to understand if the gsc-ubuntu container running inside an enclave or is it just the application inside the container running in an enclave?

 

The second. It is the application inside the container. The container itself is *not* run inside of the SGX enclave.

 

--

Dmitrii

 

From: Prateek Sahu <prat...@utexas.edu>
Sent: Tuesday, August 3, 2021 7:31 PM
To: King, Gordon <gordo...@intel.com>
Cc: Kuvaiskii, Dmitrii <dmitrii....@intel.com>; sup...@graphene-project.io
Subject: Re: GSC: Large enclave requirements

 

Thanks all for the insights.

[Prateek Sahu]

This discussion has been incredibly helpful! Thank you both.

Thanks,

Prateek