Feasibility of running go application which listens as http server and launching itself as a web service using systemctl

46 views

Skip to first unread message

ManiV

unread,

Nov 16, 2021, 11:01:59 AM11/16/21

to Graphene Support Mailing List

Greetings to Everyone,

We are facing issue while trying to run/invoke a security based application( actually a web service which verifies certificates) written in go inside graphene.

The application is called from a caller program via execvp command.

Manifest file for the caller program has details of all the files and directories accessed by this application listed under allowed files,

and executable of the caller program and security application listed as a trusted file.

The idea is that - the caller program is made as the entry point to the libOS which inturn executes the security application via execvp.

Manifest file:

loader.preload = "file:/usr/local/lib/x86_64-linux-gnu/graphene/libsysdb.so"

libos.entrypoint = "gramine-sqvs"

loader.log_level = "debug"

loader.env.LD_LIBRARY_PATH = "/lib:/usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/libs"

loader.insecure__use_cmdline_argv = true

fs.mount.lib.type = "chroot"

fs.mount.lib.path = "/lib"

fs.mount.lib.uri = "file:/usr/local/lib/x86_64-linux-gnu/graphene/runtime/glibc"

fs.mount.lib2.type = "chroot"

fs.mount.lib2.path = "/lib/x86_64-linux-gnu"

fs.mount.lib2.uri = "file:/lib/x86_64-linux-gnu"

fs.mount.lib3.type = "chroot"

fs.mount.lib3.path = "/usr/lib/x86_64-linux-gnu"

fs.mount.lib3.uri = "file:/usr/lib/x86_64-linux-gnu"

fs.mount.etc.type = "chroot"

fs.mount.etc.path = "/etc"

fs.mount.etc.uri = "file:/etc"

sgx.remote_attestation = true

sgx.sqvs_spid = ""

sgx.sqvs_linkable = 0

sgx.trusted_files.gramine-sqvs= "file:gramine-sqvs"

sgx.trusted_files.sqvs= "file:sqvs"

sgx.trusted_files.runtime = "file:/usr/local/lib/x86_64-linux-gnu/graphene/runtime/glibc/"

sgx.trusted_files.arch_libdir = "file:/lib/x86_64-linux-gnu/"

sgx.trusted_files.usr_arch_libdir = "file:/usr//lib/x86_64-linux-gnu/"

sgx.trusted_files.libs = "file:./libs/"

sgx.trusted_files.cachain = "file:certs/test-ca-sha256.crt"

sgx.allowed-files = [

"file:/etc/nsswitch.conf",

"file:/etc/ethers",

"file:/etc/host.conf",

"file:/etc/hosts",

"file:/etc/group",

"file:/etc/passwd",

"file:/etc/gai.conf",

"file:/etc/resolv.conf",

"file:/opt/sqvs/sqvs.service",

"file:/var/log/sqvs/http.log",

"file:/var/log/sqvs/sqvs-security.log",

"file:/var/log/sqvs/sqvs.log",

"file:/root/binaries/env/sqvs.env",

"file:/root/sqvs.env",

"file:/etc/sqvs/tls.key",

"file:/etc/sqvs/tls-vert.pem",

"file:/etc/sqvs/config.yml",

"file:/etc/sqvs/sqvs_signing_pub_key.pem",

"file:/etc/sqvs/sqvs_signing_priv_key.pem",

"file:/etc/sqvs/certs/trustedSGXRootCA.pem",

"file:/etc/sqvs/certs/trustedjwt/",

"file:/etc/sqvs/certs/trustedca/202d1d4f2.pem"

]

The application uses systemctl internally to deploy itself as a webservice.

However, while executing the application inside graphene on sgx enabled machine, graphene fails to load, throws Internal memory fault error and exits.

debug: Token file: gramine-sqvs.token

debug: Read dummy DCAP token

debug: enclave created:

debug: base: 0x0000000010000000

debug: size: 0x0000000010000000

debug: misc_select: 0x00000000

debug: attr.flags: 0x0000000000000007

debug: attr.xfrm: 0x00000000000002e7

debug: ssa_frame_size: 4

debug: isv_prod_id: 0x00000000

debug: isv_svn: 0x00000000

debug: adding pages to enclave: 0x1ff7a000-0x20000000 [REG:R--] (manifest) measured

debug: adding pages to enclave: 0x1ff5a000-0x1ff7a000 [REG:RW-] (ssa) measured

debug: adding pages to enclave: 0x1ff56000-0x1ff5a000 [TCS:---] (tcs) measured

debug: adding pages to enclave: 0x1ff52000-0x1ff56000 [REG:RW-] (tls) measured

debug: adding pages to enclave: 0x1ff12000-0x1ff52000 [REG:RW-] (stack) measured

debug: adding pages to enclave: 0x1fed2000-0x1ff12000 [REG:RW-] (stack) measured

debug: adding pages to enclave: 0x1fe92000-0x1fed2000 [REG:RW-] (stack) measured

debug: adding pages to enclave: 0x1fe52000-0x1fe92000 [REG:RW-] (stack) measured

debug: adding pages to enclave: 0x1fe42000-0x1fe52000 [REG:RW-] (sig_stack) measured

debug: adding pages to enclave: 0x1fe32000-0x1fe42000 [REG:RW-] (sig_stack) measured

debug: adding pages to enclave: 0x1fe22000-0x1fe32000 [REG:RW-] (sig_stack) measured

debug: adding pages to enclave: 0x1fe12000-0x1fe22000 [REG:RW-] (sig_stack) measured

debug: adding pages to enclave: 0x1b9f7000-0x1ba39000 [REG:R-X] (code) measured

debug: adding pages to enclave: 0x1ba39000-0x1ba3d000 [REG:RW-] (data) measured

debug: adding pages to enclave: 0x1ba3d000-0x1fe12000 [REG:RW-] (bss) measured

debug: adding pages to enclave: 0x10000000-0x1b9f7000 [REG:RWX] (free)

debug: enclave initializing:

debug: enclave id: 0x000000001ffff000

debug: mr_enclave: bb7049070b34abe47c011eb5647c55a997d8af533aa65991b26a845d8d396fe5

debug: Using SGX DCAP/ECDSA attestation

error: Using insecure argv source. Graphene will continue application execution, but this configuration must not be used in production!

warning: DkVirtualMemoryProtect is unimplemented in Linux-SGX PAL

[::] debug: Host: Linux-SGX

[::] debug: LibOS xsave_enabled 1, xsave_size 0xa88(2696), xsave_features 0x2e7

[::] debug: Initial VMA region 0x1b97d000-0x1b9f7000 (LibOS) bookkeeped

[::] debug: Initial VMA region 0x1ff7a000-0x20000000 (manifest) bookkeeped

[::] debug: ASLR top address adjusted to 0x1adf1000

[::] debug: Shim loaded at 0x1b97d000, ready to initialize

[::] debug: Mounting root as chroot filesystem: from file:. to /

[::] debug: Mounting special proc filesystem: /proc

[::] debug: Mounting special dev filesystem: /dev

[::] debug: Mounting terminal device /dev/tty under /dev

[::] debug: Mounting special sys filesystem: /sys

[P1:T1:] debug: Mounting as chroot filesystem: from file:/usr/local/lib/x86_64-linux-gnu/graphene/runtime/glibc to /lib

[P1:T1:] debug: Mounting as chroot filesystem: from file:/etc to /etc

[P1:T1:] debug: Mounting as chroot filesystem: from file:/lib/x86_64-linux-gnu to /lib/x86_64-linux-gnu

[P1:T1:] debug: Mounting as chroot filesystem: from file:/usr/lib/x86_64-linux-gnu to /usr/lib/x86_64-linux-gnu

[P1:T1:gramine-sqvs] debug: Allocating stack at 0x0 (size = 262144)

[P1:T1:gramine-sqvs] debug: loading "file:./gramine-sqvs"

[P1:T1:gramine-sqvs] debug: find_interp: searching for interpreter: /lib/ld-linux-x86-64.so.2

[P1:T1:gramine-sqvs] debug: loading "file:/usr/local/lib/x86_64-linux-gnu/graphene/runtime/glibc/ld-linux-x86-64.so.2"

[P1:T1:gramine-sqvs] debug: Creating pipe: pipe.srv:1

debug: sock_getopt (fd = 12, sockopt addr = 0x7fff913233b0) is not implemented and always returns 0

[P1:T1:gramine-sqvs] debug: Shim process initialized

[P1:shim] debug: IPC worker started

[P1:T1:gramine-sqvs] debug: glibc register library /lib/libc.so.6 loaded at 0x1ab82000

[P1:T1:sqvs] debug: Allocating stack at 0x0 (size = 262144)

[P1:T1:sqvs] debug: loading "file:./sqvs"

[P1:T1:sqvs] debug: execute_loadcmd: failed to map segment

[P1:T1:sqvs] error: Internal memory fault at 0x00000000 (IP = +0x4e7f7, VMID = 1, TID = 1)

debug: DkProcessExit: Returning exit code 1

Could someone throw some light on feasibility of executing an existing application( which listens as http server and launching itself as a web serviceusing systemctl ) inside graphene and help with debugging this issue?

Thanks & Regards,

Manikandan.V

ManiV

unread,

Nov 17, 2021, 9:28:00 AM11/17/21

to Graphene Support Mailing List

Hi Everyone,

we tried the same with latest gramine code.

We tried with commit 1fd075ee309af308d75a474156518f3b9807567d (HEAD -> master, origin/master, origin/HEAD), now seeing ...fatal error: failed to reserve page summary memory...

Any suggestions to resolve/understand this would be helpful. Logs are are below..

Thanks & Regards,

Manikandan.V

debug: Token file: gramine-sqvs.token
debug: Read dummy DCAP token
debug: enclave created:

debug: base: 0x0000000000000000

debug: size: 0x0000000010000000
debug: misc_select: 0x00000000

debug: attr.flags: 0x0000000000000005
debug: attr.xfrm: 0x00000000000000e7

debug: ssa_frame_size: 4
debug: isv_prod_id: 0x00000000
debug: isv_svn: 0x00000000

debug: adding pages to enclave: 0xff79000-0x10000000 [REG:R--] (manifest) measured
debug: adding pages to enclave: 0xff59000-0xff79000 [REG:RW-] (ssa) measured
debug: adding pages to enclave: 0xff55000-0xff59000 [TCS:---] (tcs) measured
debug: adding pages to enclave: 0xff51000-0xff55000 [REG:RW-] (tls) measured
debug: adding pages to enclave: 0xff11000-0xff51000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0xfed1000-0xff11000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0xfe91000-0xfed1000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0xfe51000-0xfe91000 [REG:RW-] (stack) measured
debug: adding pages to enclave: 0xfe41000-0xfe51000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0xfe31000-0xfe41000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0xfe21000-0xfe31000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0xfe11000-0xfe21000 [REG:RW-] (sig_stack) measured
debug: adding pages to enclave: 0xf9e4000-0xfa39000 [REG:R-X] (code) measured
debug: adding pages to enclave: 0xfa39000-0xfa3d000 [REG:RW-] (data) measured
debug: adding pages to enclave: 0xfa3d000-0xfe11000 [REG:RW-] (bss) measured
debug: adding pages to enclave: 0x10000-0xf9e4000 [REG:RWX] (free)
debug: enclave initializing:
debug: enclave id: 0x000000000ffff000
debug: mr_enclave: ece968e3780e658a266b77fb4e77159ab856b36e17897d0cea6f71f115029176

debug: Using SGX DCAP/ECDSA attestation

-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

- loader.log_level = warning|debug|trace|all (verbose log level, may leak information)
- loader.insecure__use_cmdline_argv = true (forwarding command-line args from untrusted host to the app)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

warning: DkVirtualMemoryProtect is unimplemented in Linux-SGX PAL
[::] debug: Host: Linux-SGX

[::] debug: LibOS xsave_enabled 1, xsave_size 0xa80(2688), xsave_features 0xe7
[::] debug: Initial VMA region 0xf51b000-0xf59c000 (LibOS) bookkeeped
[::] debug: Initial VMA region 0xff79000-0x10000000 (manifest) bookkeeped
[::] debug: ASLR top address adjusted to 0x7bbc000
[::] debug: Shim loaded at 0xf51b000, ready to initialize

[::] debug: Mounting root as chroot filesystem: from file:. to /
[::] debug: Mounting special proc filesystem: /proc
[::] debug: Mounting special dev filesystem: /dev
[::] debug: Mounting terminal device /dev/tty under /dev

[P1:T1:] debug: Mounting as chroot filesystem: from file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc to /lib

[P1:T1:] debug: Mounting as chroot filesystem: from file:/etc to /etc
[P1:T1:] debug: Mounting as chroot filesystem: from file:/lib/x86_64-linux-gnu to /lib/x86_64-linux-gnu
[P1:T1:] debug: Mounting as chroot filesystem: from file:/usr/lib/x86_64-linux-gnu to /usr/lib/x86_64-linux-gnu

[P1:T1:] debug: Mounting as chroot filesystem: from file:/root/mohanpa/gramine/CI-Examples/gramine-sqvs/gramine-sqvs to /root/mohanpa/gramine/CI-Examples/gramine-sqvs
[P1:T1:gramine-sqvs] debug: Allocating stack at 0 (size = 262144)

[P1:T1:gramine-sqvs] debug: loading "file:./gramine-sqvs"
[P1:T1:gramine-sqvs] debug: find_interp: searching for interpreter: /lib/ld-linux-x86-64.so.2

[P1:T1:gramine-sqvs] debug: loading "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2"

[P1:T1:gramine-sqvs] debug: Creating pipe: pipe.srv:1

debug: sock_getopt (fd = 12, sockopt addr = 0x7ffc0050b810) is not implemented and always returns 0

[P1:T1:gramine-sqvs] debug: Shim process initialized
[P1:shim] debug: IPC worker started

[P1:T1:gramine-sqvs] debug: glibc register library /lib/libc.so.6 loaded at 0x0794d000
[P1:T1:sqvs] debug: Allocating stack at 0 (size = 262144)

[P1:T1:sqvs] debug: loading "file:./sqvs"

[P1:T1:sqvs] debug: find_interp: searching for interpreter: /lib/ld-linux-x86-64.so.2
[P1:T1:sqvs] debug: loading "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2"
[P1:T1:sqvs] debug: execve: start execution
[P1:T1:sqvs] debug: glibc register library /lib/libpthread.so.0 loaded at 0x07b81000
[P1:T1:sqvs] debug: glibc register library /lib/libc.so.6 loaded at 0x0798e000
fatal error: failed to reserve page summary memory

runtime stack:
runtime.throw(0x85600b, 0x25)
/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0x7949d58 sp=0x7949d28 pc=0x4345d2
runtime.(*pageAlloc).sysInit(0xb97f48)
/usr/local/go/src/runtime/mpagealloc_64bit.go:80 +0x17e fp=0x7949de8 sp=0x7949d58 pc=0x42b4de
runtime.(*pageAlloc).init(0xb97f48, 0xb97f40, 0xbac5f8)
/usr/local/go/src/runtime/mpagealloc.go:297 +0x75 fp=0x7949e10 sp=0x7949de8 pc=0x429135
runtime.(*mheap).init(0xb97f40)
/usr/local/go/src/runtime/mheap.go:694 +0x274 fp=0x7949e38 sp=0x7949e10 pc=0x426354
runtime.mallocinit()
/usr/local/go/src/runtime/malloc.go:470 +0xff fp=0x7949e68 sp=0x7949e38 pc=0x40c8ef
runtime.schedinit()
/usr/local/go/src/runtime/proc.go:545 +0x60 fp=0x7949ec0 sp=0x7949e68 pc=0x437cc0
runtime.rt0_go(0x7949ef8, 0x1, 0x7949ef8, 0x7949e48, 0x7b8a0aa, 0x1, 0x7949ffa, 0x0, 0x7949faf, 0x0, ...)
/usr/local/go/src/runtime/asm_amd64.s:214 +0x125 fp=0x7949ec8 sp=0x7949ec0 pc=0x4620d5
[P1:T1:sqvs] debug: ---- shim_exit_group (returning 2)
[P1:T1:sqvs] debug: clearing POSIX locks for pid 1
[P1:T1:sqvs] debug: sync client shutdown: closing handles
[P1:T1:sqvs] debug: sync client shutdown: waiting for confirmation
[P1:T1:sqvs] debug: sync client shutdown: finished
[P1:shim] debug: IPC worker: exiting worker thread
[P1:T1:sqvs] debug: process 1 exited with status 2
debug: DkProcessExit: Returning exit code 2

Reply all

Reply to author

Forward

0 new messages