Add TUN/TAP device to GCS Container

[Raouf Rokhjavan]

I try to create a secure channel inside enclave using Gramine; in better words, inside the GSC. For this purpose, I need to redirect the application traffic to my tunnel. I plan to port network stack like lwIP to the userspace/enclave and protect the whole network stack and wrap the packets using TLS tunnel, but for now, I am trying to use TUN/TAP device for making tunnel as the first step. Accordingly, I tried to add a TUN/TAP device to the GSC container by invoking " ip tuntap add mode tun name tun0" command inside GSC to make sure it works, and develop my tunnel using watertap/dtls package in Go, but it returns an error.

Cannot open netlink socket: Function not implemented

ERROR: 1

Gramine logs also returns:

...

[[P1:T1:ip] warning: Unsupported system call prctl

[P1:T1:ip] warning: Unsupported system call prctl
[P1:T1:ip] warning: Unsupported system call prctl
[P1:T1:ip] warning: Unsupported system call prctl
[P1:T1:ip] warning: Unsupported system call prctl
[P1:T1:ip] warning: Unsupported system call prctl
[P1:T1:ip] trace: ---- shim_getuid() = 0x0
[P1:T1:ip] warning: shim_socket: unknown socket domain 16
[P1:T1:ip] trace: ---- shim_socket(NETLINK, SOCK_CLOEXEC|RAW, 0) = -38
[P1:T1:ip] trace: ---- shim_dup(2) = 0x3
[P1:T1:ip] trace: ---- shim_fcntl(3, F_GETFL, 0x18b101720) = 0x401
[P1:T1:ip] trace: ---- shim_close(3) = 0x0
[P1:T1:ip] trace: ---- shim_write(2, 0x18b188130, 0x35) ...
[P1:T1:ip] trace: ---- return from shim_write(...) = 0x35
[P1:T1:ip] debug: ---- shim_exit_group (returning 1)
[P1:T1:ip] debug: clearing POSIX locks for pid 1
[P1:T1:ip] debug: sync client shutdown: closing handles
[P1:T1:ip] debug: sync client shutdown: waiting for confirmation
[P1:T1:ip] debug: sync client shutdown: finished
[P1:shim] debug: IPC worker: exiting worker thread
[P1:T1:ip] debug: process 1 exited with status 1

It is necessary to mention that the entrypoint of the container is "/bin/bash -l -c". 

I could add tun0 TUN device to the docker container using "--entrypoint /bin/bash" argument to make sure what I am doing is right, but when it comes to the GSC, it complains. 

I am suspicious about this line, but I cannot understand it completely:

[P1:T1:ip] trace: ---- shim_socket(NETLINK, SOCK_CLOEXEC|RAW, 0) = -38

What it means? and What's the possible solution for this problem?

Thank you,

Raouf

[Borys]

> I try to create a secure channel inside enclave using Gramine; in better
> words, inside the GSC. For this purpose, I need to redirect the application
> traffic to my tunnel. I plan to port network stack like lwIP to the
> userspace/enclave and protect the whole network stack and wrap the packets
> using TLS tunnel, but for now, I am trying to use TUN/TAP device for making
> tunnel as the first step. Accordingly, I tried to add a TUN/TAP device to
> the GSC container by invoking " ip tuntap add mode tun name tun0" command
> inside GSC to make sure it works, and develop my tunnel using watertap/dtls
> package in Go, but it returns an error.
>

> *Cannot open netlink socket: Function not implemented*

Netlink sockets are not supported (and there are no plans of supporting them atm).

Your best bet would be to either just use TLS with TCP if your usecase allows for it (i.e. send data directly, not tunnel whole IP packets) or tunnel your packets over UDP/TCP.

> *[[P1:T1:ip] warning: Unsupported system call prctl*
You can ignore this warning.

> I am suspicious about this line, but I cannot understand it completely:

> *[P1:T1:ip] trace: ---- shim_socket(NETLINK, SOCK_CLOEXEC|RAW, 0) = -38*
> What it means?

It means that Gramine does not implement this syscall.

Borys

[Raouf Rokhjavan]

Thank you. In fact, I am trying to run an unmodified application in Gramine, and tunnel its traffic using DTLS tunnel. Because of that, I was thinking about TUN/TAP to redirect IP packets to my DTLS tunnel.

Accordingly, I think what I'm doing is the second case you mentioned above, but how can I redirect all network traffic to my tunnel over UDP while TUN/TAP is not allowed to be used in Gramine? Any suggestion or related work has done?

Raouf

[Raouf Rokhjavan]

As you said that netlink was not implemented in Gramine, I decided to find an alternative to manage TUN/TAP devices, so I found tunectl which uses ioctl. I changed the manifest file to this to access /dev/net/tun device file:

...

fs.mounts = [
    { type = "tmpfs", path = "/tmp" },
    { type = "chroot", path = "/dev/net", uri="file:/dev/net" },
]

sgx.allowed_files = [
    "file: /logs/",
    "file: /dev/net/",
]
...

but, I get this error 

TUNSETIFF: Function not implemented

...

[P1:T1:tunctl] trace: ---- shim_openat(AT_FDCWD, "/dev/net/tun", O_RDWR, 0000) = 0x3
[P1:T1:tunctl] trace: ---- shim_ioctl(3, OP 0x400454ca, 0x112f75c60) ...
[P1:T1:tunctl] trace: ---- return from shim_ioctl(...) = -38
[P1:T1:tunctl] trace: ---- shim_dup(2) = 0x4
[P1:T1:tunctl] trace: ---- shim_fcntl(4, F_GETFL, 0x1132c6720) = 0x401
[P1:T1:tunctl] trace: ---- shim_close(4) = 0x0
[P1:T1:tunctl] trace: ---- shim_write(2, 0x112f730f0, 0x24) ...
[P1:T1:tunctl] trace: ---- return from shim_write(...) = 0x24
[P1:T1:tunctl] debug: ---- shim_exit_group (returning 1)
[P1:T1:tunctl] debug: clearing POSIX locks for pid 1
[P1:T1:tunctl] debug: sync client shutdown: closing handles
[P1:T1:tunctl] debug: sync client shutdown: waiting for confirmation
[P1:T1:tunctl] debug: sync client shutdown: finished

[P1:shim] debug: IPC worker: exiting worker thread

[P1:T1:tunctl] debug: process 1 exited with status 1
debug: DkProcessExit: Returning exit code 1

So, does it mean that this ioctl isn't supported in Gramine? If so, how can I deal with TUN/TAP devices in Gramine?

Raouf

[Borys]

> Thank you. In fact, I am trying to run an unmodified application in
> Gramine, and tunnel its traffic using DTLS tunnel. Because of that, I was
> thinking about TUN/TAP to redirect IP packets to my DTLS tunnel.
> Accordingly, I think what I'm doing is the second case you mentioned above,
> but how can I redirect all network traffic to my tunnel over UDP while
> TUN/TAP is not allowed to be used in Gramine? Any suggestion or related
> work has done?

I meant that you have to do it on application level, but since you are running an unmodified app, then it won't work.

> As you said that netlink was not implemented in Gramine, I decided to find
> an alternative to manage TUN/TAP devices, so I found tunectl which uses
> ioctl. I changed the manifest file to this to access /dev/net/tun device
> file:

That won't work either. For now there is no way to issue arbitrary ioctls; there is some work to make this possible but it will take time.

Anyway, that wouldn't solve the problem. What you are trying to do is to have a secure tunnel with one end terminated inside the enclave. As such, it must be handled completely within Gramine, which means you would have to run all of the code handling the packets and managing the tunnel inside Gramine (basically do user-space networking). This is not supported (as you already noticed) at the moment due to various reasons. You can go with your original plan of porting a network stack to run inside enclave - that might work and be your best chance, but also there might be some major obstacles (I don't know without going into much details). Unfortunately I suspect it won't - hard to imagine how to do it without creating custom interfaces installing custom routing etc. (which would require netlink support).

Borys

[Borys]

Sorry Raouf Rokhjavan <rokhj...@gmail.com>, I've replied to the list only (not sure if you are subscribed there). You can find the replay below.
Borys