Using Go in Apple network extensions

[ala...@google.com]

Hi all,

Since many of you have experience developing mobile VPN tools, I wanted to take the opportunity to ask about Go usage in Apple network extensions. 

I work on Outline, an easy to deploy VPN client/server powered by Shadowsocks. My team is looking into migrating some of our clients' components, mostly written in C, to Go. 

Recently, we were able to replace tun2socks, one of our core components, to a Go implementation (go-tun2socks, go-tun2socks-ios) in a prototype Outline iOS client.

However, Apple imposes a memory restriction of 15MB to VPN extensions, so running Go is somewhat of a stretch since the runtime itself takes ~5MB of memory. 

We were able to reduce memory usage in go-tun2socks, but that proved insufficient. The prototype VPN process eventually hits the limit and it's terminated by the OS.

I was wondering if someone had pointers or knew of projects using Go in Apple network extensions?

Thank you,

alberto

[Joseph Lorenzo Hall]

I've heard that Wireguard struggled with this a ton but seem to have succeeded... I alas don't know more. best, Joe

--
You received this message because you are subscribed to the Google Groups "Network Traffic Obfuscation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to traffic-obf...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Joseph Lorenzo Hall
Chief Technologist, Center for Democracy & Technology [https://www.cdt.org]
1401 K ST NW STE 200, Washington DC 20005-3497
e: j...@cdt.org, p: 202.407.8825, pgp: https://josephhall.org/gpg-key
Fingerprint: 3CA2 8D7B 9F6D DBD3 4B10  1607 5F86 6987 40A9 A871

Don't miss out! CDT's Tech Prom is April 10, 2019, at The
Anthem. Please join us: https://cdt.org/annual-dinner/

[Brandon Wiley]

Yes, the WireGuard Network Extension for iOS is on the forefront of this struggle. You can find the code here: https://github.com/WireGuard/wireguard-ios

The main thing to note is the wireguard-go-bridge Makefile here: https://github.com/WireGuard/wireguard-ios/blob/master/wireguard-go-bridge/Makefile It patches and then builds a customized Go runtime.

Overall, I've found that the best plan is to forego trying to use Go code inside of a Network Extension and instead wrap C code or rewrite in Swift. For Pluggable Transports, we maintain Swift and Go versions of the most popular transports, for this reason. However, if you're committed to an existing Go codebase, then the WireGuard team has done a great job of showing just how far you can take this approach.

[Vinicius Fortuna [vee-NEE-see.oos]]

I found this exploration on how to limit memory usage in Go:

https://docs.google.com/document/d/1zn4f3-XWmoHNj702mCCNvHqaS7p9rzqQGa74uOwOBKM

It gives some good insight on the problem, but it's from 2017 and I don't know what the current status is.

The main issues are:

- We need a way to run the GC more often when there's memory pressure

- We need to know when there's memory pressure  so we can handle it in the application without crying (e.g. fail connections)

On Wed, Feb 6, 2019 at 6:32 PM alalama via Network Traffic Obfuscation <traff...@googlegroups.com> wrote:

[Ox Cart]

Something to keep in mind about WireGuard is that its memory requirements are naturally already low. It works at the IP packet level without any TCP connection tracking or anything like that, so there's very little persistent state to keep on the heap.

[Vinicius Fortuna [vee-NEE-see.oos]]

That's a good point. In Outline we need a buffer for each TCP connection + lwip's own buffering.

We are using a buffer pool, which limits the number of allocations. However that only slows down hitting the 15 MB limit and having the extension killed, even if only 10MB is actually active.

I wish the GC was called before the process is killed.

The baseline memory needed is also quite high. Our extension starts at 5MB before seeing any data.

[Rod Hynes]

Psiphon has an iOS VPN app using our core, cross-platform Go networking code in a NEPacketTunnelProvider. It's been in production for over a year: https://itunes.apple.com/us/app/psiphon/id1276263909.

Psiphon requires a fair bit of memory for its circumvention operations, as it concurrently connects to several servers, using a variety of obfuscation protocols, until it establishes a successful connection. Given the 15MB limit, this leaves very little headroom so we made the decision to switch to tunneling IP packets to avoid allocating resources per tunneled TCP flow. Most of our obfuscation protocols run over TCP, so we did some performance testing due to TCP-over-TCP concerns (https://duckduckgo.com/?q=tcp-over-tcp) and found performance to be acceptable in practise.

Psiphon for iOS uses the same Go code as other platforms, although we dial back and stagger the concurrent operations to reduce peak memory spikes. We use the stock Go compiler and runtime.

We set aggressive garbage collection via https://golang.org/pkg/runtime/debug/#SetGCPercent and retain this aggressive setting for the lifetime of the extension. While our memory spikes significantly during [re]connection, many of the network protocol packages we use continuously allocate slices per network message in some way or other.

Depending on what you use, you will need to be aware of large allocations performed by the Go standard library -- such as in "tls" ("x509" actually) and "http2", which we use in some of our obfuscation protocols.

The Go GC marks heap space as available and doesn't immediately return that memory to the OS. 

We also use https://golang.org/pkg/runtime/debug/#FreeOSMemory in an effort to speed up that process, although with mixed success. Under the hood this calls "madvise" which the OS may or may not adhere to.

In addition to the heap profiler, frequent logging of https://golang.org/pkg/runtime/#ReadMemStats values has been helpful at identifying allocation spikes. Although none of those values come close to mapping to the number used for the 15MB limit. Here's a video that has some useful info about the memory limits in extension: https://developer.apple.com/videos/play/wwdc2018/416/.

Finally, when all else fails, you can depend on the "on demand" feature to restart the VPN if it gets killed:

https://developer.apple.com/documentation/networkextension/neondemandruleconnect. Of course, you'll want your extension to stay up for a decent while before going through a kill/restart cycle.

[Adam Fisk]

Thanks for the great write up Rod! Very helpful.

-Adam

--

You received this message because you are subscribed to the Google Groups "Network Traffic Obfuscation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to traffic-obf...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
President
Brave New Software Project, Inc.
https://www.getlantern.org

A998 2B6E EF1C 373E 723F A813 045D A255 901A FD89

[Alberto Lalama]

Thanks for the helpful pointers and suggestions. 

I see Wireguard modifies the Go runtime to use a boottime instead of monotonic clock, does this affect memory efficiency? We are currently using gomobile to create the Obj-C bindings.

We have also experimented with SetGCPercent and FreeOSMemory. @Rod Hynes, do you use a heuristic to call FreeOSMemory? We have experimented with ReadMemStats to call FreeOSMemory when memory pressure is high. However, we have seen inconsistent memory stats, as the function seems to report the main application's memory, and not the VPN extension's.

Again, thanks to everyone.

Best,

alberto

[Rod Hynes]

On Mon, Feb 11, 2019 at 11:40 AM ‘Alberto Lalama’ via Network Traffic Obfuscation traff...@googlegroups.com wrote:

We have also experimented with SetGCPercent and FreeOSMemory. @Rod Hynes, do you use a heuristic to call FreeOSMemory? We have experimented with ReadMemStats to call FreeOSMemory when memory pressure is high. However, we have seen inconsistent memory stats, as the function seems to report the main application's memory, and not the VPN extension's.

We call SetGCPercent with an aggressive setting to deal with constant, small memory allocations that occur throughout our code base.

We use FreeOSMemory proactively, calling it between memory intensive operations (e.g, establish a connection to a server) with the expectation that this will cause heap to be reclaimed and reused before subsequent operations which remain blocked until FreeOSMemory returns.

Per https://golang.org/doc/go1.9#gc, only the goroutine calling FreeOSMemory blocks, so we make FreeOSMemory calls in schedulers that launch/coordinate other goroutines.

(I think that in many cases we could simply call GC instead of FreeOSMemory. FreeOSMemory calls GC and maybe returns some memory to the OS, which I’m not sure happens often. Either way, what we’re attempting to do in this case is synchronously complete some garbage collection before allocating more memory.)

[Alberto Lalama]

Thank you for further explaining, Rod. I recently found this (Darwin) patch to the Go runtime, scheduled to be released with Go 1.13, that improves how memory is released to the OS. It changes the flags when calling madvise in FreeOSMemory. We have confirmed that iOS reports memory usage more accurately with this patch. 

I hope this can help Psiphon. See this bug for more details.

Best,

alberto

--