dynamic compilation in go 1.5

[Liam Breck]

I have an application plan that requires in-app compilation of go source to a loadable module, and sandboxed execution of the module in the existing go app/process. I'm hoping that might be possible with 1.5.

On Monday, December 1, 2014 8:51:46 PM UTC-8, Rob Pike wrote:

the compiler, assembler, linker, and runtime for 1.5 will be entirely in Go.

The 1.5 compiler proposal states:

Use some tools, perhaps derived from gofix and the Go oracle to split the compiler into packages

Could a compiler package be used by a go app to compile go source?

There is also a plan for 1.5 to enable packaging the go library as a shared lib. Could this be leveraged to build a sandbox library which dynamically loaded code links against?

Finally will there be dlopen & dlsym equivalents to enable loading just-compiled modules?

[Scott Redig]

Look into native client, NACL.  The go playground compiles to NACL, which it is able to run in a sandboxed environment.  Of course, this requires having a go compiler on the computer compiling it to NACL, but even if it were currently all written in Go that would be the case.

[Liam]

NaCl provides the sandbox and presumably dlopen features, but you don't want each NaCl module statically linked with the go lib functions it calls.

Does NaCl resolve calls to functions outside a module, in a shared lib or the running app?
Could one build the complete go lib into an NaCl host app?

[Ian Lance Taylor]

On Mon, Dec 15, 2014 at 3:44 PM, Liam Breck <networ...@gmail.com> wrote:
>
> Could a compiler package be used by a go app to compile go source?

This is not a goal. However, it may be possible.

> There is also a plan for 1.5 to enable packaging the go library as a shared
> lib. Could this be leveraged to build a sandbox library which dynamically
> loaded code links against?

You'll also have to run the linker. Like the compiler, an in-app
linker is not a goal, but may be possible.

To be clear, you will almost certainly need some sort of file system
available.

> Finally will there be dlopen & dlsym equivalents to enable loading
> just-compiled modules?

I hope so but I don't know if it will be in 1.5.

Ian

[minux]

On Mon, Dec 15, 2014 at 8:16 PM, Liam <networ...@gmail.com> wrote:

NaCl provides the sandbox and presumably dlopen features, but you don't want each NaCl module statically linked with the go lib functions it calls.

Does NaCl resolve calls to functions outside a module, in a shared lib or the running app?
Could one build the complete go lib into an NaCl host app?

I think sandboxing and allowing to call external functions are fundamentally incompatible goals.

To sandbox a process, you really need to limit any direct access to the outside world.

That is, in process sandboxing is very hard, and insecure. A better solution is probably just to

use the nacl port and pass in a fd to the nacl process than use some kind of rpc protocol to

communicate with the host process.

Even if Go 1.5 provides packages for the compiler and linkers that you could link into your

application, to build Go source files, you still need either a copy of the standard library source

or precompiled version of said source.

[Dmitry Vyukov]

Have you considering just building a separate program, running it as a
separate processes and communicating with it via some IPC mechanism
(if you have just input/output args, then it can be quite trivial)? It
is simple and works today.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Joshua]

You may want to consider embedding Lua.

Joshua

[Liam Breck]

On Mon, Dec 15, 2014 at 7:59 PM, minux <mi...@golang.org> wrote:

On Mon, Dec 15, 2014 at 8:16 PM, Liam <networ...@gmail.com> wrote:

Does NaCl resolve calls to functions outside a module, in a shared lib or the running app?
Could one build the complete go lib into an NaCl host app?

...

Even if Go 1.5 provides packages for the compiler and linkers that you could link into your

application, to build Go source files, you still need either a copy of the standard library source

or precompiled version of said source.

I see. NaCl permits loading shared libraries, but the current Go implementation only runs standalone apps.

Seems like I should defer sandboxing and have dynamically compiled code link against a go shared lib.

[Liam Breck]

On Tue, Dec 16, 2014 at 7:20 AM, Joshua <jha...@gmail.com> wrote:

You may want to consider embedding Lua.

 

Ha! Um, no. What makes you imply that Lua is an acceptable facsimile of Go?

[Liam Breck]

On Tue, Dec 16, 2014 at 1:11 AM, Dmitry Vyukov <dvy...@google.com> wrote:

Have you considering just building a separate program, running it as a
separate processes and communicating with it via some IPC mechanism
(if you have just input/output args, then it can be quite trivial)?

This approach would not allow a module to serve as a library for another module.

And representing a large API surface in RPC requires a request broker.

[Dmitri Shuralyov]

> To be clear, you will almost certainly need some sort of file system 

> available.

Would it make sense for the Go compiler and linker to use the vfs.FileSystem interface, rather than hardcoding normal OS file system access? That would make it possible to supply a virtual file system with any underlying implementation.

[Dmitri Shuralyov]

Alternatively, it could use http://godoc.org/go/build#Context which allows one to replace the default OS file system access with a custom implementation of ReadDir, OpenFile, etc. (it's easy to create a build.Context that is backed by a vfs.FileSystem).