Runtime code generation?

[zeilund]

Hi gophers,

I was studying runtime code generation and was wondering if there's a way to generate a function at runtime.

Let's say we have a function: fun(a,b) {return a+b}

This function is parsed to our program as a string, parsed and compiled to a function in go.

In C# and java you can do this as you have access to the clr and jvm which can jit compile your function for you.

Now go is a statically compiled language and as far as I can see the reflect package is not powerful enough for such operations.

Are my assumptions correct that this is not currently possible?
Could it be done through cgo?  (If yes a hint to c documentation would be nice :) )
Would it be possible in go in the future?

Creative ideas are welcome but prefer portable solutions so no assembly generation through c or such :)

For performance reasons it wouldn't be beneficial to have a. Net runtime running that you sent a million requests to.

Kind regards,
Jzs.

[Kevin Gillette]

It's not currently possible, and while the runtime used to have limited code generation to support closures, the internal representation of function pointers has changed, and the runtime no longer generates any code (and on supported platforms, Go allocated memory pages are marked NOEXEC).

There is certainly an interest in supporting hot-pluggable code in Go. It will likely be a couple years before any of that would become streamlined and usable.

[Carlos Cobo]

The reflect package can't do what? http://golang.org/pkg/reflect/#MakeFunc

[Kevin Gillette]

That doesn't generate code -- it's a wrapper that allows a reflect-based generic implementation to be exposed through a conventonal non-reflect, non interface{} based func signature

[zeilund]

Thanks kevin,

I appreciate the answer :)

As long asgo marks memory pages as noexec then I'm out of luck. I am not proposing to mark all memory pages as executable as this would clearly be a bad idea.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Carlos Cobo]

Ok, reflect doesn't generate code but it does indeed, quoting, "generate a function at runtime".

Aside: no hot-plugging code in Go, right. You can generate code files at runtime, integrate them with RPC support and run those files (something like rsc's webdev), not hot-plugging but something "similar".

[adon...@google.com]

On Friday, 30 August 2013 16:21:18 UTC-4, jzs wrote:

Hi gophers,

I was studying runtime code generation and was wondering if there's a way to generate a function at runtime.

Let's say we have a function: fun(a,b) {return a+b}

This function is parsed to our program as a string, parsed and compiled to a function in go.

As others point out, you can't generate executable memory segments---but you could always write an interpreter.   Go itself isn't ideally suited to interpretation but there is no fundamental reason you couldn't implement the complete language spec that way.  It would be quite a lot of work---take a look at ssa/interp (in the go.tools repo) for a minimal (and horribly inefficient) interpreter for a large subset of Go.

[jan...@gmail.com]

Why we can't generate executable memory segments? JIT runtimes just doing this I think.

[Ian Lance Taylor]

On Wed, May 10, 2017 at 12:19 AM, <jan...@gmail.com> wrote:
>
> Why we can't generate executable memory segments? JIT runtimes just doing
> this I think.

Generating executable memory segments in a language that supports
converting integers to pointers opens up a wide attack surface in a
buggy program. An attacker who can get any sort of control over the
program can use this capability to write whatever code they like. It
means that any security bug is a full exposure of the entire system.
If the program can not create new executable memory segments then the
attacker is restricted to doing things that the program can already
do, which is very bad but still much less bad than complete exposure
of the entire system.

This kind of argument does not apply to the JVM, at least not in full
force, because the Java code being executed by the JVM is effectively
living in a separate address space, one provided by the JVM itself.
For Java code to exploit executable memory segments would require
additional security holes in the JVM itself, not just one in the
program being executed.

Ian

[YD Jiang]

Thanks for the clarification. 

[Erwin Driessens]

You could let your application generate valid Go source, call the compiler to build a plugin, and load that plugin. I haven't tried that yet, but i think it should work. Downside is that you need to Go tools on the target machine for it to work. I don't know how many times you have to generate new functions, the plugin system can't unload a plugin, so things might not go so smoothly when you need to do this a lot of times in a single run of the program.

[David Collier-Brown]

I've doen generate-and-execute in C that way, at the cost of using LD_PRELOAD and having to restart the program.  Go plugins should be better, as long as you don't have to drop one. Can you supercede one? It sort of looks like you could...  If so, Go could do theequivalent of unload.

--dave

[Max]

Hello,

as others pointed out, a possibility is to use an interpreter. Maybe it's not what the original author wanted - execution speed may be a problem - but definitely possible.

The interpreter https://github.com/cosmos72/gomacro is fairly complete (some features are still missing, for example declaring new interfaces) and could fullfill such role. Disclaimer: I am the author.

Also, the same interpreter exploits the generate-source-code + compile-and-load-plugin technique to implement "import" of non-standard packages,
so this other technique is definitely feasible too. But it will not scale much: each compiled plugin contains all its dependencies,
thus compiling and loading a new plugin for each small snippet of code generated at runtime will very quickly consume a lot of RAM.

Regards,

Max

[Eric Johnson]

Indeed. This is the same answer I was going to give.

Run-time code generation implies a whole bunch of different scenarios with a whole bunch of use-cases. There are resource sharing concerns, security concerns, and performance concerns that need to be nailed down. Otherwise it is difficult to identify whether it is possible or desirable to accomplish what the original post is inquiring about.

Eric.