open source golang interpreter?

[zhangwei(Jovi)]

Hi golang nuts,

Is there have any open source golang interpreter for Linux currently?

The reason of why I looking for golang interpreter is that I want to
use interpreter to perform some amazing job, and golang is more suit for this job
than other languages.

(More detail, I want to use interpreter to do some embedding and interoperability things
with C program, so bytecode vm is a better choice than native code generation)

I cannot found the available interpreter, so if Linux golang interpreter isn't exist,
if there have any possibility to write this interpreter based on golang source code?
to reuse compiler source code and runtime library as much as possible.
In my mind, bytecode generation for golang compiler would be another support architecture as
8c/5c/6c, but I'm not sure on this from implementation point of view.

I familiar with golang, but I don't know too much detail of compiler stuff,
that's why I want you can give me some hint:)

Thanks.

.jovi

[Sebastien Binet]

Jovi,

there is go-eval:
https://github.com/sbinet/go-eval

and its side-kick igo:
https://github.com/sbinet/igo

(which adds readline-like capabilities)

do note go-eval isn't implementing the full go-1.0.x language,
though... (yes, not even 1.0.x, let alone 1.1)

with the new interesting features provided by "reflect" it might be
possible to completely reimplement go-eval in terms of the reflect API
(and thus have something more up-to-date wrt the language features)

-s

[minux]

On Thu, May 16, 2013 at 11:58 PM, Sebastien Binet <seb....@gmail.com> wrote:

On Thu, May 16, 2013 at 10:43 AM, zhangwei(Jovi)
<jovi.z...@huawei.com> wrote:
> Is there have any open source golang interpreter for Linux currently?

there is go-eval:
 https://github.com/sbinet/go-eval

and its side-kick igo:
 https://github.com/sbinet/igo

(which adds readline-like capabilities)

besides these two, take a look at go.exp/ssa/interp:

godoc.org/code.google.com/p/go.exp/ssa/interp

it is very slow, but it's a working Go interpreter (please read its document for details),

also note it's under active development.

[Dmitri Shuralyov]

Wow, "go.exp/ssa/interp" is really cool. It's the first actual interpreter for Go that I see. I wonder if you can get it to execute one statement at a time rather than the entire program.

To expand on Sebastien's list of Go evaluators, there's also:

https://github.com/sriram-srinivasan/gore - reads and evals any number of Go lines of code, seems to be more up to date

https://github.com/shurcooL/goe - simplified version of gore that focuses on executing a single Go function at a time

[Jovi Zhang]

On Thu, May 16, 2013 at 11:58 PM, Sebastien Binet <seb....@gmail.com> wrote:

> Jovi,
>
> On Thu, May 16, 2013 at 10:43 AM, zhangwei(Jovi)
> <jovi.z...@huawei.com> wrote:
>>
>> Hi golang nuts,
>>
>> Is there have any open source golang interpreter for Linux currently?
>>
>> The reason of why I looking for golang interpreter is that I want to
>> use interpreter to perform some amazing job, and golang is more suit for this job
>> than other languages.
>>
>> (More detail, I want to use interpreter to do some embedding and interoperability things
>> with C program, so bytecode vm is a better choice than native code generation)
>>
>> I cannot found the available interpreter, so if Linux golang interpreter isn't exist,
>> if there have any possibility to write this interpreter based on golang source code?
>> to reuse compiler source code and runtime library as much as possible.
>> In my mind, bytecode generation for golang compiler would be another support architecture as
>> 8c/5c/6c, but I'm not sure on this from implementation point of view.
>>
>> I familiar with golang, but I don't know too much detail of compiler stuff,
>> that's why I want you can give me some hint:)
>
> there is go-eval:
> https://github.com/sbinet/go-eval
>
> and its side-kick igo:
> https://github.com/sbinet/igo

Thanks, that's cool.
for go-eval, where is the source of main interpreter loop?
and how about the definition of bytecode?
I cann't find it.

[Jovi Zhang]

Thanks, minux.
Actually those three golang interpreter don't quite suit for my
original purpose,
the reason is those interpreters is wrote in golang itself, and is not
in traditional
compiler+virtual machine framework(like JVM, CPython, lua).

I wish there have a golang virtual machine write by C, not golang,
because the vm
could be more portable on many hardware, even can embedding into Linux
kernel, this isn't work
if interpreter is write by golang. and I also want the compiler can
generate bytecode like JVM/Cpython/lua
does.

The reason of choose golang for my purpose is:
1) C interoperability
2) automatic memory management
3) map support
4) static typed with great productive
5) oop without class stuff
6) ...

this is the reason why I come to golang, not python/lua,
basically I don't want the interpreter implement all feature of
golang(like goroutine is not needed),
just subset is enough

(It seems that "Express go" project is mostly meet my requirement, but
it's target on Windows platform,
and not open source)

Thanks.

[Philipp Schumann]

Instead of a "VM framework", why don't you just redistribute the Go distro's core (command+package) binaries and then in your use-case just perform "go run" on a Go source?

[Jovi Zhang]

On Sat, May 18, 2013 at 5:49 PM, Philipp Schumann
<philipp....@gmail.com> wrote:
> Instead of a "VM framework", why don't you just redistribute the Go distro's
> core (command+package) binaries and then in your use-case just perform "go
> run" on a Go source?
>

Mainly because call conversion problem, golang compiler have its own
call conversion, and it's hard
to link with other programs, actually the "other programs" in here
could be Linux kernel.

See ktap project:
https://github.com/ktap/ktap

Currently I'm working on ktap project, that is for embedding a
VM(interpreter) into Linux kernel,
now that VM is forked by lua.

what I want is using a static typed small VM to replace lua as core of
ktap, for better kernel interoperability,
I think golang is very suit for that job, golang is C friendly,
no-class OOP support, high-level map support,
pointer/structure type, etc.
So I tried to find a golang small vm for the embedding purpose, but I
cannot found it.

[Archos]

There is a VM for Go but it is written in Go:

https://github.com/feyeleanor/GoLightly

[rocky]

A lot has been written about a go "interpreter", for a number of very different reasons, namely:

1. For use in an interactive Read/Eval/Print/Loop (REPL)
2. In a debugger
3. In an IDE (and possibly combining aspects 1 & 2)
4. As an embedded or extension language in another subsystem such as lower-level tracing
5. For experimenting with alternate implementations or implementations of various constructs (e.g. slices)

For 3, let me put in a plug for https://code.google.com/p/go-play . Measured against a real IDE this code this is meager. But measured against the "present" and the go playground I think it fares okay. If I can figure out how to submit patches, I'll do that for the small improvements I have made to indentation handling and error location reporting. What go-play currently lacks is the websocket communication in the "present" package. However, simple tests with chrome 27.0.1453.81 (Official Build 198567) and go's websocket gives me an error:

    WebSocket connection to 'ws://localhost:23456/sendRecvText' failed: Invalid UTF-8 sequence in header value

Contributions here to get this last important piece into go-play would be welcome.

At any rate, following the suggestion that old misc/goplay be have another home, it now has. And thus, I agree that gosource/misc/goplay should be deleted since everything it had to offer is in go-play, while go-play offers more.

Now coming back to uses 1-3. Recently there was the suggestion to look at ssa/interp. And the question was asked whether it could handle code fragments instead of a full program. Having looked a little at that code, it doesn't look to me as what I'd use for the basis as an interpreter for 1-3. Rather, my thought would be to base a simple interpreter off of the AST (go/ast). Why?

Well first there is that issue of code fragments. Something working off of an AST wouldn't have that problem. Second, looking at the SSA code in a cursory way I couldn't figure out how to get source-code position information. Given the stated goal: "to facilitate construction of source analysis tools.", I'm probably missing something; I would imagine position information useful to have for source-code analysis.

In contrast, each AST node has position information: http://godoc.org/go/ast#Node How rigorously and accurately that information is maintained is a separate question.

Given that SSA must mangle the source code more than the AST, (in fact the SSA *starts* with the AST), the SSA representation is less likely to correspond to the code in the way that a naive programmer thinks of it with respect to control constructs, variable names or having a one-to-one mapping of code construct with position in the source code.
(But again I hesitate since this seems to contradict its stated goal as a source-analysis tool.)

So as I look at the "interpreter" suggestions so far, I am kind of surprised to see something along the lines of building an interpreter off of the AST missing.

Lastly, some questions regarding comments in ssa/interp (which has been recently moved to http://godoc.org/code.google.com/p/go.tools/ssa)

1. http://godoc.org/code.google.com/p/go.tools/ssa/interp says:

  Unsafe operations, including all uses of unsafe.Pointer, are impossible to support given the "boxed" value representation we have chosen.

What does it mean the "boxed" value representation?

2. "sync/atomic" operations are not currently atomic due to the "boxed" value representation: it is not possible to read, modify and write an interface value atomically. As a consequence, Mutexes are currently broken. TODO(adonovan): provide a metacircular implementation of Mutex avoiding the broken atomic primitives.

There is that "boxed" value representation thing again. But now there is "metacircular implementation of Mutex avoiding the broken atomic primitives". What does that mean? Anyone care to venture a guess?

[minux]

On Sun, May 19, 2013 at 11:44 PM, rocky <rocky.b...@gmail.com> wrote:

1. http://godoc.org/code.google.com/p/go.tools/ssa/interp says:

  Unsafe operations, including all uses of unsafe.Pointer, are impossible to support given the "boxed" value representation we have chosen.

What does it mean the "boxed" value representation?

see https://code.google.com/p/go/source/browse/ssa/interp/value.go?repo=tools for more info.

2. "sync/atomic" operations are not currently atomic due to the "boxed" value representation: it is not possible to read, modify and write an interface value atomically. As a consequence, Mutexes are currently broken. TODO(adonovan): provide a metacircular implementation of Mutex avoiding the broken atomic primitives.

There is that "boxed" value representation thing again. But now there is "metacircular implementation of Mutex avoiding the broken atomic primitives". What does that mean? Anyone care to venture a guess?

by "metacircular implementation of Mutex", it means that provide a native implementation

(that is, use sync.Mutex in the interpreter itself) of Mutex for the interpreted program

(you might want to read more about metacircular in general, and i'd recommend the book

SICP).

[Rocky Bernstein]

On Sun, May 19, 2013 at 11:54 AM, minux <minu...@gmail.com> wrote:

On Sun, May 19, 2013 at 11:44 PM, rocky <rocky.b...@gmail.com> wrote:

1. http://godoc.org/code.google.com/p/go.tools/ssa/interp says:

  Unsafe operations, including all uses of unsafe.Pointer, are impossible to support given the "boxed" value representation we have chosen.

What does it mean the "boxed" value representation?

see https://code.google.com/p/go/source/browse/ssa/interp/value.go?repo=tools for more info.

Ok. So as I understand this now, because a "boxed" value is an interface{}  type, and since a type switch on interface{} doesn't support "unsafe.Pointer", unsafe.Pointer is not supported. I wonder then, shouldn't Go's type switch also include unsafe.Pointer?

If one were to change the interpreter code, would a reasonable possibility be to turn this into a structure and add a field for other cases which aren't covered by Go's type switch? The value nil, I guess, is another kind of value with no type, right?

2. "sync/atomic" operations are not currently atomic due to the "boxed" value representation: it is not possible to read, modify and write an interface value atomically. As a consequence, Mutexes are currently broken. TODO(adonovan): provide a metacircular implementation of Mutex avoiding the broken atomic primitives.

There is that "boxed" value representation thing again. But now there is "metacircular implementation of Mutex avoiding the broken atomic primitives". What does that mean? Anyone care to venture a guess?

by "metacircular implementation of Mutex", it means that provide a native implementation

(that is, use sync.Mutex in the interpreter itself) of Mutex for the interpreted program

(you might want to read more about metacircular in general, and i'd recommend the book

SICP).

Ok. Thanks again for the elaboration. But coming back to the comment.  Is the brokeness of sync/atomic operations in the interpreter is less a fact of the "boxed" value representation rather than it just an artifact that the interpreter has to run several go statements to get/set a value? I now understand the metacircular implementation of Mutex part, but I don't understand the broken atomic primitives part. Isn't this just a matter of adding some mutexes in the interpreter so that the interpreter can then run a sequence of go statements (read, modify and write interface values) atomically?

[Rocky Bernstein]

Thanks for the thoughtful reply. It may take me a while yet to fully understand. 

Comments and questions in line.

On Tue, May 28, 2013 at 11:27 PM, <adon...@google.com> wrote:

On Sunday, 19 May 2013 14:16:42 UTC-4, rocky wrote:

Ok. So as I understand this now, because a "boxed" value is an interface{}  type, and since a type switch on interface{} doesn't support "unsafe.Pointer", unsafe.Pointer is not supported. I wonder then, shouldn't Go's type switch also include unsafe.Pointer? 

 

If one were to change the interpreter code, would a reasonable possibility be to turn this into a structure and add a field for other cases which aren't covered by Go's type switch? The value nil, I guess, is another kind of value with no type, right? 

 

In a compiler or high-quality interpreter, the sizes of all objects in a stack frame or in a struct/array heap alloc are computed statically, and then accesses/updates are done by a combination of address arithmetic relative to the base of the frame/alloc, and "casts".  The representation of a value doesn't know its type, only the compiler does.  In contrast, ssa/interp uses a boxed representation in which every single value (even every byte in a []byte slice) is a distinct instance of interface{}, which of course contains both the value (e.g. a byte 0x12) and its type (e.g. 'byte').

This choice is fine for most things since every memory location has exactly one type, but unsafe.Pointer allows you to create aliases to the same memory location with two different pointer types, and then store it with one type and load it with another.   Since there's no way in Go to create a pointer to the value boxed inside an interface, there's no way for the interpreter to implement unsafe pointer conversions.

I have a concern that because something can't be supported totally, it isn't supported at all which includes the most common or most important situations. 

I can't pass judgment on how important, common, or useful it is to store as one type and load as another. 

The cases however of using an unsafe Pointer to be able to cast values back to access C library results is possibly important.

But there are definitely some uses of unsafe pointers that it is perfectly fine to not support. Specifically those uses where the conversion somehow assumes to know about how data is represented in memory or the overall heap/stack memory layout. For example to access the parameter in a parameter list that is after the one you have the address of. In other words, think of the kind of things used in stack overflow exploits.

if it turns out that there are simple casts from one type to another using unsafe pointers that are perfectly reasonable and common, I don't see why the magic couldn't be done inside the cast operation to unbox the boxed representation and then box it as the new one. 

It's not a deficiency of Go, merely one of the interpreter's representation.  It could be fixed by calculating the layouts of frames/allocs and using  pointer arithmetic and casts.

Sure. I wonder though if there aren't other ways to address this as well.

And at this point I should mention that the uses of a high-quality interpreter might be a bit different from the uses I would like. So let me ask, what uses are envisioned by this high-quality interpreter? 

Coming back to the comment in interp.go it says:

// This interpreter is provided as an adjunct for testing the SSA

// construction algorithm.  Its purpose is to provide a minimal

// metacircular implementation of the dynamic semantics of each SSA

// instruction.  It is not, and will never be, a production-quality Go

// interpreter.

My intended use of an interpreter, very different from the above, would be to provide interactive use as in a REPL, or possibly to implement some sort of string "eval" function. 

And also to be able to try to debug go code at some level. 

This might mean running all of the code in the interpreter (at the risk of introducing Heisenbugs)  or by replacing modules with debug-enabled versions. I'm not sure what "at some level" will be possible until digging deeper.

What is important in my use case is simplicity in tracking execution of the source code: that is, having lots of position information in there, and having the mapping be straight-forward. Execution efficiency or storage efficiency is of lesser concern. (If one can do both - great. But trading off efficiency for transparency is not so good. Probably what is likely to happen is to start of with something really inefficient and over time make it less so)  

 

Ok. Thanks again for the elaboration. But coming back to the comment.  Is the brokeness of sync/atomic operations in the interpreter is less a fact of the "boxed" value representation rather than it just an artifact that the interpreter has to run several go statements to get/set a value? I now understand the metacircular implementation of Mutex part, but I don't understand the broken atomic primitives part. Isn't this just a matter of adding some mutexes in the interpreter so that the interpreter can then run a sequence of go statements (read, modify and write interface values) atomically? 

With the boxed representation, there is no way the three sequential operations to read, modify and write a value stored in an interface{} can be made atomic.  If we had compiler-style memory layout we could just use the atomic primitives to access the memory location directly.

I don't understand why mutex's put around the read, modify, write isn't sufficient. Or perhaps that's what you are describing next? If so, why is this a hack? One important purpose of sync.mutex is to make things appear atomic, right? 

Or, the problem could be fixed by a hack: faking the sync.Mutex type so that instead of interpreting its methods as they're currently defined, the interpreter would handle them specially and implement them in terms of a real compiled sync.Mutex.  See the fake reflect package for an example of how this is done.

You mean ssa/interp/reflect.go, right? If so, I don't understand what you mean. I don't see any sync.Mutex or import of "sync" in there. Perhaps I'm looking at old source? 

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/sQSXvYr_yHE/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Rocky Bernstein]

Comments at the top of ssa/builder 

// The builder has two phases, CREATE and BUILD.  In the CREATE

// phase, all packages are constructed and type-checked and

// definitions of all package members are created, method-sets are

// computed, and bridge methods are synthesized.  The create phase

// proceeds in topological order over the import dependency graph,

// initiated by client calls to CreatePackage.

// [BUILD phase creates ssa...]

So yes, it seems like the CREATE phase would need to be done by any interpreter. Possibly down the line it might be helpful to split that off into its own thing.

But this comes back again to the differences in interpreter uses and goals. In the use case of an interpreter for eval(), REPL, or debugger, one wants to be able to deal with go language entities an the level of expressions rather than whole go programs.

So this CREATE phase would might need to work  in some sort of on-line rather than batch mode when methods and imports can be added on the fly or by "eval".

On Tue, May 28, 2013 at 11:44 PM, <adon...@google.com> wrote:

On Sunday, 19 May 2013 11:44:12 UTC-4, rocky wrote:

Given that SSA must mangle the source code more than the AST, (in fact the SSA *starts* with the AST), the SSA representation is less likely to correspond to the code in the way that a naive programmer thinks of it with respect to control constructs, variable names or having a one-to-one mapping of code construct with position in the source code.
(But again I hesitate since this seems to contradict its stated goal as a source-analysis tool.)

 

You are right that SSA form looks pretty different from the source code, and that in general it is not possible to map source locations to SSA values or instructions one-to-one, but every SSA instruction derived from source does have position information.  The idea is that the simpler representation permits more sophisticated analyses whose result can then be mapped back into the syntactic domain using the small amount of source information maintained in the SSA code.  Notice how much logic there is in ssa/builder.go and you'll realize that there's more to the language semantics than type-checked ASTs.