Google Grumpy (Python->Go)

[Shawn Milochik]

I'm surprised this hasn't hit this list yet.

https://opensource.googleblog.com/2017/01/grumpy-go-running-python.html

https://github.com/google/grumpy

TL;DR; compile your Python 2.7 to Go by replacing CPython with Grumpy.

Looks very interesting!

[Andy Balholm]

Finally, a way to deploy a Python program as a statically-linked binary!

The Go code this produces looks awful. It’s more like threaded code in Go syntax than like real Go. It translated my toy benchmark function

def count(x):
i = 1
while i <= x:
i += 1

into

// line 12: def count(x):
πF.SetLineno(12)
πTemp003 = make([]πg.FunctionArg, 1)
πTemp003[0] = πg.FunctionArg{Name: "x", Def: nil}
πTemp001 = πg.NewFunction(πg.NewCode("count", "count.py", πTemp003, 0, func(πF *πg.Frame, πArgs []*πg.Object) (*πg.Object, *πg.BaseException) {
var µx *πg.Object = πArgs[0]; _ = µx
var µi *πg.Object = πg.UnboundLocal; _ = µi
var πTemp001 *πg.Object
_ = πTemp001
var πTemp002 bool
_ = πTemp002
var πTemp003 *πg.Object
_ = πTemp003
var πE *πg.BaseException; _ = πE
for ; πF.State() >= 0; πF.PopCheckpoint() {
switch πF.State() {
case 0:
default: panic("unexpected function state")
}
// line 13: i = 1
πF.SetLineno(13)
µi = πg.NewInt(1).ToObject()
// line 14: while i <= x:
πF.SetLineno(14)
Label1:
if raised := πg.CheckLocal(πF, µi, "i"); raised != nil {
continue
}
if raised := πg.CheckLocal(πF, µx, "x"); raised != nil {
continue
}
if πTemp001, πE = πg.LE(πF, µi, µx); πE != nil {
continue
}
if πTemp002, πE = πg.IsTrue(πF, πTemp001); πE != nil {
continue
}
if !πTemp002 {
goto Label2
}
// line 15: i += 1
πF.SetLineno(15)
if raised := πg.CheckLocal(πF, µi, "i"); raised != nil {
continue
}
if πTemp003, πE = πg.IAdd(πF, µi, πg.NewInt(1).ToObject()); πE != nil {
continue
}
µi = πTemp003
goto Label1
goto Label2
Label2:
return nil, nil
}
return nil, πE
}), πF.Globals()).ToObject()

It’s also about 2x slower than the original Python. All of this indirection is necessary to preserve the dynamic semantics of Python. Basically it’s taking the calls into the runtime that a Python interpreter would make, and writing them out one after another.

So don’t expect it to be a way to magically migrate your old Python project to idiomatic Go.

But what it does provide is Python with goroutines instead of the GIL, and with extension modules written in Go instead of C.

Andy

[andrey mirtchovski]

> It’s also about 2x slower than the original Python. All of this indirection is necessary to preserve the dynamic semantics of Python. Basically it’s taking the calls into the runtime that a Python interpreter would make, and writing them out one after another.

this single-thread slowdown does appear on the performance graph
published in the article. python historically has had significant
effort thrown behind single-thread/process performance.

[Justin Israel]

I just gave this a play since I was really curious what it could do. But it seems like of your python module has imports for other python modules, they get transpiled into Go import statements that look at the grumpy stdlib location. Does anyone have any information on the extend of the support, besides the documented caveats? 

From what I gather, it seems to target the use case of single file executable scripts, in pure python, with only dependencies on the python stdlib or the Go stdlib, and use multithreading as a focus (in order for it to realize the performance benefits). Does that seem correct? If so, can anyone explain a real or hypothetical example of where grumpy is most useful? I can't say I have anything that fits this use case. But I suppose google/YouTube team have a bunch of these concurrent data crunching scripts. 

Justin 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[trotte...@gmail.com]

Cross posting to grumpy-users

On Thursday, 5 January 2017 00:42:31 UTC-8, Justin Israel wrote:

I just gave this a play since I was really curious what it could do. But it seems like of your python module has imports for other python modules, they get transpiled into Go import statements that look at the grumpy stdlib location. Does anyone have any information on the extend of the support, besides the documented caveats? 

Basically the build and import systems are pretty immature right now and so the only way to import Python libraries is by stuffing them into the same directory as the standard library. There's been a couple discussions about how to best support third party libs:

https://github.com/google/grumpy/issues/43

https://groups.google.com/forum/#!topic/grumpy-users/JJFFube8qkM

 

From what I gather, it seems to target the use case of single file executable scripts, in pure python, with only dependencies on the python stdlib or the Go stdlib, and use multithreading as a focus (in order for it to realize the performance benefits). Does that seem correct? If so, can anyone explain a real or hypothetical example of where grumpy is most useful? I can't say I have anything that fits this use case. But I suppose google/YouTube team have a bunch of these concurrent data crunching scripts. 

Pure Python, yes. C extensions are not supported. However, the long term goal is to support any third party pure Python library. The originating use case is the YouTube frontend which is a very large application with many external dependencies.

[simon place]

just been taking a look, and wondered why i couldn't "go get" it, then i see the compiler is in python.

which leads to the question; has anyone attempted to compile the compiler to go?

[Dylan Trotter]

The compiler cannot yet run under Grumpy because it uses standard libraries that aren't yet supported. The most substantial unsupported library is the ast module. Once https://github.com/google/grumpy/pull/216 is merged, I think we'll be very close to hosting the compiler in Grumpy. At that point, supporting go get would be feasible.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/OJX0TlOF-hw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[simon place]

cool, i'll keep an eye on that.

[pa...@boddie.org.uk]

On Saturday, February 4, 2017 at 5:35:33 AM UTC+1, Dylan Trotter wrote:

The compiler cannot yet run under Grumpy because it uses standard libraries that aren't yet supported. The most substantial unsupported library is the ast module. Once https://github.com/google/grumpy/pull/216 is merged, I think we'll be very close to hosting the compiler in Grumpy. At that point, supporting go get would be feasible.

As someone suggested on Reddit (or Hacker News, or whatever it is called), you could use the ast module implementation from PyPy. I've been working on my own toolchain for a Python-like language, and to move gradually towards self-hosting, I migrated from the parser module (implemented in C for CPython) to the pyparser package that is also provided by PyPy. I didn't adopt the ast module from PyPy because I've been using the compiler package all along, and it was more sensible for me to adapt pyparser and my fork of compiler to work together.

With the pyparser stuff, you'll need to remove some PyPy-specific operations related to object spaces, but it wasn't difficult to get working at all. The PyPy people have done some nice work that, unfortunately, doesn't get recognised as much as it should. Since I'm aiming to release the source code for my toolchain fairly soon, you might be interested in what (little) I did in this regard.

By the way, I like what you're attempting to do with this project. I don't use Go myself, but I totally understand and support the idea of targeting it. My own toolchain targets C, which is fairly awkward but does have some portability arguments in its favour.

Paul

[Dylan Trotter]

Interesting. Thanks for the perspective. I did take a look at the PyPy ast module but I guess I got scared off by the PyPy specific requriements.

I ended up integrating pythonparser which is a pure Python parser for Python 2.x and 3.x. I'm very impressed with the quality of the code and it has a more powerful interface than the AST module (e.g. it gives line/column ranges for different parts of a node), so it seemed like a no brainer.

I am curious what your toolchain looks like, so if you publish it, be sure to update this thread.

--
You received this message because you are subscribed to the Google Groups "Grumpy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grumpy-users...@googlegroups.com.
To post to this group, send email to grumpy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grumpy-users/86089c26-8d6a-4780-90c4-8d883aef3307%40googlegroups.com.

[Paul Boddie]

On Saturday 18. February 2017 23.45.13 Dylan Trotter wrote:
> Interesting. Thanks for the perspective. I did take a look at the PyPy ast
> module but I guess I got scared off by the PyPy specific requriements.
>
> I ended up integrating pythonparser

> <https://github.com/m-labs/pythonparser> which is a pure Python parser for

> Python 2.x and 3.x. I'm very impressed with the quality of the code and it
> has a more powerful interface than the AST module (e.g. it gives
> line/column ranges for different parts of a node), so it seemed like a no
> brainer.

I know that pyparser does provide column (or "offset") information, which was
not something exposed by the compiler module and therefore not something I was
looking to use, but I'm not sure about ranges, and it's very possible that it
doesn't support them given the actual needs of the modules that would be using
it for parsing.

I'll try and take a look at pythonparser, though. (It's interesting that it's
done by the M-Labs people and I'm familiar with the activities of at least one
of the authors. It's a small world!)

One nice thing about pyparser is that the grammar is configurable, and I've
modified it to change what language elements I support. I don't immediately
see something similar in pythonparser, although I guess it could be embedded
in one of the Python files.

> I am curious what your toolchain looks like, so if you publish it, be sure
> to update this thread.

Sure. I'm trying to write some documentation, but keep drifting off and doing
other things instead. I'll try harder. ;-)

Paul

P.S. Sorry if I shouldn't have CC'd so many people. I guess that it what
everyone in the recipient list wanted, but feel free to say otherwise!