Starlight - run python inside Go to extend your applications - easily

[Nate Finch]

I’d like to announce starlight - https://github.com/starlight-go/starlight.

Starlight wraps google’s Go implementation of the starlark python dialect (most notably found in the Bazel build tool). Starlight makes it super easy for users to extend your application by writing simple python scripts that interact seamlessly with your current Go code… with no boilerplate on your part.

Parser by google

The parser and runner are maintained by google’s bazel team, which write starlark-go. Starlight is a wrapper on top of that, which makes it so much easier to use starlark-go. The problem with the starlark-go API is that it is more built to be a used as configuration, so it assumes you want to get information out of starlark and into Go. It’s actually pretty difficult to get Go information into a starlark script…. unless you use starlight.

Easy two-way interaction

Starlight has adapters that use reflection to automatically make any Go value usable in a starlark script. Passing an *http.Request into a starlark script? Sure, you can do name = r.URL.Query()["name"][0] in the python without any work on your part.

Starlight is built to just work the way you hope it’ll work. You can access any Go methods or fields, basic types get converted back and forth seamlessly… and even though it uses reflection, it’s not as slow as you’d think. A basic benchmark wrapping a couple values and running a starlark script to work with them runs in a tiny fraction of a millisecond.

The great thing is that the changes made by the python code are reflected in your go objects, just as if it had been written in Go. So, set a field on a pointer to a struct? Your go code will see the change, no additional work needed.

100% Safe

The great thing about starlark and starlight is that the scripts are 100% safe to run. By default they have no access to other parts of your project or system - they can’t write to disk or connect to the internet. The only access they have to the outside is what you give them. Because of this, it’s safe to run untrusted scripts (as long as you’re not giving them dangerous functions to run, like os.RemoveAll). But at the same time, if you’re only running trusted scripts, you can give them whatever you want (http.Get? Sure, why not?)

Caching

In a production environment, you probably want to only read a script once and parse it once. You can do that with starlight’s Cache. This cache takes a list of directories to look in for scripts, which it will read and parse on-demand, and then store the parsed object in memory for later use. It also uses a cache for any load() calls the scripts use to load scripts they depend on.

Work Ongoing

Starlight is still a work in progress, so don’t expect the API to be perfectly stable quite yet. But it’s getting pretty close, and there shouldn’t be any earth shattering changes, but definitely pin your imports. Right now it’s more about finding corner cases where the starlight wrappers don’t work quite like you’d expect, and supporting the last few things that aren’t implemented yet (like channels).

Example

Here's a simple example of how easy it is to extend the behavior of your application with a python script.  Just pass starlight whatever go values you want your python script to act on, and any changes the python code makes get reflected in your go code.  

package main

import (

    "fmt"

    "log"

    "time"

    "github.com/starlight-go/starlight"

)

// Starlight makes it easy to get values in and out of your starlark scripts.

// Just pass in pointers to values that you want changed, or callback functions

// that propagate data.

// In theory, starlight also returns all global variables set by the script, but

// in real programs, you need well-defined outputs for your calling code to act on.

// If I write a script that creates a variable called nate_is_awesome = 1337 ... your

// go code probably isn't going to care that the variable exists.

// The best way to do it is to write a "results" struct that you pass in, just

// as you would for any other function.

type Page struct {

    Name string

    Date time.Time

    Contents string

    IsDraft bool

}

const code = `

def run():

if "nate" in page.Name:

     # capitalize words

     page.Name = page.Name.title()

page.Name += " " + page.Date.Format("2006/01/02")

page.IsDraft = False

run()

`

func main() {

    p := &Page{

        Name: "a story about nate",

        Date: time.Now(),

        Contents: "I like to write go code.",

        IsDraft: true,

    }

    globals := map[string]interface{}{

        "page": p,

    }

    _, err := starlight.Eval([]byte(code), globals, nil)

    if err != nil {

        log.Fatal(err)

    }

    fmt.Printf("%v is draft: %v\n", p.Name, p.IsDraft)

}

// Running it:

// $ go run inout.go

// A Story About Nate 2018/12/07 is draft: false

[mic...@scylladb.com]

On Friday, December 7, 2018 at 9:05:02 PM UTC+1, Nate Finch wrote:

I’d like to announce starlight - https://github.com/starlight-go/starlight.

Starlight wraps google’s Go implementation of the starlark python dialect (most notably found in the Bazel build tool). Starlight makes it super easy for users to extend your application by writing simple python scripts that interact seamlessly with your current Go code… with no boilerplate on your part.

Do you think it is suitable for porting python applications?

Usually you go through cgo like this https://www.datadoghq.com/blog/engineering/cgo-and-python/ it could be an interesting alternative. 

[Justin Israel]

On Mon, Dec 10, 2018, 9:39 PM <mic...@scylladb.com> wrote:

On Friday, December 7, 2018 at 9:05:02 PM UTC+1, Nate Finch wrote:

I’d like to announce starlight - https://github.com/starlight-go/starlight.

Starlight wraps google’s Go implementation of the starlark python dialect (most notably found in the Bazel build tool). Starlight makes it super easy for users to extend your application by writing simple python scripts that interact seamlessly with your current Go code… with no boilerplate on your part.

Do you think it is suitable for porting python applications?

Usually you go through cgo like this https://www.datadoghq.com/blog/engineering/cgo-and-python/ it could be an interesting alternative. 

 

The documentation for Starlark says it is a subset of python, originally targeted at being a configuration language for Bazel. So it wouldn't be complete enough to straight port full Python applications, unless you really backed alot of it with Go code. The cgo approach gives you full access to the CPython API and to embed an interpreter. 

Starlight does sound interesting though for allowing specific extension scripts for a Go application. 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raffaele Sena]

If you want a full python interpreter, you should look at https://github.com/go-python/gpython, but gpython also doesn't have a full standard library implemented.

-- Raffaele

[Nate Finch]

Justin got this 100% correct.  Starlight and Starlark are not intended to be used to just run existing python code.  there's a ton they don't support, so pretty much any non-trivial python program would not be compatible.

It's really more of a way to give users the ability to write small to moderately sized scripts to customize behavior of your application without having to recompile and potentially without even having to restart the application.

[Jason E. Aten]

For full, actual python inside Go, one could combine:

(a) https://github.com/iodide-project/pyodide  has the python scientific stack compiled to wasm (python + numpy + scipy + numplotlib)

and

(b) either https://github.com/go-interpreter/wagon or https://github.com/perlin-network/life: each provide a wasm byte-code interpreter as a Go library.

[mic...@scylladb.com]

On Tuesday, December 11, 2018 at 3:35:37 AM UTC+1, Jason E. Aten wrote:

For full, actual python inside Go, one could combine:

(a) https://github.com/iodide-project/pyodide  has the python scientific stack compiled to wasm (python + numpy + scipy + numplotlib)

and

(b) either https://github.com/go-interpreter/wagon or https://github.com/perlin-network/life: each provide a wasm byte-code interpreter as a Go library.

Thanks for sharing that, that's some fresh thinking here.