Announcing: Skylark in Go

[Alan Donovan]

I'm pleased to announce the launch of Skylark in Go: an interpreter for Skylark, implemented in Go.

        github.com/google/skylark

Skylark is a dialect of Python intended for use as a configuration language. Like Python, it is an untyped dynamic language with high-level data types, first-class functions with lexical scope, and garbage collection. Unlike CPython, independent Skylark threads execute in parallel, so Skylark workloads scale well on parallel machines. Skylark is a small and simple language with a familiar and highly readable syntax. You can use it as an expressive notation for structured data, defining functions to eliminate repetition, or you can use it to add scripting capabilities to an existing application.

[Nate Finch]

This looks super cool, btw.  I've been wanting a way to extend go programs, and lua wasn't really doing it for me.  Python(ish) is much more comfortable for most people, I'd think.  Can't wait to try it out.

[Zellyn]

This looks neat. Are you able to provide more context on what this is/will be used for at Google?

[Ben Hoyt]

That looks really neat. I will dive into the code!

I'm very curious how the performance of Skylark in Go compares to Skylark in Java (and CPython 3.6 for that matter) -- any benchmarks on that?

-Ben

On Monday, October 2, 2017 at 12:39:43 PM UTC-4, Alan Donovan wrote:

[Keith Brown]

Cool tool.

Are there any native golang tools simar to this which work on Windows/Linux/OSX? 

[adon...@google.com]

On Saturday, 4 November 2017 11:52:05 UTC-4, Keith Brown wrote:

Are there any native golang tools simar to this which work on Windows/Linux/OSX? 

The Skylark interpreter doesn't make any particular assumptions about the CPU or OS, so it should be highly portable.  Please file an issue if you find otherwise.

On Thursday, November 2, 2017 at 9:42:27 PM UTC-4, Ben Hoyt wrote:

I'm very curious how the performance of Skylark in Go compares to Skylark in Java (and CPython 3.6 for that matter) -- any benchmarks on that?

I don't have any rigorous comparisons, but my informal testing on a number of small benchmarks suggests that CPython is about 2x faster than Skylark in Go (in a single thread), and that Skylark in Go is about 10x faster than Skylark in Java.

[Ben Hoyt]

On Thursday, November 2, 2017 at 9:42:27 PM UTC-4, Ben Hoyt wrote:

I'm very curious how the performance of Skylark in Go compares to Skylark in Java (and CPython 3.6 for that matter) -- any benchmarks on that?

I don't have any rigorous comparisons, but my informal testing on a number of small benchmarks suggests that CPython is about 2x faster than Skylark in Go (in a single thread), and that Skylark in Go is about 10x faster than Skylark in Java

2x as fast as CPython sounds pretty good to me -- nice!

I'm curious why you wrote the dict implementation from scratch (hashtable.go) instead of using Go maps as a base, and adding a secondary data structure (slice of keys?) to keep track of insertion order? I'm presuming there's a good technical reason, but at first glance it seems like it would be faster and simpler to use Go maps to start with.

-Ben

[Alan Donovan]

On 7 November 2017 at 15:06, Ben Hoyt <ben...@gmail.com> wrote:

2x as fast as CPython sounds pretty good to me -- nice!

No, CPython is 2x as fast as Skylark in Go.  It's implemented in C, so it can do things that are sadly impossible in Go, like implement a threaded bytecode interpreter.

I'm curious why you wrote the dict implementation from scratch (hashtable.go) instead of using Go maps as a base, and adding a secondary data structure (slice of keys?) to keep track of insertion order? I'm presuming there's a good technical reason, but at first glance it seems like it would be faster and simpler to use Go maps to start with.

There are many reasons Go maps would not work.  First, they do not allow you to define the hash function and equivalence relation for keys.  Skylark considers 1.0 == 1, for instance, whereas Go does not; also.  Second, Go maps require that keys be comparable, but Skylark tuples are represented as slices, for example. Third, Go maps have non-deterministic iteration order whereas Skylark maps are ordered by insertion. You could maintain a separate slice of keys for the iteration order, but at that point you're halfway to building you're own hash table.

Why should Go maps be faster?  Go's map is implemented in Go.  There's no reason a different implementation should be equally fast, or perhaps even faster.  The basic design of the Skylark-in-Go hash table is actually very similar to Go's map.

[Ben Hoyt]

On Tue, Nov 7, 2017 at 3:16 PM, Alan Donovan <adon...@google.com> wrote:

On 7 November 2017 at 15:06, Ben Hoyt <ben...@gmail.com> wrote:

2x as fast as CPython sounds pretty good to me -- nice!

No, CPython is 2x as fast as Skylark in Go.  It's implemented in C, so it can do things that are sadly impossible in Go, like implement a threaded bytecode interpreter.

Sorry, I actually grokked what you said but then repeated it incorrectly. I meant to say: half as fast as CPython sounds pretty good.

I'm curious why you wrote the dict implementation from scratch (hashtable.go) instead of using Go maps as a base, and adding a secondary data structure (slice of keys?) to keep track of insertion order? I'm presuming there's a good technical reason, but at first glance it seems like it would be faster and simpler to use Go maps to start with.

There are many reasons Go maps would not work.  First, they do not allow you to define the hash function and equivalence relation for keys.  Skylark considers 1.0 == 1, for instance, whereas Go does not; also.  Second, Go maps require that keys be comparable, but Skylark tuples are represented as slices, for example. Third, Go maps have non-deterministic iteration order whereas Skylark maps are ordered by insertion. You could maintain a separate slice of keys for the iteration order, but at that point you're halfway to building you're own hash table.

Why should Go maps be faster?  Go's map is implemented in Go.  There's no reason a different implementation should be equally fast, or perhaps even faster.  The basic design of the Skylark-in-Go hash table is actually very similar to Go's map.

Thanks for the thorough response, that's really helpful.

I hadn't yet realized that Go maps don't allow you to define the hash / equality functions.

And good point about Go's map also being implemented in Go too. I'm a lot more familiar with CPython internals, and was kind of assuming Go maps were written in C+asm, but it looks like the entire runtime is written in Go now (with a bit of assembly).

-Ben

[Zellyn]

Bump, since y'all seem to be actively posting here :-)  An only-2x-slower than CPython interpreter is pretty cool. Just very curious what y'all are doing with it.

[Alan Donovan]

On 7 November 2017 at 15:36, Zellyn <zel...@gmail.com> wrote:

An only-2x-slower than CPython interpreter is pretty cool. Just very curious what y'all are doing with it.

Various infrastructure projects (such as Copybara) are using Skylark as a configuration language since it has proven itself on a large scale within Bazel, is familiar to everyone at Google, and is pretty rational for a configuration language.  Some of those projects use Go, or would use Go if only it had an implementation of Skylark.

[Zellyn Hunter]

Oh neat. So is it a sort of competitor to jsonnet? I guess jsonnet is usually used to generate actual config files, not live-interpret executable config.

Zellyn

[Alan Donovan]

On 7 November 2017 at 15:54, Zellyn Hunter <zel...@gmail.com> wrote:

Oh neat. So is it a sort of competitor to jsonnet? I guess jsonnet is usually used to generate actual config files, not live-interpret executable config.

Jsonnet is more of a templating language whereas Skylark is really arbitrary computation used to build data structures and express policy or behavior.  Sorry if that's vague.  Bazel is (unsurprisingly) a good example of Skylark: a build dependency graph consists of thousands of nodes, and for each node, Bazel executes, in parallel, a Skylark function to figure out what compile commands need to be executed for it.

Fun fact: Dave (who developed Jsonnet) works down the corridor. There are at least three configuration languages developed from this one room.  If nothing else it suggests we need smaller offices.

[bsr]

Like Nate mentioned, I too like to try it instead of lua. Is there a good example on how to embed skylark to define custom logic. Thanks.

On Monday, October 2, 2017 at 12:39:43 PM UTC-4, Alan Donovan wrote:

[bsr]

looks like Nate has the answer :-) https://github.com/hippogryph/skyhook

[adon...@google.com]

Starlark is the new name for the Skylark configuration language. (The old name was the code name for a subproject of Bazel and was not suitable for a project in its own right.)

The Starlark in Go implementation has moved. The code is now hosted at

    https://github.com/google/starlark-go

but the import path you should use in Go programs and go get commands is 

    go.starlark.net

For example:

   $ go get go.starlark.net/cmd/starlark

To avoid breaking existing programs, the old repository will continue to exist but will no longer be updated. Please update your programs to use the new import path.