Some Jsonnet benchmarks

[Haoyi Li]

I recently put in some work optimizing our Sjsonnet interpreter, and ran some rough benchmarks. Not sure if Jsonnet compilation performance is a concern for anyone else, but we have a huge pile of Jsonnet code at work configuring everything under the sun, so any slowness does cause inconvenience. Here are the numbers, each one being the average and std dev of time-per-run, from 3 batchs of runs, each batch taking 20s each on a ubuntu 16.04 m4.4xlarge EC2 box:

Time taken to execute a subset of the Jsonnet test suite:

.	Sjsonnet 0.1.5	Sjsonnet 0.1.6
Scala 2.13.0	14.26ms ± 0.22	6.59ms ± 0.27
Scala 2.12.8	18.07ms ± 0.30	9.29ms ± 0.26

google/jsonnet	google/go-jsonnet
~1277ms	~274ms

This is running on an arbitrary subset of the jsonnet test suite defined here:

- https://github.com/databricks/sjsonnet/blob/master/sjsonnet/test/src-jvm/sjsonnet/SjsonnetTestMain.scala#L6

The rough benchmark code is there as well.

google/jsonnet was built from source on commit f59758d1904bccda99598990f582dd2e1e9ad263, and go-jsonnet was the v0.13.0 that go get downloaded. Due to the pretty long durations of each google/jsonnet run, and the consistent times taken, I wasn't able to get a meaningful standard deviation for those runs.

google/jsonnet and google/go-jsonnet were run in subprocesses, while Sjsonnet was run as a persistent daemon with parse-to-AST-caching enabled (this is the same setup our engineers use)

I also ran some benchmarks on a handful of our production Jsonnet files: these look pretty different from the Jsonnet test suite, big/nested structures rather than tons of small asserts. Each Jsonnet file in this set ended up importing a few dozen others, and materialized to 20-40kb of JSON, adding up to ~100kb in total. The time taken to evaluate these is below:

.	Sjsonnet 0.1.5	Sjsonnet 0.1.6
Scala 2.13.0	33.73ms ± 0.44	16.66ms ± 0.08
Scala 2.12.8	40.46ms ± 0.53	20.99ms ± 0.75

google/jsonnet
~6000ms

google/go-jsonnet is not compatible enough to evaluate these Jsonnet files, failing with google/go-jsonnet#314.

Hope this is of interest to someone!

[Dave Cunningham]

Thanks for reporting these numbers, they are very interesting.  Discussion is ongoing about the named parameter issue in https://github.com/google/go-jsonnet/issues/314 

It would be possible to do a more accurate evaluation by using the library versions of go-jsonnet and c++ jsonnet although the latter is known to be much slower and there's not much point even testing itnow.  Then you can keep the interpreter objects alive between runs to preserve the internal cache.  Otherwise, you're reparsing and also redoing the I/O each time as well as spawning processes so it's not a fair experiment.

--
You received this message because you are subscribed to the Google Groups "Jsonnet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/2d0c263a-dc67-4cc1-83c1-29074f8e6cef%40googlegroups.com.

[Haoyi Li]

Yeah I wouldn't mind doing more benchmarks using persistent daemons and caches. It's obviously not 100% equivalent to use parse-to-AST-caching and a persistent daemon for sjsonnet but spawning subprocesses for the go/C++ implementations, but those are the only execution models I'm familiar with. Maybe next time if I have some time to learn some Go or C++, I could try using those implementations as libraries!

On Tuesday, September 10, 2019 at 8:23:47 PM UTC+8, Dave Cunningham wrote:

Thanks for reporting these numbers, they are very interesting.  Discussion is ongoing about the named parameter issue in https://github.com/google/go-jsonnet/issues/314 

It would be possible to do a more accurate evaluation by using the library versions of go-jsonnet and c++ jsonnet although the latter is known to be much slower and there's not much point even testing itnow.  Then you can keep the interpreter objects alive between runs to preserve the internal cache.  Otherwise, you're reparsing and also redoing the I/O each time as well as spawning processes so it's not a fair experiment.

To unsubscribe from this group and stop receiving emails from it, send an email to jso...@googlegroups.com.

[Brett Viren]

"'Dave Cunningham' via Jsonnet" <jso...@googlegroups.com> writes:

> It would be possible to do a more accurate evaluation by using the library versions of
> go-jsonnet and c++ jsonnet although the latter is known to be much slower and there's not
> much point even testing itnow.

o/

I just wanted to raise my hand to say the C++ libjsonnet++.so has users
that are impacted by its speed (or rather its lack)!

It seems interest in libjsonnet++.so is falling away. Is there any path
people see for C++ applications to achieve faster Jsonnet parsing?

My fall back is to have my application call system() to run Sjsonnet or
other fast parsers as separate programs and then consume the resulting
JSON.

-Brett.

[Dave Cunningham]

There is a C API to the go-jsonnet build that is a drop in replacement for libjsonnet.so (and therefore libjsonnetc++.so).  It is pretty raw still, but the idea is to use this for the Python module as well as C/C++ apps.

[Dave Cunningham]

But it's still worth considering the system() approach if you ever want to use ulimit to sandbox the execution (i.e. avoid accidental or deliberate cpu time and memory blowups).

[Eugene Berman]

Folks, is the 0.1.6 jar available at the Maven Central yet? I would like to test the new version but maven can’t find it.

---

	Eugene Berman, Solutions Architect
	7525 SE 24th St., Seattle, WA 98040
	650.235.0288 | eugene...@modusbox.com

--
You received this message because you are subscribed to the Google Groups "Jsonnet" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.

[Haoyi Li]

Hah, I had totally forgotten about uploading it to maven central, and only uploaded the uberjar and JS-bundle to github releases. I have sent maven central the 0.1.6 jars and they should appear there in a few hours

[Tommy Xiang]

I am using jsonnet for a custom game engine to do configuration and prefab definition etc. I am using libsonnetc++ and felt a little concerned about the performance should my configurations get large. Is there any chance of the C++ implementation also getting object field caching and other implementations? Thanks!

[Dave Cunningham]

You should be able to use the C API of the Go port behind the existing C++ API.  Unlike Java, Go is natively compiled so unless you build it yourself you won't even know that it uses Go behind the scenes.

--
You received this message because you are subscribed to the Google Groups "Jsonnet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/cbc1ac7c-8ddd-47e1-87a4-d6fc7cf6b545%40googlegroups.com.

[Tommy Xiang]

The benchmark here https://github.com/databricks/sjsonnet#performance shows that even the Go API isn't the fastest; What exactly did Sjsonnet do to optimize the performance that well? :P

To unsubscribe from this group and stop receiving emails from it, send an email to jso...@googlegroups.com.

[Dave Cunningham]

They don't measure exactly the same thing :)

To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/4fa6b5fe-156a-4723-be82-baf16edc394d%40googlegroups.com.

[Tommy Xiang]

Interesting; Can you elaborate a little?

To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/4fa6b5fe-156a-4723-be82-baf16edc394d%40googlegroups.com.

[Dave Cunningham]

See my first response in this thread.  By the way, is your game engine open source?  Can we try using your jsonnet code as a benchmark?

To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/5414cf95-4c20-4832-8dd7-caa2c00ae93e%40googlegroups.com.

[Tommy Xiang]

Ah, I see. The game engine isn't open source yet because it's still (very) experimental, and I am currently using the approach of embedding jsonnet into the engine to save cross-platform hassle. Seems like if I want to use the Go version I would have to do something separate :'(

To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/5414cf95-4c20-4832-8dd7-caa2c00ae93e%40googlegroups.com.

[Dave Cunningham]

Where we want to be (and are not there yet) is prebuilt go binaries and c++ compatible libraries built every release and available in tarballs from github.  That would be for Mac and Linux at least, and also windows via Appveyor.  So you could have your build system grab the right tarball for the platform, or you build the Go yourself (or have your end users do that if it's distributed via source).

To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/06173006-41b8-4408-90e4-f0889ce6cb89%40googlegroups.com.