Juju at Canonical

[Dave Cheney]

Hello,

I wanted to share a Go related success story from Canonical.

Over the past year our small team have been busily engaged in porting
Juju[1] from Python to Go. Last week marked the first non experimental
release[2] of our rewrite of Juju in Go.

In addition to porting Juju itself, a large number of packages were
developed from scratch to replace existing Python dependencies. These
include crowd favorites like the MongoDB driver, mgo[3], and our AWS
and OpenStack packages, goamz[4] and goose[5]. Also written were many
utility packages, like tomb, goyaml, gnuflag and an enhanced testing
package, gocheck.

To date, this is some 109,000 lines of Go code (including tests), all
go get'able.

If you are interested in checking out the project itself, you can find
it online at https://launchpad.net/juju-core.

Cheers

Dave
(On behalf of the Juju team)

[1] https://juju.ubuntu.com/
[2] https://lists.ubuntu.com/archives/juju/2013-February/002131.html
[3] http://labix.org/mgo
[4] https://launchpad.net/goamz
[5] https://launchpad.net/goose

[Patrick Mylund Nielsen]

Congratulations guys!

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[SteveD]

On Monday, February 11, 2013 10:43:09 AM UTC+2, Patrick Mylund Nielsen wrote:

Congratulations guys!

Totally!  What of course many would like to know is how much more efficient the resulting implementation is.  I'm sure there's a lot of people looking at Go as a good direction to take their Python code bases ;)

[Dave Cheney]

Efficiency is a hard question to answer. Along the way we've made a
lot of improvements, switched data stores from Zookeeper to Mongo,
etc. Juju is an asynchronous system, which fits the channel model
well, but as almost all actions involve a network round trip, usually
to a 3rd party API, it is difficult to draw any direct performance
comparisons between the two versions of Juju based solely on their
runtime.

I'm sure the other Juju developers will be able to chime in with some
more specific observations.

Cheers

Dave

[SteveD]

On Monday, February 11, 2013 12:44:58 PM UTC+2, Dave Cheney wrote:

well, but as almost all actions involve a network round trip, usually
to a 3rd party API, it is difficult to draw any direct performance
comparisons between the two versions of Juju

That makes sense. But (subjectively) does it feel better and more manageable?  I'd imagine it would certainly be easier to deploy.

 

[Dave Cheney]

Yes to all three.

[Thomas Kappler]

Congratulations! It would be very interesting to hear what the team now thinks about Go's strengths and weaknesses, after such a substantial amount of code written.

[Gustavo Niemeyer]

On Mon, Feb 11, 2013 at 3:41 AM, Dave Cheney <da...@cheney.net> wrote:
> I wanted to share a Go related success story from Canonical.
>
> Over the past year our small team have been busily engaged in porting
> Juju[1] from Python to Go. Last week marked the first non experimental
> release[2] of our rewrite of Juju in Go.
>
> In addition to porting Juju itself, a large number of packages were
> developed from scratch to replace existing Python dependencies. These
> include crowd favorites like the MongoDB driver, mgo[3], and our AWS
> and OpenStack packages, goamz[4] and goose[5]. Also written were many
> utility packages, like tomb, goyaml, gnuflag and an enhanced testing
> package, gocheck.

Thanks for spreading the love Dave.

As a remark worth pointing out mainly due to licensing and copyright,
there are personal packages in that list that predate juju and
Canonical's adoption of Go as a mainstream language (mgo, tomb,
gocheck, specifically).


gustavo @ http://niemeyer.net

[Benoît Amiaux]

Would you mind to share what motivated the migration from python to go ?

Where did you get the most benefits from go ?

[Andrew Wilkins]

On Monday, 11 February 2013 13:41:26 UTC+8, Dave Cheney wrote:

Over the past year our small team have been busily engaged in porting
Juju[1] from Python to Go. Last week marked the first non experimental
release[2] of our rewrite of Juju in Go.

Congratulations! Juju looks really cool, though I've yet to use it in anger.

Nice to see another significant piece of work written in Go.

[Dave Cheney]

> Congratulations! It would be very interesting to hear what the team now
> thinks about Go's strengths and weaknesses, after such a substantial amount
> of code written.

Sure. What follows are my experiences working with Go on Juju. For the
avoidance of doubt, they do not represent any official position by
Canonical. Blah Blah.

The Good:

* static deployment. We currently support three versions of Ubuntu and
in the future an unknown number more, relying on anything in the
distribution is fraught with danger, and multiplies the testing load.
Being able to produce a stand alone binary which we can stick in a tar
ball is one of the cornerstones of our design.

* channels. The central idea is Juju agents watch or subscribe to
changes in the data model, which is stored centrally. Modelling
watches as channels which emit values is central to the design and a
strong fit for Go. In general an agent only does one thing at a time,
be it watching or responding, but we need to have exquisite control of
what that goroutine is doing. This is what the tomb package gives us,
via channels and select. Gustavo has written extensively on this
topic, http://blog.labix.org/2011/10/09/death-of-goroutines-under-control
is recommended reading.

* static typing. The previous Python tests spent a lot of time
ensuring that a list of things didn't contain a string where it should
be an int and so forth. Static typing eliminates this whole class of
runtime errors.

* compilation speed. I don't know if you could list this as an
advantage over the python version, but I certainly appreciate it.

* cross platform. I've successfully built and used juju on linux/arm
and darwin/amd64 without a single issue, I've been told that windows
also works, but not officially supported yet as there are a lot of
unixisms to expunge from the test code. This opens up a huge range of
platforms that people can use to control Juju deployments essentially
for free. This is a feature that will continue to pay benefits for us
in the future.

* super stable Go 1.0 API. Although most of the team use go 1.0.2/3, I
develop against tip. This has resulted in exactly one build breakage,
and that was actually a bug in escape analysis which was quickly
fixed. I was glad that Juju could exercise the compiler to uncover
this bug before it ever hit a release version of Go.

The Less Good:

* the elephant in the room is obviously the package version problem
that many people have encountered. If you've tried to go get Juju,
you'll see there are many dependencies, most of which are under the
launchpad.net namespace and being developed alongside juju-core
itself. At the moment we've resorted to an email semaphore to tell
everyone else to updated their dependencies. If we weren't all working
for the same company, or didn't have good lines of communications
between the groups, I can see this not working well at all. Versioning
imports via a scheme adopted by the mgo package has been suggested,
but as many of the packages are under active development the noise of
continually renaming paths and updating imports this hasn't received a
lot of support internally.

* linking speed. While package compilation is very fast, it always
ends in linking one of our two main binaries (either the juju command
line, or the jujud daemon) and this is getting slower as we write more
code. When I say slower, this is relative the lightening speed of the
compiler, it's just over a second on my machine. This also has an
impact in running unit tests as they require a pass through the
linker. Linker speed is a well known issue and I'm sure it will get
attention soon enough.

That is all I can think of for the moment. Feel free to contact me off
list if you have more specific questions.

Cheers

Dave