vitess: new open source project in go

[Sugu Sougoumarane]

We just released the vitess project to open source:

http://code.google.com/p/vitess/

The main goal of the project is to provide servers and tools to facilitate scaling of MySQL databases for the web. This was predominantly developed in go, and is already being used in a large scale production environment.

As part of the project, we developed a few go packages that could come in handy for other developers:

- bson: bson encoding/decoding that's integrated with go's rpc framework.

- cache: A generic LRU cache.

- logfile: A 'Writer' interface that provides transparent logfile rotation and purging. Works with go's log package.

- mysql: An efficient brain-dead client to MySQL.

- relog: A wrapper around go's log package that provides a python-like logging API (Info, Warning, etc.).

- rpcwrap: Convenience functions that make it easy to integrate new encoding schemes into go's rpc.

- stats: Provides advanced data types for go's expvar for reporting rates, histograms, etc.

- timer: A wrapper around ticker for managing house-keeping goroutines that need to regularly wake up and run.

- umgmt: A package that lets you restart your http-server with zero downtime.

We have yet to migrate the build system to use the new go tool. So, the recommended go release for these packages is weekly.2012-01-27.

[Archos]

How reliable is being the Go's garbage collector in that kind of
server? Aren't there unpredictable stalls in transaction processing?

[Sugu Sougoumarane]

On popular demand, we've added a clarification that vitess is the core of YouTube's new MySQL serving infrastructure.

[Michael Shields]

On Tue, Feb 28, 2012 at 1:55 PM, Sugu Sougoumarane <sou...@google.com> wrote:

We just released the vitess project to open source:

http://code.google.com/p/vitess/

Is it called vitess because it's fas?

[Sugu Sougoumarane]

On Tue, Feb 28, 2012 at 2:43 PM, Archos <raul...@sent.com> wrote:

How reliable is being the Go's garbage collector in that kind of
server? Aren't there unpredictable stalls in transaction processing?

For the load we serve, the pause is predictable. It's about 200ms every 3-4 secs. Transactions can tolerate that delay. 

[Sugu Sougoumarane]

Note that there are garbage collector improvements in the pipeline that drastically reduce this pause, which I've mentioned in the wiki:

http://code.google.com/p/vitess/wiki/InstallHowto

[Yunge]

I was so excited!

First, I'm using mysql now.

Most important, Go starts to rewrite "the whole world", from Google
inside, and becomes next-gen C.

[roger peppe]

This is nice to see. Thanks for making it available.

I might be being blind, but I couldn't seem to find a Go
client for vitess. Am I missing something?

[bugpowder]

On Wednesday, February 29, 2012 2:24:04 PM UTC+2, rog wrote:

This is nice to see. Thanks for making it available.

I might be being blind, but I couldn't seem to find a Go
client for vitess. Am I missing something?

Vitess is in Go itself.

Check the code repo.

[roger peppe]

Indeed the vtocc server is written in Go.
But the "operations" page only gives an example
that uses a python client, and I can't quickly see an
equivalent client package for Go.

I was perhaps expecting to see a driver for the database/sql
package.

actually, perhaps vitess/mysql is what i'm looking for,
but i'm not sure - that might just be the package
that the server uses to talk to a given mysql backend.

[Sugu Sougoumarane]

It didn't occur to us that people will be interested in a go client, which should be fairly trivial to write. I can work on releasing one if there's interest.

[Brian Ketelsen]

Of course there's interest, this is a Go list!

[Kyle Lemons]

Of course there's interest, this is a Go list!

+1

Thanks again for releasing this publicly :-).  Now that both Google and a few other companies have open sourced pieces of production stacks, it should make it easier to make a business case for using Go if you're at a company that hasn't signed on yet!

[Sugu Sougoumarane]

I just pushed the go client api (vt/client2). It complies with go's database/sql/driver requirements.

Not very well tested, but it's functional.

On Wed, Feb 29, 2012 at 8:32 AM, roger peppe <rogp...@gmail.com> wrote:

[André Moraes]

> Thanks again for releasing this publicly :-).  Now that both Google and a
> few other companies have open sourced pieces of production stacks, it should
> make it easier to make a business case for using Go if you're at a company
> that hasn't signed on yet!

Go becoming the next Java (without all the bad parts)!

--
André Moraes
http://andredevchannel.blogspot.com/

[JONNALAGADDA Srinivas]

        Good to know of so large-scale a deployment as in YouTube.  The packages are nice, too!  Thanks.

                                                            Greetings,

                                                                    JS

____

http://oneofmanyworlds.blogspot.com

[Sugu Sougoumarane]

We've updated vitess to use go's latest weekly.2012-03-04 and its new build system. All the utility packages should work with 'go get'. However, the main vtocc still requires configuration due to its external dependencies. If you need further updates on this project, please subscribe to http://groups.google.com/group/vitess.