Re: [go-nuts] Re: dl.google.com now served by Go

[Brad Fitzpatrick]

Multiple levels of paranoia:

We snapshot tip into Google's version control system every so often, but only after running all affected tests and verifying they still pass.  The tip updates are not automatic.

But even then, when we roll out a new binary we have end-to-end integration tests that don't use real user traffic, and even after that it goes out to a small canary group for monitoring (new crashiness, new slowness, etc) before it goes out 100%.

On Fri, Oct 26, 2012 at 7:42 AM, Luke Mauldin <lukem...@gmail.com> wrote:

How have you been able to keep the system stable running from tip?  When you compile new versions, do you always update from tip and then compile?  Or did you pull down a copy of tip and you always compile against that copy?  How do you verify that patches applied to tip do not introduce unexpected effects into your system?

Luke

On Friday, October 26, 2012 9:40:03 AM UTC-5, bradfitz wrote:

On Fri, Oct 26, 2012 at 7:26 AM, Luke Mauldin <lukem...@gmail.com> wrote:

Brad,

Thank you for sharing, that story is a great encouragement.  What version of Go are you using for dl.google.com?  Are you using the standard 1.0.3 version or a customized build with selective patches applied to the core?

tip.  no patches.

 

--
 
 

[si guy]

On Friday, October 26, 2012 7:53:34 AM UTC-7, bradfitz wrote:

canary group

Hmm... Rolling out some new software myself, thanks for the idea.

[Matt Kane's Brain]

On Fri, Oct 26, 2012 at 12:44 PM, si guy <sjw...@gmail.com> wrote:
> Hmm... Rolling out some new software myself, thanks for the idea.

Seems like SOP for large deployments.

--
matt kane's brain
http://hydrogenproject.com

[Brad Fitzpatrick]

On Fri, Oct 26, 2012 at 7:13 AM, ab <cid...@gmail.com> wrote:

Hi Brad,

Thank you for sharing the news. I was wondering about the "boring glue and tests"...
You said you use mainly http.ServeContent but what about the muxer, the sessions and stuff you talked about like migration from local to network file systems, and memory/request caching?

Did you write a kind of library that will be released as an official go package or is it really "raw" code you'll keep secret?

sessions: no sessions.

muxer: no fancy custom muxer.  just a map[string]*Payload basically.  nothing that would be useful to share.

local filesystem to network file system: just a different implementation of an io.ReadSeekCloser, using a Colossus one instead of an *os.File. nothing here to share, since Colossus is not open source.

memory/request caching: I did write a reusable library here, which I have plans to open source, but it's not ready yet.  It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.  I want to open source this once I get some time.  In the meantime you can just use https://github.com/bradfitz/gomemcache or something, which is approximately the same.

Just to know before we start coding our web framework :)

Please try to use net/http by itself or the existing web "frameworks" before you write your own.

[Paddy Foran]

It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.

So this is love. I have so many questions about this, I don't even know where to start.

--
 
 

[Christoph Hack]

On Friday, October 26, 2012 11:02:12 PM UTC+2, Paddy Foran wrote:

It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.

So this is love. I have so many questions about this, I don't even know where to start.

On Friday, October 26, 2012 at 3:28 PM, Brad Fitzpatrick wrote:

Yep, that sounds even better than the original announcement (and that was already pretty exciting). I would also like to hear more about that package. Hopefully Brad gets some "free" time soon :-)

[Rémy Oudompheng]

On 2012/10/26 Brad Fitzpatrick <brad...@golang.org> wrote:
> local filesystem to network file system: just a different implementation of
> an io.ReadSeekCloser, using a Colossus one instead of an *os.File. nothing
> here to share, since Colossus is not open source.

Is that Colossus a network service (meaning NFS-like) that you access
through a client that happens to implement ReadSeekCloser or is it a
C/C++ library ?

Nice to hear that my coworkers will no longer complain about apt-get
google-chrome being slow (and even nicer to hear that the improvement
was done using Go).

Rémy.

[Brad Fitzpatrick]

On Fri, Oct 26, 2012 at 3:24 PM, Rémy Oudompheng <remyoud...@gmail.com> wrote:

On 2012/10/26 Brad Fitzpatrick <brad...@golang.org> wrote:
> local filesystem to network file system: just a different implementation of
> an io.ReadSeekCloser, using a Colossus one instead of an *os.File. nothing
> here to share, since Colossus is not open source.

Is that Colossus a network service (meaning NFS-like) that you access
through a client that happens to implement ReadSeekCloser or is it a
C/C++ library ?

The answer is complicated.  The simplest answer is "yes".  There are many client access models, and many available to Go with different trade-offs.  But Go doesn't care... we can switch between them and it's just a ReadSeekCloser.

 

[ab]

Hi Brad,

memory/request caching: I did write a reusable library here, which I have plans to open source, but it's not ready yet.  It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.  I want to open source this once I get some time.  In the meantime you can just use https://github.com/bradfitz/gomemcache or something, which is approximately the same.

Thanks your these useful answers. I'll have a look at gomemcache.

 

Please try to use net/http by itself or the existing web "frameworks" before you write your own.

Well, yes, when I was saying "our own web framework", I was thinking of using net/http and parts of gorilla or gomvc. Any suggestion on this?

Best,
 

[Brad Fitzpatrick]

On Sun, Oct 28, 2012 at 5:25 PM, ab <cid...@gmail.com> wrote:

Please try to use net/http by itself or the existing web "frameworks" before you write your own.

Well, yes, when I was saying "our own web framework", I was thinking of using net/http and parts of gorilla or gomvc. Any suggestion on this?

I have no experience with either, but I heard good things about gorilla at least, and its docs seem nice.  gomvc might be nice too.  I don't know.

I'd figure out what you need first and then decide whether it's easier to write it, or go find it (and then you can evaluate how well the packages you find solve your problem).  It seems that you're starting with "which packages should I use?" rather than "what do I need?"

[ab]

On Sunday, October 28, 2012 5:59:37 PM UTC+1, bradfitz wrote:

I have no experience with either, but I heard good things about gorilla at least, and its docs seem nice.  gomvc might be nice too.  I don't know.

I'd figure out what you need first and then decide whether it's easier to write it, or go find it (and then you can evaluate how well the packages you find solve your problem).  It seems that you're starting with "which packages should I use?" rather than "what do I need?"

:) Actually, I am a lazy django-python coder so I first looked for a go framework offering similar packages (like the muxer, gorilla's looks nice).
But you're right, I'll pick what's needed when needed and might post the results of some performance comparisons between frameworks around here.

[gary lucas]

Apologies for resurrecting a thread:

I've seen mentioned a couple of speaking events where Brad discussed the conversion, and I think there's at least one video floating around however I can't find anything with a search so far..

Does anyone happen to have a link to this discussion? 

TIA

Gary

On Wednesday, 7 November 2012 11:59:26 UTC-8, bj310...@gmail.com wrote:

On Thursday, October 25, 2012 6:44:23 PM UTC-5, bradfitz wrote:

Google uses Go for many internal projects, but for confidentiality reasons it's rare that we can point to a specific example. YouTube’s open source vitess project (http://code.google.com/p/vitess/) is one high-profile success story, and now I'm happy to announce another.

The service that runs dl.google.com--the source for Chrome, Earth, Android SDK, and other large Google downloads--has been rewritten in Go. In fact, if you’ve had Chrome installed during the past few months then you have almost certainly talked to this Go program. :-)

Why rewrite in Go? It all started back in April of this year, when I was running "apt-get update" at home and noticed that downloads from dl.google.com were stalling for no apparent reason. I asked about this on our internal Google+ and was put in touch with the right people. It turned out that the existing C++ dl.google.com service was no longer actively maintained (short of being kept alive) and that it relied on some assumptions about Google's internal architecture which were no longer true.

The C++ version had grown thorny over the years (so many callbacks!), and it still used some of Google’s oldest C++ libraries, long since deprecated. It also had a few HTTP details wrong and a fair bit of duplicated code, sometimes missing pieces in some copies which were present in others. Much work was required to fix it (or rewrite it in modern C++), so we decided instead to rewrite it in Go.

The Go version is much less code, more readable, more testable, doesn’t have blocking problems, and fixes a number of HTTP correctness issues from the old version.  It also compiles quickly. We were prepared to take a hit in CPU and/or memory usage in exchange for readable code, but it turned out that the CPU was the same and the memory usage actually dropped!

The Go version also uses interfaces to abstract out the file system, so migrating from local disk (as used in the Go version while transitioning from C++) to networked file systems (like our Colossus) was trivial. For the same reason, it was easy to introduce a memory caching layer for the most popular downloads.

All this is to say: apt-get against dl.google.com is fast again.  I’m happy.  :-)

Thanks to all my coworkers who helped with the rewrite and getting it into production!

- Brad

[Brad Fitzpatrick]

It was video taped by New Relic, but I never saw it go online.

The slides are at http://talks.golang.org/2013/oscon-dl.slide#1

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[gary lucas]

Ok, that was it.

Unfortunate it never got posted and thank you for linking the slides.

Gary

[Brad Fitzpatrick]

[Patrick Mylund Nielsen]

Very cool. Nice example for people skeptical about using Go for web stuff.

- Brad

--
 
 

[Steve McCoy]

Sweet! And it uses the net/http server?

[David Symonds]

On Fri, Oct 26, 2012 at 11:09 AM, Steve McCoy <mcc...@gmail.com> wrote:

> Sweet! And it uses the net/http server?

Yes.

[Herbert Fischer]

Great!

- Brad

--
 
 

[Brad Fitzpatrick]

On Oct 25, 2012 5:09 PM, "Steve McCoy" <mcc...@gmail.com> wrote:
>
> Sweet! And it uses the net/http server?

Yup. And the majority of the Handler is just a call to http.ServeContent.

[Henrique Dante de Almeida]

 Source code or it didn't happen.

[David Leimbach]

Congrats! Another win!

[Brad Fitzpatrick]

Here ya go:
http://code.google.com/p/go/source/browse#hg%2Fsrc%2Fpkg%2Fnet%2Fhttp

The rest is boring glue and tests.

--
 
 

[i3dmaster]

It is exciting to see Go in production usage! Great work Brad! Though I have one question just for my own curiosity. My impression is that the core Go team is mainly responsible for (or generally work on) the language design/improvement/fix and related core libraries development, it is somewhat rare (but interesting) to see you guys start tackling (or tightly involved) in specific products. Will the Go team keep supporting the products/services they wrote down the road or at some point, these projects will be transitioned to some dedicated engineering teams? 

Thanks,

Jim

[Brad Fitzpatrick]

On Thu, Oct 25, 2012 at 10:45 PM, i3dmaster <i3dm...@gmail.com> wrote:

It is exciting to see Go in production usage! Great work Brad! Though I have one question just for my own curiosity. My impression is that the core Go team is mainly responsible for (or generally work on) the language design/improvement/fix and related core libraries development, it is somewhat rare (but interesting) to see you guys start tackling (or tightly involved) in specific products. Will the Go team keep supporting the products/services they wrote down the road or at some point, these projects will be transitioned to some dedicated engineering teams? 

The dl.google.com port was done in cooperation with the downloads team.  The same group carrying the pagers for years will continue to do so, and were very involved in porting the code & tests from C++ and doing all the gradual roll-outs and monitoring.  One of the download guys had even done a bunch of surgery on the C++ codebase to modernize it and fix some of the performance problems but then redirected his energy into hacking on the Go version instead, since it was more fun and readable.

The Go team figured any time investment on our side would help us find things to improve in Go, in the standard library, in Google's standard library, in the runtime, in the GC, in protobuf, in our protobuf RPC system, etc.  And it has.  (for instance, run "hg log" and look at some of the more obscure net/http patches I've done in the past few months)  But dl.google.com isn't "our" service or anything.... we're just helping.  If other teams want help and it'd benefit Go somehow, we'd consider a similar arrangement.

Plus, look... marketing.  :-)

[i3dmaster]

Ah... I see. Very reasonable approach. Thanks for the comments!

[ChrisLu]

Is there any performance numbers? How many servers, how many requests/sec, etc?

Chris

On Thursday, October 25, 2012 11:06:59 PM UTC-7, bradfitz wrote:

[stevewang]

Cool. Expect more stories about performance analysis and tuning behind this success story.

[Johann Höchtl]

Am Freitag, 26. Oktober 2012 04:25:23 UTC+2 schrieb bradfitz:

Here ya go:
http://code.google.com/p/go/source/browse#hg%2Fsrc%2Fpkg%2Fnet%2Fhttp

The rest is boring glue and tests.

Sometimes it's sad not being able to upvote an article ... but I kinda like this answer ;)
 

[Paulo Pinto]

Thanks for sharing this.

[Luke Mauldin]

Brad,

Thank you for sharing, that story is a great encouragement.  What version of Go are you using for dl.google.com?  Are you using the standard 1.0.3 version or a customized build with selective patches applied to the core?

Luke

On Thursday, October 25, 2012 6:44:23 PM UTC-5, bradfitz wrote:

[Brad Fitzpatrick]

On Fri, Oct 26, 2012 at 7:26 AM, Luke Mauldin <lukem...@gmail.com> wrote:

Brad,

Thank you for sharing, that story is a great encouragement.  What version of Go are you using for dl.google.com?  Are you using the standard 1.0.3 version or a customized build with selective patches applied to the core?

tip.  no patches.

 

[Luke Mauldin]

How have you been able to keep the system stable running from tip?  When you compile new versions, do you always update from tip and then compile?  Or did you pull down a copy of tip and you always compile against that copy?  How do you verify that patches applied to tip do not introduce unexpected effects into your system?

Luke

[Brad Fitzpatrick]

On Thu, Oct 25, 2012 at 11:31 PM, ChrisLu <chri...@gmail.com> wrote:

Is there any performance numbers? How many servers, how many requests/sec, etc?

None that I can probably share.

It's in a lot of data centers (pretty pictures: http://www.google.com/about/datacenters/), at least.  But to give credit where it's due, the bulk of many downloads (like Chrome updates) are actually served by edge caches in even more locations, so our statement that you've used Go if you've used Chrome is actually a stretch... that assumes you've had an edge cache miss and filled from a dl.google.com server, but I don't know those cache hit rate numbers off hand.  But at least the edge cache for Chrome downloads was populated by Go now.