Re: [go-nuts] Re: dl.google.com now served by Go

27,466 views
Skip to first unread message

Brad Fitzpatrick

unread,
Oct 26, 2012, 10:53:25 AM10/26/12
to Luke Mauldin, golan...@googlegroups.com
Multiple levels of paranoia:

We snapshot tip into Google's version control system every so often, but only after running all affected tests and verifying they still pass.  The tip updates are not automatic.

But even then, when we roll out a new binary we have end-to-end integration tests that don't use real user traffic, and even after that it goes out to a small canary group for monitoring (new crashiness, new slowness, etc) before it goes out 100%.


On Fri, Oct 26, 2012 at 7:42 AM, Luke Mauldin <lukem...@gmail.com> wrote:
How have you been able to keep the system stable running from tip?  When you compile new versions, do you always update from tip and then compile?  Or did you pull down a copy of tip and you always compile against that copy?  How do you verify that patches applied to tip do not introduce unexpected effects into your system?

Luke

On Friday, October 26, 2012 9:40:03 AM UTC-5, bradfitz wrote:


On Fri, Oct 26, 2012 at 7:26 AM, Luke Mauldin <lukem...@gmail.com> wrote:
Brad,

Thank you for sharing, that story is a great encouragement.  What version of Go are you using for dl.google.com?  Are you using the standard 1.0.3 version or a customized build with selective patches applied to the core?

tip.  no patches.

 

--
 
 

si guy

unread,
Oct 26, 2012, 12:44:24 PM10/26/12
to golan...@googlegroups.com, Luke Mauldin


On Friday, October 26, 2012 7:53:34 AM UTC-7, bradfitz wrote:
canary group
Hmm... Rolling out some new software myself, thanks for the idea.

Matt Kane's Brain

unread,
Oct 26, 2012, 12:48:51 PM10/26/12
to si guy, golan...@googlegroups.com, Luke Mauldin
On Fri, Oct 26, 2012 at 12:44 PM, si guy <sjw...@gmail.com> wrote:
> Hmm... Rolling out some new software myself, thanks for the idea.

Seems like SOP for large deployments.

--
matt kane's brain
http://hydrogenproject.com

Brad Fitzpatrick

unread,
Oct 26, 2012, 3:28:52 PM10/26/12
to ab, golan...@googlegroups.com
On Fri, Oct 26, 2012 at 7:13 AM, ab <cid...@gmail.com> wrote:
Hi Brad,

Thank you for sharing the news. I was wondering about the "boring glue and tests"...
You said you use mainly http.ServeContent but what about the muxer, the sessions and stuff you talked about like migration from local to network file systems, and memory/request caching?

Did you write a kind of library that will be released as an official go package or is it really "raw" code you'll keep secret?

sessions: no sessions.

muxer: no fancy custom muxer.  just a map[string]*Payload basically.  nothing that would be useful to share.

local filesystem to network file system: just a different implementation of an io.ReadSeekCloser, using a Colossus one instead of an *os.File. nothing here to share, since Colossus is not open source.

memory/request caching: I did write a reusable library here, which I have plans to open source, but it's not ready yet.  It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.  I want to open source this once I get some time.  In the meantime you can just use https://github.com/bradfitz/gomemcache or something, which is approximately the same.

Just to know before we start coding our web framework :)

Please try to use net/http by itself or the existing web "frameworks" before you write your own.

Paddy Foran

unread,
Oct 26, 2012, 5:01:43 PM10/26/12
to Brad Fitzpatrick, ab, golan...@googlegroups.com
It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.
So this is love. I have so many questions about this, I don't even know where to start.
--
 
 

Christoph Hack

unread,
Oct 26, 2012, 5:43:49 PM10/26/12
to golan...@googlegroups.com
On Friday, October 26, 2012 11:02:12 PM UTC+2, Paddy Foran wrote:
It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.
So this is love. I have so many questions about this, I don't even know where to start.

On Friday, October 26, 2012 at 3:28 PM, Brad Fitzpatrick wrote:

Yep, that sounds even better than the original announcement (and that was already pretty exciting). I would also like to hear more about that package. Hopefully Brad gets some "free" time soon :-)

Rémy Oudompheng

unread,
Oct 26, 2012, 6:24:21 PM10/26/12
to Brad Fitzpatrick, ab, golan...@googlegroups.com
On 2012/10/26 Brad Fitzpatrick <brad...@golang.org> wrote:
> local filesystem to network file system: just a different implementation of
> an io.ReadSeekCloser, using a Colossus one instead of an *os.File. nothing
> here to share, since Colossus is not open source.

Is that Colossus a network service (meaning NFS-like) that you access
through a client that happens to implement ReadSeekCloser or is it a
C/C++ library ?

Nice to hear that my coworkers will no longer complain about apt-get
google-chrome being slow (and even nicer to hear that the improvement
was done using Go).

Rémy.

Brad Fitzpatrick

unread,
Oct 26, 2012, 6:37:38 PM10/26/12
to Rémy Oudompheng, ab, golan...@googlegroups.com
On Fri, Oct 26, 2012 at 3:24 PM, Rémy Oudompheng <remyoud...@gmail.com> wrote:
On 2012/10/26 Brad Fitzpatrick <brad...@golang.org> wrote:
> local filesystem to network file system: just a different implementation of
> an io.ReadSeekCloser, using a Colossus one instead of an *os.File. nothing
> here to share, since Colossus is not open source.

Is that Colossus a network service (meaning NFS-like) that you access
through a client that happens to implement ReadSeekCloser or is it a
C/C++ library ?

The answer is complicated.  The simplest answer is "yes".  There are many client access models, and many available to Go with different trade-offs.  But Go doesn't care... we can switch between them and it's just a ReadSeekCloser.
 

ab

unread,
Oct 28, 2012, 12:25:18 PM10/28/12
to golan...@googlegroups.com, ab
Hi Brad,



memory/request caching: I did write a reusable library here, which I have plans to open source, but it's not ready yet.  It's basically peer-to-peer memcached without versioning (which prevents hot-spotting) and coordinated cache filling (where only one of many waiting parties actually fills the cache), so there aren't thundering herds to backends on cache misses.  I want to open source this once I get some time.  In the meantime you can just use https://github.com/bradfitz/gomemcache or something, which is approximately the same.

Thanks your these useful answers. I'll have a look at gomemcache.

 
Please try to use net/http by itself or the existing web "frameworks" before you write your own.

Well, yes, when I was saying "our own web framework", I was thinking of using net/http and parts of gorilla or gomvc. Any suggestion on this?

Best,
 

Brad Fitzpatrick

unread,
Oct 28, 2012, 12:59:27 PM10/28/12
to ab, golan...@googlegroups.com
On Sun, Oct 28, 2012 at 5:25 PM, ab <cid...@gmail.com> wrote:

Please try to use net/http by itself or the existing web "frameworks" before you write your own.

Well, yes, when I was saying "our own web framework", I was thinking of using net/http and parts of gorilla or gomvc. Any suggestion on this?

I have no experience with either, but I heard good things about gorilla at least, and its docs seem nice.  gomvc might be nice too.  I don't know.

I'd figure out what you need first and then decide whether it's easier to write it, or go find it (and then you can evaluate how well the packages you find solve your problem).  It seems that you're starting with "which packages should I use?" rather than "what do I need?"

ab

unread,
Oct 28, 2012, 2:10:42 PM10/28/12
to golan...@googlegroups.com, ab


On Sunday, October 28, 2012 5:59:37 PM UTC+1, bradfitz wrote:


I have no experience with either, but I heard good things about gorilla at least, and its docs seem nice.  gomvc might be nice too.  I don't know.

I'd figure out what you need first and then decide whether it's easier to write it, or go find it (and then you can evaluate how well the packages you find solve your problem).  It seems that you're starting with "which packages should I use?" rather than "what do I need?"


:) Actually, I am a lazy django-python coder so I first looked for a go framework offering similar packages (like the muxer, gorilla's looks nice).
But you're right, I'll pick what's needed when needed and might post the results of some performance comparisons between frameworks around here.
Message has been deleted

gary lucas

unread,
Dec 11, 2013, 10:37:11 PM12/11/13
to golan...@googlegroups.com, bj310...@gmail.com
Apologies for resurrecting a thread:

I've seen mentioned a couple of speaking events where Brad discussed the conversion, and I think there's at least one video floating around however I can't find anything with a search so far..

Does anyone happen to have a link to this discussion? 

TIA

Gary


On Wednesday, 7 November 2012 11:59:26 UTC-8, bj310...@gmail.com wrote:


On Thursday, October 25, 2012 6:44:23 PM UTC-5, bradfitz wrote:
Google uses Go for many internal projects, but for confidentiality reasons it's rare that we can point to a specific example. YouTube’s open source vitess project (http://code.google.com/p/vitess/) is one high-profile success story, and now I'm happy to announce another.

The service that runs dl.google.com--the source for Chrome, Earth, Android SDK, and other large Google downloads--has been rewritten in Go. In fact, if you’ve had Chrome installed during the past few months then you have almost certainly talked to this Go program. :-)

Why rewrite in Go? It all started back in April of this year, when I was running "apt-get update" at home and noticed that downloads from dl.google.com were stalling for no apparent reason. I asked about this on our internal Google+ and was put in touch with the right people. It turned out that the existing C++ dl.google.com service was no longer actively maintained (short of being kept alive) and that it relied on some assumptions about Google's internal architecture which were no longer true.

The C++ version had grown thorny over the years (so many callbacks!), and it still used some of Google’s oldest C++ libraries, long since deprecated. It also had a few HTTP details wrong and a fair bit of duplicated code, sometimes missing pieces in some copies which were present in others. Much work was required to fix it (or rewrite it in modern C++), so we decided instead to rewrite it in Go.

The Go version is much less code, more readable, more testable, doesn’t have blocking problems, and fixes a number of HTTP correctness issues from the old version.  It also compiles quickly. We were prepared to take a hit in CPU and/or memory usage in exchange for readable code, but it turned out that the CPU was the same and the memory usage actually dropped!

The Go version also uses interfaces to abstract out the file system, so migrating from local disk (as used in the Go version while transitioning from C++) to networked file systems (like our Colossus) was trivial. For the same reason, it was easy to introduce a memory caching layer for the most popular downloads.

All this is to say: apt-get against dl.google.com is fast again.  I’m happy.  :-)

Thanks to all my coworkers who helped with the rewrite and getting it into production!

- Brad

Brad Fitzpatrick

unread,
Dec 12, 2013, 1:03:07 AM12/12/13
to gary lucas, golang-nuts, bj310...@gmail.com
It was video taped by New Relic, but I never saw it go online.




--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

gary lucas

unread,
Dec 12, 2013, 1:08:11 AM12/12/13
to golan...@googlegroups.com, gary lucas, bj310...@gmail.com
Ok, that was it.

Unfortunate it never got posted and thank you for linking the slides.

Gary
Message has been deleted

Brad Fitzpatrick

unread,
Oct 25, 2012, 7:44:13 PM10/25/12
to golang-nuts

Patrick Mylund Nielsen

unread,
Oct 25, 2012, 7:47:10 PM10/25/12
to Brad Fitzpatrick, golang-nuts
Very cool. Nice example for people skeptical about using Go for web stuff.


- Brad

--
 
 

Steve McCoy

unread,
Oct 25, 2012, 8:09:16 PM10/25/12
to golan...@googlegroups.com
Sweet! And it uses the net/http server?

David Symonds

unread,
Oct 25, 2012, 8:10:11 PM10/25/12
to Steve McCoy, golan...@googlegroups.com
On Fri, Oct 26, 2012 at 11:09 AM, Steve McCoy <mcc...@gmail.com> wrote:

> Sweet! And it uses the net/http server?

Yes.

Herbert Fischer

unread,
Oct 25, 2012, 7:48:17 PM10/25/12
to Brad Fitzpatrick, golang-nuts
Great!


- Brad

--
 
 

Brad Fitzpatrick

unread,
Oct 25, 2012, 8:19:57 PM10/25/12
to Steve McCoy, golang-nuts


On Oct 25, 2012 5:09 PM, "Steve McCoy" <mcc...@gmail.com> wrote:
>
> Sweet! And it uses the net/http server?

Yup. And the majority of the Handler is just a call to http.ServeContent.

Henrique Dante de Almeida

unread,
Oct 25, 2012, 8:40:36 PM10/25/12
to golan...@googlegroups.com

 Source code or it didn't happen.

David Leimbach

unread,
Oct 25, 2012, 8:59:16 PM10/25/12
to golan...@googlegroups.com
Congrats! Another win!

Brad Fitzpatrick

unread,
Oct 25, 2012, 10:25:12 PM10/25/12
to Henrique Dante de Almeida, golang-nuts

Here ya go:
http://code.google.com/p/go/source/browse#hg%2Fsrc%2Fpkg%2Fnet%2Fhttp

The rest is boring glue and tests.

--
 
 

i3dmaster

unread,
Oct 26, 2012, 1:45:29 AM10/26/12
to golan...@googlegroups.com, Henrique Dante de Almeida
It is exciting to see Go in production usage! Great work Brad! Though I have one question just for my own curiosity. My impression is that the core Go team is mainly responsible for (or generally work on) the language design/improvement/fix and related core libraries development, it is somewhat rare (but interesting) to see you guys start tackling (or tightly involved) in specific products. Will the Go team keep supporting the products/services they wrote down the road or at some point, these projects will be transitioned to some dedicated engineering teams? 

Thanks,
Jim

Brad Fitzpatrick

unread,
Oct 26, 2012, 2:06:51 AM10/26/12
to i3dmaster, golan...@googlegroups.com, Henrique Dante de Almeida
On Thu, Oct 25, 2012 at 10:45 PM, i3dmaster <i3dm...@gmail.com> wrote:
It is exciting to see Go in production usage! Great work Brad! Though I have one question just for my own curiosity. My impression is that the core Go team is mainly responsible for (or generally work on) the language design/improvement/fix and related core libraries development, it is somewhat rare (but interesting) to see you guys start tackling (or tightly involved) in specific products. Will the Go team keep supporting the products/services they wrote down the road or at some point, these projects will be transitioned to some dedicated engineering teams? 

The dl.google.com port was done in cooperation with the downloads team.  The same group carrying the pagers for years will continue to do so, and were very involved in porting the code & tests from C++ and doing all the gradual roll-outs and monitoring.  One of the download guys had even done a bunch of surgery on the C++ codebase to modernize it and fix some of the performance problems but then redirected his energy into hacking on the Go version instead, since it was more fun and readable.

The Go team figured any time investment on our side would help us find things to improve in Go, in the standard library, in Google's standard library, in the runtime, in the GC, in protobuf, in our protobuf RPC system, etc.  And it has.  (for instance, run "hg log" and look at some of the more obscure net/http patches I've done in the past few months)  But dl.google.com isn't "our" service or anything.... we're just helping.  If other teams want help and it'd benefit Go somehow, we'd consider a similar arrangement.

Plus, look... marketing.  :-)

i3dmaster

unread,
Oct 26, 2012, 2:20:08 AM10/26/12
to golan...@googlegroups.com, i3dmaster, Henrique Dante de Almeida
Ah... I see. Very reasonable approach. Thanks for the comments!

ChrisLu

unread,
Oct 26, 2012, 2:31:51 AM10/26/12
to golan...@googlegroups.com, i3dmaster, Henrique Dante de Almeida
Is there any performance numbers? How many servers, how many requests/sec, etc?

Chris


On Thursday, October 25, 2012 11:06:59 PM UTC-7, bradfitz wrote:

stevewang

unread,
Oct 26, 2012, 4:21:55 AM10/26/12
to golan...@googlegroups.com
Cool. Expect more stories about performance analysis and tuning behind this success story.

Johann Höchtl

unread,
Oct 26, 2012, 4:52:59 AM10/26/12
to golan...@googlegroups.com, Henrique Dante de Almeida


Am Freitag, 26. Oktober 2012 04:25:23 UTC+2 schrieb bradfitz:

Here ya go:
http://code.google.com/p/go/source/browse#hg%2Fsrc%2Fpkg%2Fnet%2Fhttp

The rest is boring glue and tests.

Sometimes it's sad not being able to upvote an article ... but I kinda like this answer ;)
 

Paulo Pinto

unread,
Oct 26, 2012, 5:31:59 AM10/26/12
to golang-nuts
Thanks for sharing this.

Luke Mauldin

unread,
Oct 26, 2012, 10:26:17 AM10/26/12
to golan...@googlegroups.com
Brad,

Thank you for sharing, that story is a great encouragement.  What version of Go are you using for dl.google.com?  Are you using the standard 1.0.3 version or a customized build with selective patches applied to the core?

Luke


On Thursday, October 25, 2012 6:44:23 PM UTC-5, bradfitz wrote:

Brad Fitzpatrick

unread,
Oct 26, 2012, 10:39:52 AM10/26/12
to Luke Mauldin, golan...@googlegroups.com
On Fri, Oct 26, 2012 at 7:26 AM, Luke Mauldin <lukem...@gmail.com> wrote:
Brad,

Thank you for sharing, that story is a great encouragement.  What version of Go are you using for dl.google.com?  Are you using the standard 1.0.3 version or a customized build with selective patches applied to the core?

tip.  no patches.

 

Luke Mauldin

unread,
Oct 26, 2012, 10:42:19 AM10/26/12
to golan...@googlegroups.com, Luke Mauldin
How have you been able to keep the system stable running from tip?  When you compile new versions, do you always update from tip and then compile?  Or did you pull down a copy of tip and you always compile against that copy?  How do you verify that patches applied to tip do not introduce unexpected effects into your system?

Luke

Brad Fitzpatrick

unread,
Oct 26, 2012, 10:46:00 AM10/26/12
to ChrisLu, golan...@googlegroups.com, i3dmaster, Henrique Dante de Almeida
On Thu, Oct 25, 2012 at 11:31 PM, ChrisLu <chri...@gmail.com> wrote:
Is there any performance numbers? How many servers, how many requests/sec, etc?

None that I can probably share.

It's in a lot of data centers (pretty pictures: http://www.google.com/about/datacenters/), at least.  But to give credit where it's due, the bulk of many downloads (like Chrome updates) are actually served by edge caches in even more locations, so our statement that you've used Go if you've used Chrome is actually a stretch... that assumes you've had an edge cache miss and filled from a dl.google.com server, but I don't know those cache hit rate numbers off hand.  But at least the edge cache for Chrome downloads was populated by Go now.
Reply all
Reply to author
Forward
0 new messages