builder documentation?

[shawn....@oracle.com]

Hi,

I was hoping to better understand what exactly the technical requirements for builders are and how they're supposed to work.

In particular:

* What's required for an "old-style" builder (aka buildlet?)
  (Is a simple unprivileged user account on a box sufficient for these?)

* What's required for a "new-style" builder?
  (I assume a plugin for the coordinator to orchestrate starting and stopping VMs, etc.)

Thanks,
-Shawn

[Brad Fitzpatrick]

On Fri, Jun 23, 2017 at 9:26 AM, <shawn....@oracle.com> wrote:

Hi,

I was hoping to better understand what exactly the technical requirements for builders are and how they're supposed to work.

In particular:

* What's required for an "old-style" builder (aka buildlet?)
  (Is a simple unprivileged user account on a box sufficient for these?)

A simple unprivileged user account would work for either old or new style, actually.

The main requirement difference used to be network requirements (we assumed a much faster network with the new-style builders), but there's an option to disable post-make.bash snapshotting for the new style builders these days, so the network difference between the two is pretty minimal.

* What's required for a "new-style" builder?
  (I assume a plugin for the coordinator to orchestrate starting and stopping VMs, etc.)

That is nice, but not required.

Basically we give you a key and then just run the x/build/cmd/buildlet binary under some process supervisor, and run it with the right flags:

   buildlet --reverse-type=host-solaris-shawnwalker --reboot=false --coordinator=farmer.golang.org

And we modify:

   https://github.com/golang/build/blob/master/dashboard/builders.go

... to add your builder. And restart the build coordinator (farmer.golang.org).

And that's it. Ideally you run more than 1, but 1 works.

The old-style builder should really be deleted, but that requires coordinating with everybody who still uses it. But it shouldn't be used more. It has known problems we don't intend to fix.

[Shawn Walker-Salas]

On 06/23/17 09:31 AM, Brad Fitzpatrick wrote:
> On Fri, Jun 23, 2017 at 9:26 AM, <shawn....@oracle.com
> <mailto:shawn....@oracle.com>> wrote:
>
> Hi,
>
> I was hoping to better understand what exactly the technical
> requirements for builders are and how they're supposed to work.
>
> In particular:
>
> * What's required for an "old-style" builder (aka buildlet?)
> (Is a simple unprivileged user account on a box sufficient for
> these?)
>
>
> A simple unprivileged user account would work for either old or new
> style, actually.
>
> The main requirement difference used to be network requirements (we
> assumed a much faster network with the new-style builders), but there's
> an option to disable post-make.bash snapshotting for the new style
> builders these days, so the network difference between the two is pretty
> minimal.
>
>
> * What's required for a "new-style" builder?
> (I assume a plugin for the coordinator to orchestrate starting
> and stopping VMs, etc.)
>
>
> That is nice, but not required.
>
> Basically we give you a key and then just run the x/build/cmd/buildlet
> binary under some process supervisor, and run it with the right flags:
>
> buildlet --reverse-type=host-solaris-shawnwalker --reboot=false

> --coordinator=farmer.golang.org <http://farmer.golang.org>

>
> And we modify:
>
> https://github.com/golang/build/blob/master/dashboard/builders.go
>
> ... to add your builder. And restart the build coordinator

> (farmer.golang.org <http://farmer.golang.org>).

>
> And that's it. Ideally you run more than 1, but 1 works.
>
> The old-style builder should really be deleted, but that requires
> coordinating with everybody who still uses it. But it shouldn't be used
> more. It has known problems we don't intend to fix.

I'm aware the old-style builders are deprecated, but hoping to get
something simple going in the immediate future and then work on getting
a new-style builder going. Unless the requirements between the two are
really no different.

Is the example command you gave above for an "old-style" or "new-style"
builder?

For running this on a new platform, is there anything special required?
Do I just build the Go binary, run it, and that's it?

What sort of things might I need to modify or update?

Is there any way to try this out locally for testing purposes so I can
first stage this environment, and then validate the configuration before
offering a builder?

Just trying to figure out how this all ties together; for example, I see
env/solaris-amd64 that jtsylve added, but I'm not sure where else I
might need to make changes.

Thanks,
-Shawn

[Brad Fitzpatrick]

On Fri, Jun 23, 2017 at 9:54 AM, Shawn Walker-Salas <shawn....@oracle.com> wrote:

On 06/23/17 09:31 AM, Brad Fitzpatrick wrote:

On Fri, Jun 23, 2017 at 9:26 AM, <shawn....@oracle.com <mailto:shawn....@oracle.com>> wrote:

    Hi,

    I was hoping to better understand what exactly the technical
    requirements for builders are and how they're supposed to work.

    In particular:

    * What's required for an "old-style" builder (aka buildlet?)
       (Is a simple unprivileged user account on a box sufficient for
    these?)


A simple unprivileged user account would work for either old or new style, actually.

The main requirement difference used to be network requirements (we assumed a much faster network with the new-style builders), but there's an option to disable post-make.bash snapshotting for the new style builders these days, so the network difference between the two is pretty minimal.


    * What's required for a "new-style" builder?
       (I assume a plugin for the coordinator to orchestrate starting
    and stopping VMs, etc.)


That is nice, but not required.

Basically we give you a key and then just run the x/build/cmd/buildlet binary under some process supervisor, and run it with the right flags:

    buildlet --reverse-type=host-solaris-shawnwalker --reboot=false --coordinator=farmer.golang.org <http://farmer.golang.org>

And we modify:

https://github.com/golang/build/blob/master/dashboard/builders.go

... to add your builder. And restart the build coordinator (farmer.golang.org <http://farmer.golang.org>).

And that's it. Ideally you run more than 1, but 1 works.

The old-style builder should really be deleted, but that requires coordinating with everybody who still uses it. But it shouldn't be used more. It has known problems we don't intend to fix.

I'm aware the old-style builders are deprecated, but hoping to get something simple going in the immediate future and then work on getting a new-style builder going.  Unless the requirements between the two are really no different.

Do not use the old-style one. It's actually more painful for all parties and comes with no support.

 

Is the example command you gave above for an "old-style" or "new-style" builder?

New.

 

For running this on a new platform, is there anything special required? Do I just build the Go binary, run it, and that's it?

Yes, but it'll exit after each build, so you need to run it under a process supervisor which will restart it every time.

 

What sort of things might I need to modify or update?

Probably nothing.

 

Is there any way to try this out locally for testing purposes so I can first stage this environment, and then validate the configuration before offering a builder?

There's a staging server, but it's just as much work for me to configure it there as in prod, where it won't affect anybody anyway, not being a trybot. So we can just use prod.

 

Just trying to figure out how this all ties together; for example, I see env/solaris-amd64 that jtsylve added, but I'm not sure where else I might need to make changes.

The env/* directory is for describing how to build a hermetic VM/container for a builder, which we try to do in general. But if you want to manually setup a machine, that's better than no builder.

So you can ignore env/*.

I'll email you a key.

 

 

[Shawn Walker-Salas]

On 06/23/17 09:58 AM, Brad Fitzpatrick wrote:
>
>
> On Fri, Jun 23, 2017 at 9:54 AM, Shawn Walker-Salas

> <shawn....@oracle.com <mailto:shawn....@oracle.com>> wrote:
...

> For running this on a new platform, is there anything special
> required? Do I just build the Go binary, run it, and that's it?
>
>
> Yes, but it'll exit after each build, so you need to run it under a
> process supervisor which will restart it every time.

By process supervisor, then, I assume all you need is something that
runs the process again every time it exits?

It doesn't need to do anything else?

What network access does this process need?

Can it operate behind an http proxy? (I assume so, requisite proper
environment variables are set.)

Will this builder process run multiple builds at a time? How much
memory / disk space, roughly, should I reserve for its needs?

From what I can tell, each build of Go needs about ~1GB of disk space
(max) for both the source and the produced binaries, and memory
requirements are fairly minimal (1GB at most).

Does this builder process handle the try-bot, etc. as well, or does that
require something additional? Is it just used for build.golang.org
requests?

Thanks,
-Shawn

[Brad Fitzpatrick]

On Fri, Jun 23, 2017 at 10:05 AM, Shawn Walker-Salas <shawn....@oracle.com> wrote:

On 06/23/17 09:58 AM, Brad Fitzpatrick wrote:

On Fri, Jun 23, 2017 at 9:54 AM, Shawn Walker-Salas <shawn....@oracle.com <mailto:shawn....@oracle.com>> wrote:

...

    For running this on a new platform, is there anything special
    required? Do I just build the Go binary, run it, and that's it?


Yes, but it'll exit after each build, so you need to run it under a process supervisor which will restart it every time.

By process supervisor, then, I assume all you need is something that runs the process again every time it exits?

Yes.

On Joyent we use:

svcs -x buildlet.xml

https://github.com/golang/build/blob/master/env/solaris-amd64/joyent/buildlet.xml

It doesn't need to do anything else?

Not unless you want it to.

 

What network access does this process need?

It'll download the Go source once per build. (not all of git, just a tarball of the git rev it's testing)

 

Can it operate behind an http proxy? (I assume so, requisite proper environment variables are set.)

Depends on the proxy. But probably. It doesn't require any inbound connections, but it makes a single outbound HTTPS connection and then flips that connection around with a custom multiplexing protocol (similar to SOCKS5 or HTTP/2) permitting the server (farmer.golang.org) to then make connections back to its own HTTP server.

So if it permits outbound HTTPS and doesn't try to MITM it, I imagine it'll work.

Will this builder process run multiple builds at a time? 

No.

 

How much memory / disk space, roughly, should I reserve for its needs?

1 GB of memory, few hundred MB of disk?

 

From what I can tell, each build of Go needs about ~1GB of disk space (max) for both the source and the produced binaries, and memory requirements are fairly minimal (1GB at most).

Yup.

 

Does this builder process handle the try-bot, etc. as well, or does that require something additional?  Is it just used for build.golang.org requests?

That's just configuration:

https://godoc.org/golang.org/x/build/dashboard#BuildConfig.TryBot

But we don't set that to true unless we have a bunch of capacity since trybot usage can be bursty and slows down people wanting to submit code. The goal is for Trybots to finish in 5 minutes. (sometimes 6 lately)  For linux-arm, for instance, we shard tests out over 6-8 of our 50 arm machines.