Reduce list of GOOS/GOARCH crossbuilt for each PR

[Sylvain Rabot]

Hi,

I'm wondering if we could reduce the list of GOOS/GOARCH that are crossbuilt for every PR by circle-ci.

The building of the complete list seems like a waste of time & resources to me.

Maybe we could select a few and only build the complete list when building tags ?

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

[Krasimir Georgiev]

I think that is a very good idea.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CADjtP1FJKyVj_gq-hgVgyyVbJ%3D-pECFqcPK-QviXmKB1R-oAgg%40mail.gmail.com.

[Matthias Rampke]

There are some exceptions like node exporter where it's important that all variants at least build, but that has a custom setup already.

What would be a sufficient subset? Do we need to worry about endianness and 32 bit architectures, or would just building not catch issues specific to these anyway?

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/2508CDF1-CC2A-4AC6-B9EE-D68B53AFF166%40getmailspring.com.

[Sylvain Rabot]

I did not say it but I was speaking of prometheus/prometheus, I haven't checked others repos for their full cross-building time.

I think we can come up with a minimal list of GOOS/GOARCH for PRs but, if you think building the complete list on tags only is not enough, we could do it on tags & master.

If we were to choose to build the complete list for tags only I would suggest to build this for PRs:

- linux/amd64
- linux/386
- linux/arm
- linux/arm64
- darwin/amd64
- windows/amd64
- freebsd/amd64
- openbsd/amd64
- netbsd/amd64
- dragonfly/amd64

If we were to choose to build the complete list for tags & master then I would suggest an even more reduced one:

- linux/amd64
- darwin/amd64
- windows/amd64
- freebsd/amd64

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

[Matthias Rampke]

I would build everything on master, that way we catch before starting a release if there is something wrong.

/MR

[Chris Marchbanks]

I also support this, waiting 2-3 hours for the build job to finish is frustrating. I know that building on 32 bit architectures does not catch all issues, specifically the alignment bug using the atomic package. Perhaps add at least one 32 bit build on the pull request though?

Is it worth it to build everything on every master, or should a build all job be added to the nightly build? I agree that we should build everything on a cadence more frequent than a release.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAFU3N5V6MiMGH32a4OB0KMfJmV7FBZbJWEe6HZ-z9%2BmiOkqusQ%40mail.gmail.com.

[Krasimir Georgiev]

I can't remember even a single fail on the 32bit builds so I think it is ok to build these only with a pre-release and a release

[Łukasz Mierzwa]

One way to speed up builds (including cross-compilation) is to re-use the $GOCACHE directory between jobs. Not sure if that's doable with Circle but for TravisCI I have this cache config:

  cache:

    directories:

      - $HOME/.cache/go-build

      - $HOME/gopath/pkg/mod

Note that this directory can get huge (>GB).

[Ben Kochie]

Part of the problem is that we run the builds in serial. We have a lot more compute capacity in our CircleCI for running more docker tasks in parallel, but promu doesn't know how to distribute the work.

But I also think we could cut the build step down for PRs. But I think we should keep the full build in master.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CANVFovU3e_avaLtSnTwYzYz5v-Aj7M_VZSi%3DA05i90T3txZ6EA%40mail.gmail.com.

[Simon Pasquier]

FWIW I'm looking into this. From my understanding, the bulk of the
issue is that "promu crossbuild" can't reuse the Go build cache
between CI runs hence it rebuilds everything from scratch every time.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABbyFmr98ZO%2B0Tgw%2BY4XkjsTMJ68g0J_aKDBEzPMsc7E4Un%2BJg%40mail.gmail.com.

[Sylvain Rabot]

I've also been looking into it.

The sharing the go build cache would be the ultimate solution but for prometheus, for all current goos/goarch, it accounts for more than 6GB and I believe that is too much for circleci to handle (they recommend keeping cache under 500MB).

However I made a PR on promu that replace current crossbuild process with one that (amongst other things) does not use docker anymore and it seems that it halves the duration of building all current goos/goarch:

- https://github.com/prometheus/promu/pull/177

- https://github.com/prometheus/prometheus/pull/6824
- https://app.circleci.com/github/prometheus/prometheus/pipelines/838ffe3c-b738-430b-8cf0-c707842e7e8f/workflows/dd50dabc-04e3-4b48-8a36-c85456668c2f

Regards.

--

Sylvain Rabot <syl...@abstraction.fr>

[Tobias Schmidt]

On Sun, Feb 16, 2020 at 11:38 AM Sylvain Rabot <syl...@abstraction.fr> wrote:

I've also been looking into it.

The sharing the go build cache would be the ultimate solution but for prometheus, for all current goos/goarch, it accounts for more than 6GB and I believe that is too much for circleci to handle (they recommend keeping cache under 500MB).

However I made a PR on promu that replace current crossbuild process with one that (amongst other things) does not use docker anymore and it seems that it halves the duration of building all current goos/goarch:

How does this work for node_exporter with its CGO usage?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CADjtP1GSQea1ATq6eE9crF73kWOsW_kRXB5nROPDHDhemCyvgA%40mail.gmail.com.

[Sylvain Rabot]

On Sun, 16 Feb 2020 at 13:41, Tobias Schmidt <tob...@gmail.com> wrote:

On Sun, Feb 16, 2020 at 11:38 AM Sylvain Rabot <syl...@abstraction.fr> wrote:

I've also been looking into it.

The sharing the go build cache would be the ultimate solution but for prometheus, for all current goos/goarch, it accounts for more than 6GB and I believe that is too much for circleci to handle (they recommend keeping cache under 500MB).

However I made a PR on promu that replace current crossbuild process with one that (amongst other things) does not use docker anymore and it seems that it halves the duration of building all current goos/goarch:

How does this work for node_exporter with its CGO usage?

I don't know, I'll try it. But if it does not work, I moved the docker cross-building in a separate promu command ("promu crossbuild-docker") so we could still use that for node_exporter.

--

Sylvain Rabot <syl...@abstraction.fr>

[Julien Pivotto]

What is the reasoning behind the removal of the docker build?

----- Original Message -----
From: Sylvain Rabot <syl...@abstraction.fr>
To: Tobias Schmidt <tob...@gmail.com>
Cc: Simon Pasquier <spas...@redhat.com>, Ben Kochie <sup...@gmail.com>, Chris Marchbanks <csmarc...@gmail.com>, Matthias Rampke <m...@soundcloud.com>, Krasimir Georgiev <kgeo...@redhat.com>, Prometheus Developers <prometheus...@googlegroups.com>
Sent: Sun, 16 Feb 2020 14:19:55 +0100 (CET)
Subject: Re: [prometheus-developers] Reduce list of GOOS/GOARCH crossbuilt for each PR

On Sun, 16 Feb 2020 at 13:41, Tobias Schmidt <tob...@gmail.com> wrote:

>
>
> On Sun, Feb 16, 2020 at 11:38 AM Sylvain Rabot <syl...@abstraction.fr>
> wrote:
>
>> I've also been looking into it.
>>
>> The sharing the go build cache would be the ultimate solution but for
>> prometheus, for all current goos/goarch, it accounts for more than 6GB and
>> I believe that is too much for circleci to handle (they recommend keeping
>> cache under 500MB).
>>
>> However I made a PR on promu that replace current crossbuild process with
>> one that (amongst other things) does not use docker anymore and it seems

>> that it *halves* the duration of building all current goos/goarch:

>>> https://groups.google.com/d/msgid/prometheus-developers/2508CDF1-CC2A-4AC6-B9EE-D68B53AFF166%40getmailspring.com
>>> .

>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Sylvain Rabot <syl...@abstraction.fr>
>>> >>>
>>> >>> --
>>> >>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> >>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to prometheus-devel...@googlegroups.com.
>>> >>> To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/prometheus-developers/CAFU3N5V6MiMGH32a4OB0KMfJmV7FBZbJWEe6HZ-z9%2BmiOkqusQ%40mail.gmail.com
>>> .

>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> >> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to prometheus-devel...@googlegroups.com.
>>> >> To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/prometheus-developers/CANVFovU3e_avaLtSnTwYzYz5v-Aj7M_VZSi%3DA05i90T3txZ6EA%40mail.gmail.com
>>> .

>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups "Prometheus Developers" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-devel...@googlegroups.com.
>>> > To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/prometheus-developers/CABbyFmr98ZO%2B0Tgw%2BY4XkjsTMJ68g0J_aKDBEzPMsc7E4Un%2BJg%40mail.gmail.com
>>> .

>>>
>>>
>>
>> --
>> Sylvain Rabot <syl...@abstraction.fr>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-devel...@googlegroups.com.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/prometheus-developers/CADjtP1GSQea1ATq6eE9crF73kWOsW_kRXB5nROPDHDhemCyvgA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/prometheus-developers/CADjtP1GSQea1ATq6eE9crF73kWOsW_kRXB5nROPDHDhemCyvgA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .

>>
>

--
Sylvain Rabot <syl...@abstraction.fr>

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CADjtP1ExeX9Hd01A%3D5GJ%3Dth63WDz5Vev0eDzF-bRTxq1tAugyQ%40mail.gmail.com.

[Sylvain Rabot]

I'm not removing it, I am adding an in host crossbuild process.

Results I have show that doing the full cross-building on the host is 10 minutes faster (and not twice faster like I said previously) that the 2 hours it takes with docker on prometheus/prometheus (https://app.circleci.com/jobs/github/prometheus/prometheus/25973).

The real improvement (without sharing go cache across builds) comes from removing the "-a" flag in the go build flags.

--

Sylvain Rabot <syl...@abstraction.fr>

[Sylvain Rabot]

So I tried to reproduce what has been done in the golang-builder docker image on the circleci host to be able to build with cgo but I'm stuck on an error I do not understand:

https://app.circleci.com/jobs/github/prometheus/node_exporter/7835

~ building up to 1 concurrent crossbuilds

~ building up to 1 concurrent binaries

< building platform darwin/amd64

< builing binary darwin/amd64 node_exporter

+ GOOS=darwin GOARCH=amd64 go build -o .build/darwin-amd64/node_exporter -ldflags "-X github.com/prometheus/common/version.Version=0.18.1 -X github.com/prometheus/common/version.Revision=eba0002f1fba4f032722566699dc9036107a9a43 -X github.com/prometheus/common/version.Branch=pull/1606 -X github.com/prometheus/common/version.BuildUser=circleci@default-5cf5bcf5-1942-4bca-b1f3-b262e5474fe1 -X github.com/prometheus/common/version.BuildDate=20200216-20:57:30 -extldflags '-static'" -mod=vendor -tags 'netgo static_build' github.com/prometheus/node_exporter

# crypto/x509

ld: warning: object file (/tmp/go-build262275209/b077/_cgo_main.o) was built for newer OSX version (10.10) than being linked (10.6)

ld: warning: object file (/tmp/go-build262275209/b077/_x001.o) was built for newer OSX version (10.10) than being linked (10.6)

ld: warning: object file (/tmp/go-build262275209/b077/_x002.o) was built for newer OSX version (10.10) than being linked (10.6)

# github.com/prometheus/node_exporter

/usr/local/go/pkg/tool/linux_amd64/link: running o64-clang failed: exit status 1

ld: library not found for -lcrt0.o

clang: error: linker command failed with exit code 1 (use -v to see invocation)

If anyone has insights about this I would be very grateful.

--

Sylvain Rabot <syl...@abstraction.fr>