linux-arm builders/trybots seem to be hanging

[Ian Lance Taylor]

On build.golang.org, the linux-arm builders have hung since
https://golang.org/cl/27473 was committed. The linux-arm trybots also
seem to be hanging; at least, when I look at
http://farmer.golang.org/try?commit=f847c07a, all the trybots have
completed (and passed) but the linux-arm trybots are still getting
started. I note that https://golang.org/cl/27473 never had a
successful trybot run. I don't know whether this is a hardware
problem or a problem with the CL.

Ian

[Ian Lance Taylor]

Sorry, I missed that patchset 1 of CL 27473 had a successful trybot
run, but none of the later patchsets did.

Ian

[Brad Fitzpatrick]

Trybots stop running once it's been submitted.

On Mon, Aug 22, 2016 at 7:28 PM, Ian Lance Taylor <ia...@golang.org> wrote:

[Brad Fitzpatrick]

... or if a new patchset is updated.

[David Chase]

That was the CL that killed PPCLE builds, and the CL that fixes them is https://go-review.googlesource.com/c/27511/

Perhaps something similar is requried on Arm.  The symptom, on a PPC64LE box, is that ./make.bash

runs just fine, but the resulting Go binary dies with "file not found" because of a reference to a missing

(old) version of ld.so.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad Fitzpatrick]

I see the 50 arm builders running make.bash and then failing to find the list of tests to run:

Error: runTests: distTestList: Remote error: fork/exec /workdir/go/bin/go: no such file or directory, 

And then things get confused.

I can modify the builder to handle that better.

[Ian Lance Taylor]

On Mon, Aug 22, 2016 at 7:46 PM, David Chase <drc...@google.com> wrote:

> That was the CL that killed PPCLE builds, and the CL that fixes them is
> https://go-review.googlesource.com/c/27511/
>
> Perhaps something similar is requried on Arm. The symptom, on a PPC64LE
> box, is that ./make.bash
> runs just fine, but the resulting Go binary dies with "file not found"
> because of a reference to a missing
> (old) version of ld.so.

How did it work before? Before CL 27473 I don't see anything that
sets the dynamic linker to /lib64/ld64.so.2.

Ian

[Ian Lance Taylor]

On Mon, Aug 22, 2016 at 8:00 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
> I see the 50 arm builders running make.bash and then failing to find the
> list of tests to run:
>
> Error: runTests: distTestList: Remote error: fork/exec /workdir/go/bin/go:
> no such file or directory,
>
> And then things get confused.
>
> I can modify the builder to handle that better.

That is also what I am seeing using gomote.

That does make it seem like a dynamic interpreter problem.

Ian

[Ian Lance Taylor]

The code to handle the cgo_dynamic_linker pragma is broken. It no
longer correctly tests whether the flag was set.

It normally should not matter what the linker thinks the default
interpreter is, because for a cgo program it comes from
cgo_dynamic_linker and a non-cgo program is normally linked statically
and so the interpreter doesn't matter. David's patch to set the right
interpreter is a good change but it covers up the actual bug.

Will send a fix later tonight.

Ian

[David Chase]

"...go/bin/go: no such file or directory" is exactly what I saw.

[Michael Matloob]

I'm wondering if we should roll back that change

[Brad Fitzpatrick]

Probably? Or find the fix and roll forward is fine too. It's dev cycle. Yay!

You only ran the trybots on patchset 1 of that CL.  You tried a bunch of other times, but then it kept getting stopped whenever a new patchset was uploaded.

It got up to patchset 8 and the trybots never ran from PS 1 (which was happy) to:

The diff from PS1 to PS8 (the untested parts) is quite large:

https://go-review.googlesource.com/c/27473/1..8

[Michael Hudson-Doyle]

On 23 August 2016 at 15:54, Brad Fitzpatrick <brad...@golang.org> wrote:
> Probably? Or find the fix and roll forward is fine too. It's dev cycle. Yay!

FWIW, https://go-review.googlesource.com/#/c/27571

Happy to take Ian's fix instead if he had something significantly
different in mind.

Cheers,
mwh

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to golang-dev+...@googlegroups.com.

[Ian Lance Taylor]

On Mon, Aug 22, 2016 at 8:57 PM, Michael Hudson-Doyle
<michael...@canonical.com> wrote:
> On 23 August 2016 at 15:54, Brad Fitzpatrick <brad...@golang.org> wrote:
>> Probably? Or find the fix and roll forward is fine too. It's dev cycle. Yay!
>
> FWIW, https://go-review.googlesource.com/#/c/27571
>
> Happy to take Ian's fix instead if he had something significantly
> different in mind.

https://golang.org/cl/27562

Ian

[Brad Fitzpatrick]

On Mon, Aug 22, 2016 at 8:54 PM, Brad Fitzpatrick <brad...@golang.org> wrote:

Probably? Or find the fix and roll forward is fine too. It's dev cycle. Yay!

You only ran the trybots on patchset 1 of that CL.  You tried a bunch of other times, but then it kept getting stopped whenever a new patchset was uploaded.

It got up to patchset 8 and the trybots never ran from PS 1 (which was happy) to:

The diff from PS1 to PS8 (the untested parts) is quite large:

https://go-review.googlesource.com/c/27473/1..8

It look like the last trybot kick-off was 6:30pm (some timezone) and it hadn't finished at 7:09 AM (AM!) the next morning when the next patchset was uploaded, which would've canceled the build.

You likely thought the trybots were screwed at that point (they were: but because they were hung on the CL and not reporting the error properly) and submitted without waiting for them.

In general, if the trybots don't finish in 10-15 minutes, feel free to ignore them. It's my fault at that point. But let me know anyway.

When the new OS X builders come online soon, I hope to drop the expected wait time for trybots down to more like 5 minutes.