ARMv7 backtrace help needed

[Nick Craig-Wood]

Can anyone help me interpret this backtrace from an ARMv7 NAS running my
rclone program?

* Why are there so many init() functions in the backtrace?
* Why does
/home/ncw/Code/Go/src/github.com/ncw/rclone/redirect_stderr_unix.go
appear in the backtrace - I don't think that is run at all
* Why did it crash? Signal 0xb is SEGV I think.

I suspect some sort of incompatibility between the go runtime and this
particular ARM device as rclone usually runs fine on ARM.

OS: ARMv7l (on Western Digital NAS)
#uname -a
Linux NasWD 3.2.26 #1 SMP Thu Jul 9 11:14:15 PDT 2015 wd-2.4-rel armv7l
GNU/Linux
The command: GOTRACEBACK=system rclone config

The binary is http://downloads.rclone.org/rclone-v1.28-linux-arm.zip

The code is here: https://github.com/ncw/rclone/tree/v1.28

rclone issue is here: https://github.com/ncw/rclone/issues/426

Full backtrace: https://github.com/ncw/rclone/files/208682/rclone_err.txt

Any help most appreciated!

----------------------

unexpected fault address 0x4e1648
fatal error: fault
[signal 0xb code=0x2 addr=0x4e1648 pc=0x4e1648]

goroutine 1 [running, locked to thread]:
runtime.throw(0x618288, 0x5)
/usr/local/go/src/runtime/panic.go:530 +0x78 fp=0x10be5ed8 sp=0x10be5ecc
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x10be5f04
sp=0x10be5ed8
github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init()
/home/ncw/Code/Go/src/github.com/aws/aws-sdk-go/private/protocol/query/queryutil/queryutil.go:230 fp=0x10be5f08 sp=0x10be5f08
github.com/aws/aws-sdk-go/private/protocol/query.init()
/home/ncw/Code/Go/src/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal_error.go:40 +0x58 fp=0x10be5f0c sp=0x10be5f08
github.com/aws/aws-sdk-go/private/protocol/restxml.init()
/home/ncw/Code/Go/src/github.com/aws/aws-sdk-go/private/protocol/restxml/restxml.go:69 +0x5c fp=0x10be5f10 sp=0x10be5f0c
github.com/aws/aws-sdk-go/service/s3.init()
/home/ncw/Code/Go/src/github.com/aws/aws-sdk-go/service/s3/waiters.go:117 +0x60 fp=0x10be5f34 sp=0x10be5f10
github.com/ncw/rclone/s3.init()
/home/ncw/Code/Go/src/github.com/ncw/rclone/s3/v2sign.go:115 +0x8c
fp=0x10be5f58 sp=0x10be5f34
github.com/ncw/rclone/fs/all.init()
/home/ncw/Code/Go/src/github.com/ncw/rclone/fs/all/all.go:16 +0x6c
fp=0x10be5f5c sp=0x10be5f58
main.init()
/home/ncw/Code/Go/src/github.com/ncw/rclone/redirect_stderr_unix.go:19
+0x70 fp=0x10be5f84 sp=0x10be5f5c
runtime.main()
/usr/local/go/src/runtime/proc.go:177 +0x274 fp=0x10be5fac sp=0x10be5f84
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:990 +0x4 fp=0x10be5fac sp=0x10be5fac

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[Dave Cheney]

Which version of go was this built with?

[Nick Craig-Wood]

On 08/04/16 12:03, Dave Cheney wrote:
> Which version of go was this built with?

Sorry, should have put that in the original...

$ go version
go version go1.6 linux/amd64

cross compiled for linux/arm

[Dave Cheney]

I'm not sure how that error is happening. There are only three revisions of queryutil.go in the aws-go-sdk repo and none of them have any package level variables or other declarations which would generate an init function.

[Nick Craig-Wood]

I disassembled the rclone binary and apparently there is an init in the
queryutil.go - it seems to mostly be for calling the init functions for
imported functions

The SEGV happens on the first instruction which indicates r10 (AKA g) is
corrupt maybe.

004e1648 <github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init>:
4e1648: e59a1008 ldr r1, [r10, #8]
4e164c: e15d0001 cmp r13, r1
4e1650: 9a00001a bls 4e16c0
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x78>
4e1654: e52de004 push {r14} ; (str r14, [r13, #-4]!)
4e1658: e59fb070 ldr r11, [pc, #112] ; 4e16d0
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x88>
4e165c: e5db0000 ldrb r0, [r11]
4e1660: e3500000 cmp r0, #0
4e1664: 0a000006 beq 4e1684
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x3c>
4e1668: e59fb060 ldr r11, [pc, #96] ; 4e16d0
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x88>
4e166c: e5db0000 ldrb r0, [r11]
4e1670: e3500002 cmp r0, #2
4e1674: 1a000000 bne 4e167c
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x34>
4e1678: e49df004 pop {r15} ; (ldr r15, [r13], #4)
4e167c: ebed7eaa bl 4112c <runtime.throwinit>
4e1680: f7fabcfd ; <UNDEFINED> instruction: 0xf7fabcfd
4e1684: e3a00001 mov r0, #1
4e1688: e59fb040 ldr r11, [pc, #64] ; 4e16d0
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x88>
4e168c: e5cb0000 strb r0, [r11]
4e1690: ebf3a0a2 bl 1c9920 <encoding/base64.init>
4e1694: ebeec880 bl 9389c <fmt.init>
4e1698: ebf5e301 bl 25a2a4 <net/url.init>
4e169c: ebf12714 bl 12b2f4 <reflect.init>
4e16a0: ebf0cb9f bl 114524 <strconv.init>
4e16a4: ebef2aba bl ac194 <strings.init>
4e16a8: ebef8028 bl c1750 <time.init>
4e16ac: ebfead26 bl 48cb4c <github.com/aws/aws-sdk-go/private/protocol.init>
4e16b0: e3a00002 mov r0, #2
4e16b4: e59fb014 ldr r11, [pc, #20] ; 4e16d0
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x88>
4e16b8: e5cb0000 strb r0, [r11]
4e16bc: e49df004 pop {r15} ; (ldr r15, [r13], #4)
4e16c0: e1a0300e mov r3, r14
4e16c4: ebee5a79 bl 780b0 <runtime.morestack_noctxt>
4e16c8: eaffffde b 4e1648
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init>
4e16cc: eafffffe b 4e16cc
<github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init+0x84>
4e16d0: 0090aa09 addseq r10, r0, r9, lsl #20

[Anthony Martin]

Dave Cheney <da...@cheney.net> once said:
> I'm not sure how that error is happening. There are only three revisions of
> queryutil.go in the aws-go-sdk repo and none of them have any package level
> variables or other declarations which would generate an init function.

You still get an init function if you import a package that, itself, has an
init function.

I think something odd is going on. Note that the signal was a SIGSEGV
with an si_code of SEGV_ACCERR. Curiously, the panic happens in

github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init

which lies at the end of the text segment in Nick's binary.

Anthony

[Nick Craig-Wood]

I got the user to try running the go compiler as downloaded from
golang.org which had the same SIGSEGV.

I also tried compiling rclone with tip with exactly the same results.

Here is a cat /proc/cpuinfo just in case that sheds any light!

Processor : ARMv7 Processor rev 1 (v7l)
processor : 0
BogoMIPS : 1299.25
processor : 1
BogoMIPS : 1292.69
Features : swp half thumb fastmult vfp edsp neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc09
CPU revision : 1
Hardware : Comcerto 2000 EVM
Revision : 0001
Serial : 0000000000000000


Backtrace compiled with tip

GOTRACEBACK=system rclone-v1.28-28-g9539bbf-arm-go-tip config

unexpected fault address 0x3dddd0
fatal error: fault
[signal 0xb code=0x2 addr=0x3dddd0 pc=0x3dddd0]

goroutine 1 [running, locked to thread]:

runtime.throw(0x45a6df, 0x5)
/home/ncw/Code/go/src/runtime/panic.go:566 +0x78 fp=0x10909ed0
sp=0x10909ec4
runtime.sigpanic()
/home/ncw/Code/go/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x10909efc
sp=0x10909ed0
github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init()
/home/ncw/Code/go/src/github.com/aws/aws-sdk-go/private/protocol/query/queryutil/queryutil.go:231 fp=0x10909f00 sp=0x10909f00
github.com/aws/aws-sdk-go/private/protocol/query.init()
/home/ncw/Code/go/src/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal_error.go:67 +0x58 fp=0x10909f04 sp=0x10909f00
github.com/aws/aws-sdk-go/private/protocol/restxml.init()
/home/ncw/Code/go/src/github.com/aws/aws-sdk-go/private/protocol/restxml/restxml.go:70 +0x5c fp=0x10909f08 sp=0x10909f04
github.com/aws/aws-sdk-go/service/s3.init()
/home/ncw/Code/go/src/github.com/aws/aws-sdk-go/service/s3/waiters.go:124 +0x60 fp=0x10909f2c sp=0x10909f08
github.com/ncw/rclone/s3.init()
/home/ncw/Code/go/src/github.com/ncw/rclone/s3/v2sign.go:116 +0x8c
fp=0x10909f50 sp=0x10909f2c
github.com/ncw/rclone/fs/all.init()
/home/ncw/Code/go/src/github.com/ncw/rclone/fs/all/all.go:17 +0x6c
fp=0x10909f54 sp=0x10909f50
main.init()
/home/ncw/Code/go/src/github.com/ncw/rclone/redirect_stderr_unix.go:20
+0x70 fp=0x10909f7c sp=0x10909f54
runtime.main()
/home/ncw/Code/go/src/runtime/proc.go:177 +0x274 fp=0x10909fa4
sp=0x10909f7c
runtime.goexit()
/home/ncw/Code/go/src/runtime/asm_arm.s:993 +0x4 fp=0x10909fa4
sp=0x10909fa4


Any help much appreciated, even if it is just a go isn't compatible with
this chip!

Thanks

Nick

[Anthony Martin]

Nick Craig-Wood <ni...@craig-wood.com> once said:
> On 09/04/16 04:24, Anthony Martin wrote:
> > Dave Cheney <da...@cheney.net> once said:
> >> I'm not sure how that error is happening. There are only three revisions of
> >> queryutil.go in the aws-go-sdk repo and none of them have any package level
> >> variables or other declarations which would generate an init function.
> >
> > You still get an init function if you import a package that, itself, has an
> > init function.
> >
> > I think something odd is going on. Note that the signal was a SIGSEGV
> > with an si_code of SEGV_ACCERR. Curiously, the panic happens in
> >
> > github.com/aws/aws-sdk-go/private/protocol/query/queryutil.init
> >
> > which lies at the end of the text segment in Nick's binary.
>
> I got the user to try running the go compiler as downloaded from
> golang.org which had the same SIGSEGV.
>
> I also tried compiling rclone with tip with exactly the same results.

To me, this looks like a page size issue. What device is this? And what
kernel is it running? I seem to remember there was a similar issue in the
past.

Cheers,
Anthony

[Anthony Martin]

It looks like I remembered correctly. The relevant discussion and open
issues are here:

https://groups.google.com/forum/#!topic/golang-nuts/29RraQ-t3gc
https://github.com/syncthing/syncthing/issues/928
https://github.com/golang/go/issues/10180
https://github.com/golang/go/issues/12480

Cheers,
Anthony

[Nick Craig-Wood]

The hardware is a Western Digital NAS running Linux NasWD 3.2.26 #1 SMP

Thu Jul 9 11:14:15 PDT 2015 wd-2.4-rel armv7l GNU/Linux

I see from looking at the links you posted that the WD NAS range
probably has 64k pages, so that is likely the correct diagnosis.

Is there a timescale for fixing those issues - 1.7 maybe?

Also does anyone have the patch for the compiler to enable 64k pages? I
have seen one for 1.4 but not for later?

[Dave Cheney]

You can adjust the value of PhyPageSize 

https://github.com/golang/go/blob/release-branch.go1.6/src/runtime/internal/sys/arch_arm.go

It moves around in releases, but is always called the same.'

Note: while it is safe to make this change on all arm systems, even 4k ones, running a Go binary that thinks the page size is 64k when the page size is actually 4k will _disable_ the scavenger.