ARM v5 in QEMU: Illegal instruction

1,402 views
Skip to first unread message

lbovet

unread,
Dec 18, 2010, 11:10:43 AM12/18/10
to golang-nuts
Hello,

I am aiming at go-developing for the Sharp Zaurus SL-5500, which
features an ARM v4 processor (DEC SL-1100).

In order to have an emulated target platform close to this one, I
tried to build go on the following qemu'ed debian etch:
http://people.debian.org/~aurel32/qemu/arm/

I took care of setting GOARM=5, could build the 5* tools from the
latest hg distrib of go.

Unfortunately I get the dreaded "Illegal instruction" whatever I try
to execute (even without FP). Note that the default emulated processor
is a ARM926EJ-S (ARM v5).

Have some of you any recipe to get a ARM v5 proc execute go-compiled
code? Could you give me some pointers to get some debugging
information about the illegal instruction (which one, in which
function)?

bflm

unread,
Dec 18, 2010, 11:53:09 AM12/18/10
to golang-nuts
On Dec 18, 5:10 pm, lbovet <laurent.bo...@gmail.com> wrote:
> Unfortunately I get the dreaded "Illegal instruction" whatever I try
> to execute (even without FP).
The trap happens in 5g or the app compiled by 5g?

>Note that the default emulated processor
> is a ARM926EJ-S (ARM v5).
>
> Have some of you any recipe to get a ARM v5 proc execute go-compiled
> code? Could you give me some pointers to get some debugging
> information about the illegal instruction (which one, in which
> function)?
Don't know how about ARM, but generally Go GDB support has recently
improved really a lot (though here is x86-64). Is it possible to try
running the problem app within GDB inside the QEMUlated SL-5500 OS
(which is AFAIK Linux)? It may, hopefully, then identify the problem
spot with file/line/func info.

lbovet

unread,
Dec 18, 2010, 12:13:01 PM12/18/10
to golang-nuts

> The trap happens in 5g or the app compiled by 5g?

5g and 5l compile. The trap happens when launching the compiled
executable.

> >Note that the default emulated processor
> > is a ARM926EJ-S (ARM v5).
> Don't know how about ARM, but generally Go GDB support has recently
> improved really a lot (though here is x86-64). Is it possible to try
> running the problem app within GDB inside the QEMUlated SL-5500 OS
> (which is AFAIK Linux)? It may, hopefully, then identify the problem
> spot with file/line/func info.

For now, I just try to make that run in a debian etch in QEMU. I am
now trying to figure out how to enable the file/line/func info... All
I get from gdb is:

user@debian-arm:~$ gdb 5.out
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "arm-linux-gnu"...(no debugging symbols
found)
Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) run
Starting program: /home/user/5.out
warning: shared library handler failed to enable breakpoint

Program received signal SIGILL, Illegal instruction.
0x00019a88 in ?? ()
(gdb)

bflm

unread,
Dec 18, 2010, 12:45:31 PM12/18/10
to golang-nuts
On Dec 18, 6:13 pm, lbovet <laurent.bo...@gmail.com> wrote:
> For now, I just try to make that run in a debian etch in QEMU. I am
> now trying to figure out how to enable the file/line/func info... All
> I get from gdb is:
>
> user@debian-arm:~$ gdb 5.out
> GNU gdb 6.4.90-debian
The much improved GDB support requires GDB 7.X, but just now I tried
to verify that and I can see - it covers only 6g and 8g, sorry:
http://blog.golang.org/2010/11/debugging-go-code-status-report.html

Hope someone from the dev team can hint/help you anyway.

ron minnich

unread,
Dec 18, 2010, 1:51:19 PM12/18/10
to bflm, golang-nuts
FWIW when I have this kind of weird problem I get into gdb with the
binary and use
display/i $pc


then stepi to walk the assembly. When it is a strange problem like
this the error can frequently be found in the very early code.

ron

Skip Tavakkolian

unread,
Dec 18, 2010, 2:47:28 PM12/18/10
to lbovet, golang-nuts
i think this is a problem with the latest release. i see the same
thing on sheeva plug after pulling the 12/15 release. other than the
problem noted in issue #1240, previous releases were generally
functional.

-Skip

lbovet

unread,
Dec 18, 2010, 3:27:53 PM12/18/10
to golang-nuts
What was the last release reported to work on your sheeva plug?

On 18 déc, 20:47, Skip Tavakkolian <skip.tavakkol...@gmail.com> wrote:
> i think this is a problem with the latest release.  i see the same
> thing on sheeva plug after pulling the 12/15 release. other than the
> problem noted in issue #1240, previous releases were generally
> functional.
>
> -Skip
>

lbovet

unread,
Dec 19, 2010, 3:34:48 PM12/19/10
to golang-nuts
I tried with release.2010-12-02, reported to work on sheeva plug (same
proc as emulated one) by Skip.

Same result:

user@debian-arm:~$ 5g -V
5g version 6821 release.2010-12-02
user@debian-arm:~$ 5g plusthree.go
user@debian-arm:~$ 5l -V
5l version 6821 release.2010-12-02
user@debian-arm:~$ 5l plusthree.5
user@debian-arm:~$ ./5.out
Illegal instruction



user@debian-arm:~$ cat plusthree.go
package main

func main() {
var n int
for d:=3; d>0; d-- {
n = n + 1

Kai Backman

unread,
Dec 31, 2010, 5:19:28 PM12/31/10
to lbovet, golang-nuts
I'm a bit late to the party but I'd like to higlight what Ron said earlier:

On Sat, Dec 18, 2010 at 8:51 PM, ron minnich <rmin...@gmail.com> wrote:
> FWIW when I have this kind of weird problem I get into gdb with the
> binary and use
> display/i $pc
>
>
> then stepi to walk the assembly. When it is a strange problem like

To fix something like this you need to tell what the illegal
instruction was. Especially if it is a regression like Skip suspects.

Kai

--
Kai Backman, programmer
http://tinkercad.com - The unprofessional solid CAD

lbovet

unread,
Jan 3, 2011, 4:31:32 PM1/3/11
to golang-nuts
Here is what I do in the qemu'ed etch:

user@debian-arm:~$ cat hello.go
package main

import "fmt"

func main() {
fmt.Printf("hello, world\n")
}
user@debian-arm:~$ 5g hello.go && 5l -F hello.5 && gdb 5.out
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "arm-linux-gnu"...(no debugging symbols
found)
Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) run
Starting program: /home/user/5.out
warning: shared library handler failed to enable breakpoint

Program received signal SIGILL, Illegal instruction.
0x000210ac in ?? ()
(gdb) x/10i $pc
0x210ac: add pc, lr, #0 ; 0x0
0x210b0: mov r7, #173 ; 0xad
0x210b4: svc 0x00000000
0x210b8: add pc, lr, #0 ; 0x0
0x210bc: ldr r1, [r10]
0x210c0: cmp sp, r1
0x210c4: movcc r1, #0 ; 0x0
0x210c8: movcc r2, #12 ; 0xc
0x210cc: movcc r3, lr
0x210d0: blcc 0xd314

Note that the same code fails a the same place on a Sharp Zaurus
SL-5500, which
features an ARM v4 processor (DEC SL-1100), but with a segfault:

root@collie:/home/shaman# gdb 5.out
GNU gdb 5.2
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "armv4l-unknown-linux-gnu"...
(no debugging symbols found)...
(gdb) run
Starting program: /home/shaman/5.out
warning: shared library handler failed to enable breakpoint

Program received signal SIGSEGV, Segmentation fault.
0x000210ac in ?? ()
(gdb) x/10i $pc
0x210ac: add pc, lr, #0 ; 0x0
0x210b0: mov r7, #173 ; 0xad
0x210b4: swi 0x00000000
0x210b8: add pc, lr, #0 ; 0x0
0x210bc: ldr r1, [r10]
0x210c0: cmp sp, r1
0x210c4: movcc r1, #0 ; 0x0
0x210c8: movcc r2, #12 ; 0xc
0x210cc: movcc r3, lr
0x210d0: blcc 0xd314


On 31 déc 2010, 23:19, Kai Backman <kai.back...@gmail.com> wrote:
> I'm a bit late to the party but I'd like to higlight what Ron said earlier:
>

lbovet

unread,
Jan 3, 2011, 5:32:56 PM1/3/11
to golang-nuts
On the qemu etch, I disassembled a bit around the $pc and found this
at 0x21034:

0x00021020: bl 0x20edc
0x00021024: ldr r0, [pc, #12] ; 0x21038
0x00021028: ldr r1, [pc, #12] ; 0x2103c
0x0002102c: str r0, [r1]
0x00021030: b 0x21030
0x00021034: undefined instruction 0xfffffff0
0x00021038: ldreqd r0, [r0], -r2
0x0002103c: andeq r0, r0, sp, ror #7
0x00021040: ldr r0, [sp, #4]
0x00021044: ldr r1, [sp, #8]
0x00021048: mov r2, #0 ; 0x0
0x0002104c: ldr r7, [pc, #8] ; 0x2105c
0x00021050: svc 0x00000000
0x00021054: add pc, lr, #0 ; 0x0
0x00021058: b 0x21058
0x0002105c: andeq r0, pc, r2
0x00021060: ldr r0, [sp, #4]
0x00021064: ldr r1, [sp, #8]
0x00021068: mov r7, #186 ; 0xba
0x0002106c: svc 0x00000000
0x00021070: add pc, lr, #0 ; 0x0
0x00021074: add pc, lr, #0 ; 0x0
0x00021078: str lr, [sp, #-28]!
0x0002107c: ldr r10, [r9, #44]
0x00021080: str r0, [sp, #4]
0x00021084: str r1, [sp, #8]
0x00021088: str r2, [sp, #12]
0x0002108c: bl 0x1dbf8
0x00021090: ldr pc, [sp], #28
0x00021094: ldr r0, [sp, #4]
0x00021098: ldr r1, [sp, #8]
0x0002109c: ldr r2, [sp, #12]
0x000210a0: ldr r3, [sp, #16]
0x000210a4: mov r7, #174 ; 0xae
0x000210a8: svc 0x00000000
0x000210ac: add pc, lr, #0 ; 0x0

Can it be the cause?

Interestingly, the same assembly portion shows another instruction on
the SL-5500:

0x21000: swi 0x00000000
0x21004: str r0, [r9, #36]
0x21008: ldr r0, [sp, #8]
0x2100c: add sp, sp, #16 ; 0x10
0x21010: add lr, pc, #0 ; 0x0
0x21014: add pc, r0, #0 ; 0x0
0x21018: mov r0, #0 ; 0x0
0x2101c: str r0, [sp, #4]
0x21020: bl 0x20edc
0x21024: ldr r0, [pc, #12] ; 0x21038
0x21028: ldr r1, [pc, #12] ; 0x2103c
0x2102c: str r0, [r1]
0x21030: b 0x21030
0x21034: swinv 0x00fffff0
0x21038: ldreqd r0, [r0], -r2
0x2103c: andeq r0, r0, sp, ror #7
0x21040: ldr r0, [sp, #4]
0x21044: ldr r1, [sp, #8]
0x21048: mov r2, #0 ; 0x0
0x2104c: ldr r7, [pc, #8] ; 0x2105c
0x21050: swi 0x00000000
0x21054: add pc, lr, #0 ; 0x0

With my minimal knowledge about this, that could explain the
difference of behaviour between both systems. However, this wrong
instruction is far from the $pc where the signal is caught. Maybe the
pipeline, but I doubt it contains 30 instructions.

I am absolutely assembly-illiterate but I am investigating anyway
further... Please tell me if you see something obvious for you.

Kai Backman

unread,
Jan 4, 2011, 4:11:31 AM1/4/11
to lbovet, golang-nuts
The value in 0x21034 is probably data.

Can you dump the value of lr at the crash site? My guess is that it's
0 which would explain the crash.

The next step would be to test with older versions of go and try to
pinpoint which CL caused the regression. Skip says it is a problem
with the newest release so you could start by binary searching that
interval.

Kai

--

lbovet

unread,
Jan 4, 2011, 9:52:10 AM1/4/11
to golang-nuts
I don't think it is a regression, because some other user sent me a
"hello world" 5.out that works on his sheeva plug but raises the same
exception on both my systems (qemu and sl-5500) and I had the same
issue with older go versions as well. In fact I never had a working go
executable on both these systems...

Could you tell me how I can find from which module comes this bizarre
instruction? Should I try to search the instruction byte sequence
inside the .a files?
> >> > Kai Backman, programmerhttp://tinkercad.com-Theunprofessional solid CAD
>
> --

lbovet

unread,
Jan 4, 2011, 4:43:34 PM1/4/11
to golang-nuts
The value of lr after crash is not zero:

(gdb) run
Starting program: /home/user/5.out
warning: shared library handler failed to enable breakpoint

Program received signal SIGILL, Illegal instruction.
0x000210ac in ?? ()
(gdb) info registers
r0 0x3 3
r1 0xe6924 944420
r2 0x0 0
r3 0x8 8
r4 0xbf114 782612
r5 0x0 0
r6 0x0 0
r7 0xae 174
r8 0x0 0
r9 0xe7308 946952
r10 0xe6f8c 946060
r11 0xe6928 944424
r12 0xcafebabe -889275714
sp 0xbef90a18 -1090975208
lr 0x1dfe4 122852
pc 0x210ac 135340
fps 0x1001000 16781312
cpsr 0x60000010 1610612752
> ...
>
> plus de détails »

ron minnich

unread,
Jan 5, 2011, 11:06:07 AM1/5/11
to lbovet, golang-nuts


You're close but you're focusing on the wrong thing. The add pc, lr,
#0 is a pretty simple register-to-register move that's not going to
cause a fault. Remember, when you see that PC is x when you stop, the
faulting instruction was at x-4. The swc 0x0000000 may be a bad thing.
It's a supervisor call. So at the start I'm not real clear on two
things:
1) why the go linker would emit that instruction
2) how the kernel on different ARMs manage that instruction

So it's a 2-deep puzzle at the moment.

It's interesting to consider the following little sequence however:


> 0x210b0:        mov     r7, #173        ; 0xad
> 0x210b4:        svc     0x00000000
> 0x210b8:        add     pc, lr, #0      ; 0x

The first instruction puts 173 in R7; the second does a software
interrupt; what's that third thing?

It's moving the lr (link register) to pc, and on arm, you get an
immediate 16-bit thing you can add for free; so what this looks like
to me is moving the lr to pc. What's the lr?
See any function call:
bl xyz
branch and link: branches to xyz, puts a return pc in the lr.
SO, add pc, lr, #0 is nothing more I think than a
return

See http://infocenter.arm.com/help/topic/com.arm.doc.qrc0001m/QRC0001_UAL.pdf
for more info.

173 is sigreturn.

So my guess here is you've got a syscall which the kernel is upset
about and the kernel lets you know by sending you a signal. This is a
possible outcome when the kernel is very upset with something you're
trying to do :-)

OK, let's look at libc.
gdb /usr/lib/libc.so.6, and
disass syscall

and you'll see that the standard system call sequence on arm is basically
set up args
system call in r7
swc 0

So I guessed right. It's very useful sometimes to take a known-good
thing like a shared library and use gdb to show you sequences of code
for comparison.

Note that gcc is callee-save and 5g looks like caller-save (as in Plan
9 compilers) so you an see interesting differences in the code.

All guesses based on possibly too little coffee and a long night on
planes, but I wanted to say something since I've been on travel and
not helpful since my last comment in this thread :-) Also, there's a
bit of "teach a man to fish" in this note as you can see :-)

Which leads to my next question: can you try running this program with
strace -f and see what you see?

ron

chris dollin

unread,
Jan 5, 2011, 11:11:23 AM1/5/11
to ron minnich, lbovet, golang-nuts
On 5 January 2011 16:06, ron minnich <rmin...@gmail.com> wrote:

> It's moving the lr (link register) to pc, and on arm, you get an
> immediate 16-bit thing you can add for free

(fx:nitpick)

12 bits, which are an 8-bit value and a 4-bit rotate-twice-this-many-bits count.

IIRC.

Chris

--
Chris "left or your other left?" Dollin

ron minnich

unread,
Jan 5, 2011, 11:53:39 AM1/5/11
to chris dollin, lbovet, golang-nuts
On Wed, Jan 5, 2011 at 8:11 AM, chris dollin <ehog....@googlemail.com> wrote:
> On 5 January 2011 16:06, ron minnich <rmin...@gmail.com> wrote:
>
>> It's moving the lr (link register) to pc, and on arm, you get an
>> immediate 16-bit thing you can add for free
>
> (fx:nitpick)
>
> 12 bits, which are an 8-bit value and a 4-bit rotate-twice-this-many-bits count.
>
> IIRC.

I did not look closely but that makes a lot of sense, since the return
address has to be 32-bit aligned anyway.

The ARM is an interesting architecture, because things seem to have
had some real thought applied; a refreshing change from things you
find in the x86 at times :-)

thanks

ron

lbovet

unread,
Jan 5, 2011, 4:38:15 PM1/5/11
to golang-nuts
user@debian-arm:~$ strace -f ./5.out
execve("./5.out", ["./5.out"], [/* 19 vars */]) = 0
syscall: unknown syscall trap 0x3b9aca00


On 5 jan, 17:53, ron minnich <rminn...@gmail.com> wrote:
> On Wed, Jan 5, 2011 at 8:11 AM, chris dollin <ehog.he...@googlemail.com> wrote:

ron minnich

unread,
Jan 5, 2011, 6:04:38 PM1/5/11
to lbovet, golang-nuts
On Wed, Jan 5, 2011 at 1:38 PM, lbovet <lauren...@gmail.com> wrote:
> user@debian-arm:~$ strace -f ./5.out
> execve("./5.out", ["./5.out"], [/* 19 vars */]) = 0
> syscall: unknown syscall trap 0x3b9aca00

well, that's interesting.

to recap, sorry, my memory is not so good at this point, when you
don't use strace you just get an illegal instruction trap?

ron

ron minnich

unread,
Jan 5, 2011, 6:23:37 PM1/5/11
to lbovet, golang-nuts
On Wed, Jan 5, 2011 at 1:38 PM, lbovet <lauren...@gmail.com> wrote:
> user@debian-arm:~$ strace -f ./5.out
> execve("./5.out", ["./5.out"], [/* 19 vars */]) = 0
> syscall: unknown syscall trap 0x3b9aca00

Here's what my arm binaries do when they start.
rt_sigaction(SIGQUIT, {0x3fb10, ~[], SA_STACK|SA_SIGINFO|0x4000000}, NULL, 8) =
0
rt_sigaction(SIGILL, {0x3fb10, ~[], SA_STACK|SA_SIGINFO|0x4000000}, NULL, 8) = 0
rt_sigaction(SIGTRAP, {0x3fb10, ~[], SA_STACK|SA_SIGINFO|0x4000000}, NULL, 8) =
0

First thing after the execve.

http://linux.die.net/man/2/rt_sigaction

Now, as weird as this sounds, it would be nice to verify that whatever
kernel you're running agrees with your binary about the support for
that system call at that system call number. It's a little hard for me
to believe it does not, but .... something really weird going on here.

Here's a simple test I did to make sure *something* uses that call on
my arm system, and that strace
can see and report about it:
strace -e trace=rt_sigaction /bin/ls
rt_sigaction(SIGRTMIN, {0x401ad2d8, [], SA_SIGINFO|0x4000000}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x401ad170, [],
SA_RESTART|SA_SIGINFO|0x4000000}, NULL, 8) = 0


ron

Reply all
Reply to author
Forward
0 new messages