build failures for 7.5 on debian armhf

[David Bremner]

I get segfault from racketcgc trying to run compile-startup.rkt

https://buildd.debian.org/status/fetch.php?pkg=racket&arch=armhf&ver=7.5%2Bdfsg2-1&stamp=1577755807&raw=0

I can duplicate the segfault in GDB, but the backtrace looks corrupted
at first glance [1].

I'm happy to gather more data if someone has an idea what to try.

We're tracking this in Debian at [0]

[0]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=948789
[1]: output from gdb

#0 0xb6ea3254 in malloc_stats () from /lib/arm-linux-gnueabihf/libc.so.6
#1 <function called from gdb>
#2 hamt_make2 (kind=-1368709120, shift=747399,
shift@entry=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=3204429948,
key1=0x6, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2594
#3 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#4 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#5 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#6 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#7 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#8 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#9 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#10 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599
#11 0x0044c722 in hamt_make2 (kind=<optimized out>,
shift=<error reading variable: DWARF-2 expression error: Loop detected (257).>, code1=<optimized out>,
key1=<optimized out>, val1=val1@entry=0xb5d15ae8, code2=code2@entry=3204429948, key2=key2@entry=0xb6916410,
val2=val2@entry=0xa) at ./src/racket/src/hash.c:2599

[Matthew Flatt]

Since the build log says "illegal instruction", what instruction is it
trying to execute? Since there's so much variation in supported ARM
instructions, maybe Racket's JIT is trying to use one not supported by
the machine. Or maybe execution has just jumped to a bad place, such as
the middle of an instruction.

In gdb, you should be able to use `disassemble` in the vicinity of the
address where Racket crashes (like 0xb6ea3254), but you may need to use

set arm fallback-mode thumb

or (less likely)

set arm fallback-mode arm

to get sensible results.


If execution appears to have jumped to a bad place, it may be worth
changing line 149 of "jit.h" to comment out `#define NEED_LONG_JUMPS`.
If that changes anything, then maybe the JIT is not switching property
between short- and long-jump mode.

Another possible reason from jumping to a bad place is mismanaging ARM
vs. Thumb function pointers. Disabling Thumb support by changing
"racket/src/lightning/arm/asm.h" could be a way to check that.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/87blr4zkco.fsf%40tethera.net.

[David Bremner]

Matthew Flatt <mfl...@cs.utah.edu> writes:

> Since the build log says "illegal instruction", what instruction is it
> trying to execute? Since there's so much variation in supported ARM
> instructions, maybe Racket's JIT is trying to use one not supported by
> the machine. Or maybe execution has just jumped to a bad place, such as
> the middle of an instruction.
>
> In gdb, you should be able to use `disassemble` in the vicinity of the
> address where Racket crashes (like 0xb6ea3254), but you may need to use
>

I tried both modes, at least to my inexpert eye they look the same.

The hardware claims to be arm v7 (Freescale i.MX53).

(gdb) set arm fallback-mode arm
(gdb) disassemble
Dump of assembler code for function malloc_stats:
0xb6ea324c <+0>: ldr r1, [pc, #328] ; (0xb6ea3398 <malloc_stats+332>)
0xb6ea324e <+2>: ldr r2, [pc, #332] ; (0xb6ea339c <malloc_stats+336>)
0xb6ea3250 <+4>: add r1, pc
0xb6ea3252 <+6>: ldr r3, [pc, #332] ; (0xb6ea33a0 <malloc_stats+340>)
=> 0xb6ea3254 <+8>: stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}
0xb6ea3258 <+12>: add r3, pc
0xb6ea325a <+14>: ldr r2, [r1, r2]
0xb6ea325c <+16>: sub sp, #60 ; 0x3c
0xb6ea325e <+18>: ldr r5, [pc, #324] ; (0xb6ea33a4 <malloc_stats+344>)
0xb6ea3260 <+20>: ldr r2, [r2, #0]
0xb6ea3262 <+22>: str r2, [sp, #52] ; 0x34
0xb6ea3264 <+24>: mov.w r2, #0
0xb6ea3268 <+28>: ldr r2, [r3, #64] ; 0x40
0xb6ea326a <+30>: add r5, pc
0xb6ea326c <+32>: ldr r7, [r3, #36] ; 0x24
0xb6ea326e <+34>: cmp r2, #0
0xb6ea3270 <+36>: blt.w 0xb6ea338c <malloc_stats+320>
0xb6ea3274 <+40>: ldr r3, [pc, #304] ; (0xb6ea33a8 <malloc_stats+348>)
0xb6ea3276 <+42>: add.w r9, sp, #12
0xb6ea327a <+46>: ldr r4, [pc, #304] ; (0xb6ea33ac <malloc_stats+352>)
0xb6ea327c <+48>: movs r6, #0
0xb6ea327e <+50>: Cannot access memory at address 0xb6ea327e
(gdb) set arm fallback-mode thumb
(gdb) disassemble
Dump of assembler code for function malloc_stats:
0xb6ea324c <+0>: ldr r1, [pc, #328] ; (0xb6ea3398 <malloc_stats+332>)
0xb6ea324e <+2>: ldr r2, [pc, #332] ; (0xb6ea339c <malloc_stats+336>)
0xb6ea3250 <+4>: add r1, pc
0xb6ea3252 <+6>: ldr r3, [pc, #332] ; (0xb6ea33a0 <malloc_stats+340>)
=> 0xb6ea3254 <+8>: stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}
0xb6ea3258 <+12>: add r3, pc
0xb6ea325a <+14>: ldr r2, [r1, r2]
0xb6ea325c <+16>: sub sp, #60 ; 0x3c
0xb6ea325e <+18>: ldr r5, [pc, #324] ; (0xb6ea33a4 <malloc_stats+344>)
0xb6ea3260 <+20>: ldr r2, [r2, #0]
0xb6ea3262 <+22>: str r2, [sp, #52] ; 0x34
0xb6ea3264 <+24>: mov.w r2, #0
0xb6ea3268 <+28>: ldr r2, [r3, #64] ; 0x40
0xb6ea326a <+30>: add r5, pc
0xb6ea326c <+32>: ldr r7, [r3, #36] ; 0x24
0xb6ea326e <+34>: cmp r2, #0
0xb6ea3270 <+36>: blt.w 0xb6ea338c <malloc_stats+320>
0xb6ea3274 <+40>: ldr r3, [pc, #304] ; (0xb6ea33a8 <malloc_stats+348>)
0xb6ea3276 <+42>: add.w r9, sp, #12
0xb6ea327a <+46>: ldr r4, [pc, #304] ; (0xb6ea33ac <malloc_stats+352>)
0xb6ea327c <+48>: movs r6, #0
0xb6ea327e <+50>: Cannot access memory at address 0xb6ea327e
(gdb)

[Matthew Flatt]

At Fri, 17 Jan 2020 08:31:22 -0400, David Bremner wrote:
> I tried both modes, at least to my inexpert eye they look the same.
>
> The hardware claims to be arm v7 (Freescale i.MX53).
>
> (gdb) set arm fallback-mode arm
> (gdb) disassemble
> Dump of assembler code for function malloc_stats:
> 0xb6ea324c <+0>: ldr r1, [pc, #328] ; (0xb6ea3398
> <malloc_stats+332>)
> 0xb6ea324e <+2>: ldr r2, [pc, #332] ; (0xb6ea339c
> <malloc_stats+336>)
> 0xb6ea3250 <+4>: add r1, pc
> 0xb6ea3252 <+6>: ldr r3, [pc, #332] ; (0xb6ea33a0
> <malloc_stats+340>)
> => 0xb6ea3254 <+8>: stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}
> 0xb6ea3258 <+12>: add r3, pc

That certainly looks like a valid ARM instruction. Maybe the processor
is expecting Thumb instructions. What does `print $cpsr` report?

[David Bremner]

Matthew Flatt <mfl...@cs.utah.edu> writes:

> At Fri, 17 Jan 2020 08:31:22 -0400, David Bremner wrote:

>> I tried both modes, at least to my inexpert eye they look the same.
>>
>> The hardware claims to be arm v7 (Freescale i.MX53).
>>
>> (gdb) set arm fallback-mode arm
>> (gdb) disassemble
>> Dump of assembler code for function malloc_stats:
>> 0xb6ea324c <+0>: ldr r1, [pc, #328] ; (0xb6ea3398
>> <malloc_stats+332>)
>> 0xb6ea324e <+2>: ldr r2, [pc, #332] ; (0xb6ea339c
>> <malloc_stats+336>)
>> 0xb6ea3250 <+4>: add r1, pc
>> 0xb6ea3252 <+6>: ldr r3, [pc, #332] ; (0xb6ea33a0
>> <malloc_stats+340>)
>> => 0xb6ea3254 <+8>: stmdb sp!, {r4, r5, r6, r7, r8, r9, r10, r11, lr}
>> 0xb6ea3258 <+12>: add r3, pc
>

> That certainly looks like a valid ARM instruction. Maybe the processor
> is expecting Thumb instructions. What does `print $cpsr` report?

(gdb) print $cpsr
$3 = 196656
(gdb)

[Matthew Flatt]

Since bit 5 is set, I think that means the processor was expecting
Thumb instructions, which at least explains the error.

To confirm that it's some bad jump or mismanagement of the mode by the
Racket JIT, does changing "racket/src/lightning/arm/asm.h" to disable
Thumb support allow the build to work?

[David Bremner]

Matthew Flatt <mfl...@cs.utah.edu> writes:

> At Fri, 17 Jan 2020 09:43:15 -0400, David Bremner wrote:

> Since bit 5 is set, I think that means the processor was expecting
> Thumb instructions, which at least explains the error.
>
> To confirm that it's some bad jump or mismanagement of the mode by the
> Racket JIT, does changing "racket/src/lightning/arm/asm.h" to disable
> Thumb support allow the build to work?

I'm not very sure I'm doing the right thing, but With the following
change, I get the same build failure.

% diff -u asm.h~ asm.h
--- asm.h~ 2019-12-30 19:12:36.000000000 +0000
+++ asm.h 2020-01-17 16:06:45.964573092 +0000
@@ -2299,4 +2299,7 @@
#define THUMB2_CLZ 0xfab0f080
#define T2_CLZ(rd,rm) thumb2_orrr(THUMB2_CLZ,rd,rm,rm)

+# undef JIT_ARM_THUMB
+# define JIT_ARM_THUMB 0
+
#endif /* __lightning_asm_h */

[Paulo Matos]

Hi David,

What's the status of this issue?

If it still persists, maybe it would be a good idea to open an issue in github.

Kind regards.

Paulo Matos

[David Bremner]

Paulo Matos <pma...@linki.tools> writes:

> Hi David,
>
> What's the status of this issue?
>
> If it still persists, maybe it would be a good idea to open an issue in
> github.
>

Nothing changed on my side since my message of January 17. I can file a
github issue later when I have the relevant credentials. I'm not really
sure how to summarize this thread, so any suggestions there would be
welcome.

d

[Paulo Matos]

I have gone through the thread and was wondering what are the
capabilities of the i.MX53. Would you be able to post the /proc/cpuinfo?

Also, generally an A8 is thumb2 capable but maybe not this specific
cpu. I am trying to create a cortex-a8 cross compiler without thumb2
support to try and compile racket to see if I can reproduce, since I
cannot repro on my of my Pis (but all of them have thumb2 support
anyway).

--
Paulo Matos

[Paulo Matos]

Can you try to define that earlier?
https://github.com/racket/racket/blob/5377d00c90480e73e0863203d35d7a1dc373195d/racket/src/racket/src/lightning/arm/asm.h#L150

Remove lines 150-152 and do the same:
#undef JIT_ARM_THUMB
#define JIT_ARM_THUMB 0

It looks like this header file assumes all armv7 have thumb2 support which is why its
generating that instruction. (line 132)

--
Paulo Matos

[David Bremner]

Paulo Matos <pma...@linki.tools> writes:

>
> I have gone through the thread and was wondering what are the
> capabilities of the i.MX53. Would you be able to post the /proc/cpuinfo?

model name : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 66.66
Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc08
CPU revision : 5

Hardware : Freescale i.MX53 (Device Tree Support)

>
> Also, generally an A8 is thumb2 capable but maybe not this specific
> cpu. I am trying to create a cortex-a8 cross compiler without thumb2
> support to try and compile racket to see if I can reproduce, since I
> cannot repro on my of my Pis (but all of them have thumb2 support
> anyway).

Not sure if A8 is ARMv8, but this is ARMv7

[Paulo Matos]

David Bremner writes:

> Paulo Matos <pma...@linki.tools> writes:
>
>>
>> I have gone through the thread and was wondering what are the
>> capabilities of the i.MX53. Would you be able to post the /proc/cpuinfo?
>
> model name : ARMv7 Processor rev 5 (v7l)
> BogoMIPS : 66.66
> Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
> CPU implementer : 0x41
> CPU architecture: 7
> CPU variant : 0x2
> CPU part : 0xc08
> CPU revision : 5
>
> Hardware : Freescale i.MX53 (Device Tree Support)

That's very similar to a 32bit rpi. I can understand why lightning
issues thumb2 instructions but can't understand why your system is not
happy with them.

>
>>
>> Also, generally an A8 is thumb2 capable but maybe not this specific
>> cpu. I am trying to create a cortex-a8 cross compiler without thumb2
>> support to try and compile racket to see if I can reproduce, since I
>> cannot repro on my of my Pis (but all of them have thumb2 support
>> anyway).
>
> Not sure if A8 is ARMv8, but this is ARMv7

Naming in ARM world sucks. ARMv7 can be indeed a Cortex-A8 and is indeed
in the case of a i.MX53.

This list helps:
https://en.wikipedia.org/wiki/ARM_architecture#Cores

Have you attempted to recompile by shifting the definition of
JIT_ARM_THUMB upwards in the file?

--
Paulo Matos

[David Bremner]

Paulo Matos <pma...@linki.tools> writes:

> Have you attempted to recompile by shifting the definition of
> JIT_ARM_THUMB upwards in the file?

Yes, I was waiting for the build to finish, but it did overnight and it
seems to have segfaulted in the same place.

make[6]: Entering directory '/home/bremner/racket-7.5+dfsg1/build/racket'
/usr/bin/make cstartup_`./racketcgc -cu /home/bremner/racket-7.5+dfsg1/src/racket/src/startup-select.rkt`
make[7]: Entering directory '/home/bremner/racket-7.5+dfsg1/build/racket'
./racketcgc -cu /home/bremner/racket-7.5+dfsg1/src/racket/src/compile-startup.rkt cstartup.inc cstartup.zo /home/bremner/racket-7.5+dfsg1/src/racket/src/startup.inc /home/bremner/racket-7.5+dfsg1/src/racket/src/schvers.h

[Paulo Matos]

You mean it hit an illegal instruction again?
We might have to dig deeper in this case. I remember I got once
temporary permissions into the Debian build servers. Any way to
reactivate this so I can take a look?

--
Paulo Matos