Building Julia on Lego Mindstorms EV3

[Robin Deits]

Hello julia-dev!

For the past week or so I've been working on a (possibly crazy) idea to get Julia running on the Lego Mindstorms EV3 brick (Lego and Julia being two of my favorite technologies), so that we can expand Julia to educational robotics. The ev3 is a small, ARM-powered Linux computer, and it seems like it just *might* be powerful enough to run Julia. However, so far, I've been unable to get a working REPL. The ev3 is an armv5 processor, so as far as I know, this is uncharted territory. Specifically, it has:

cpu: arm926ej-s

march: armv5te

float-abi: soft

fpu: (none)

RAM: 64MB

I've tried a *lot* of different approaches, all of which have failed at various points. All of my builds have been inside brickstrap: https://github.com/ev3dev/brickstrap a tool for building EV3 images and cross-compiling using qemu. The builds are based on ev3dev, which is itself based on Debian jessie. I've gotten several complete builds to work, but none have actually resulted in a working REPL. Specifically, I've tried:

1) Letting Julia build all of its own dependencies except OpenBLAS (building OpenBLAS through Julia fails, because it assumes a hard-float system). To get OpenBLAS working, I downloaded the latest source release, and configured it with the following options:

TARGET = ARMV5

DYNAMIC_ARCH = 0

USE_THREAD = 0

NO_LAPACK = 1

NO_LAPACKE = 1

MAX_STACK_ALLOC = 64

ifeq ($(CORE), ARMV5)

CCOMMON_OPT += -marm -mfloat-abi=soft -march=armv5te

FCOMMON_OPT += -marm -mfloat-abi=soft -march=armv5te

endif

This resulted in a functional OpenBLAS (or, at least, one for which the ctests passed), on ev3. That was very exciting, since various reports had indicated that OpenBLAS could not possibly be run on a soft-float machine. 

With that OpenBLAS, I edited Julia's Make.user and set:

override MARCH=armv5te

override JULIA_CPU_TARGET=arm926ej-s

override prefix=/usr/local/

override USE_SYSTEM_BLAS=1

override LIBBLAS := -lopenblas

override LIBBLASNAME := libopenblas

Eventually, the Julia build completed successfully. But running it on the EV3 segfaults. GDB reports:

(gdb) r

Starting program: /opt/julia-0.4.1/julia

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/arm-linux-gnueabi/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.

0x00000000 in ?? ()

(gdb) bt

#0  0x00000000 in ?? ()

#1  0xb6e84310 in extensions () from /opt/julia-0.4.1/usr/bin/../lib/libjulia.so

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb)

2) I tried a completely separate build with mostly pre-packaged dependencies, but with a few flags set in Julia:

override USE_SYSTEM_BLAS=1
override USE_SYSTEM_LAPACK=1
override USE_SYSTEM_LIBM=1
override USE_SYSTEM_FFTW=1
override USE_SYSTEM_GMP=1
override USE_SYSTEM_MPFR=1

override USE_SYSTEM_ARPACK=1

prefix=/usr/local/

override MARCH=armv5te

override JULIA_CPU_TARGET=arm926ej-s

LLVM_FLAGS+="--with-float=soft --with-abi=aapcs-linux"

That resulted in a totally trouble-free build, which is great! But launching the Julia REPL gave another segfault. Again, gdb says:

(gdb) r

Starting program: /opt/julia-0.4.1/julia

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/arm-linux-gnueabi/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.

0x00000000 in ?? ()

(gdb) bt

#0  0x00000000 in ?? ()

#1  0xb6e844e0 in extensions () from /opt/julia-0.4.1/usr/bin/../lib/libjulia.so

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

3) I tried building Julia master in case that helped, using the separate OpenBLAS install, with: 

override MARCH=armv5te

override JULIA_CPU_TARGET=arm926ej-s

override prefix=/usr/local/

override USE_SYSTEM_BLAS=1

LLVM_FLAGS+="--with-float=soft --with-abi=aapcs-linux"

override LIBBLAS := -lopenblas

override LIBBLASNAME := libopenblas

but this build failed with:

    CC src/runtime_ccall.o

In file included from /opt/julia-master/usr/include/llvm/ADT/SmallVector.h:20:0,

                 from /opt/julia-master/usr/include/llvm/Support/Allocator.h:24,

                 from /opt/julia-master/usr/include/llvm/ADT/StringMap.h:18,

                 from /opt/julia-master/usr/include/llvm/Support/Host.h:17,

                 from /opt/julia-master/src/runtime_ccall.cpp:6:

/opt/julia-master/usr/include/llvm/Support/MathExtras.h:74:7: warning: "_MSC_VER" is not defined [-Wundef]

 #elif _MSC_VER

       ^

/opt/julia-master/usr/include/llvm/Support/MathExtras.h:90:7: warning: "_MSC_VER" is not defined [-Wundef]

 #elif _MSC_VER

       ^

/opt/julia-master/usr/include/llvm/Support/MathExtras.h:143:7: warning: "_MSC_VER" is not defined [-Wundef]

 #elif _MSC_VER

       ^

/opt/julia-master/usr/include/llvm/Support/MathExtras.h:159:7: warning: "_MSC_VER" is not defined [-Wundef]

 #elif _MSC_VER

       ^

    CC src/threadgroup.o

{standard input}: Assembler messages:

{standard input}:690: Error: selected processor does not support ARM mode `dmb'

{standard input}:740: Error: selected processor does not support ARM mode `wfe'

{standard input}:819: Error: selected processor does not support ARM mode `dmb'

{standard input}:918: Error: selected processor does not support ARM mode `wfe'

{standard input}:1047: Error: selected processor does not support ARM mode `wfe'

Makefile:94: recipe for target 'threadgroup.o' failed

make[1]: *** [threadgroup.o] Error 1

Makefile:87: recipe for target 'julia-src-release' failed

make: *** [julia-src-release] Error 2

(brickstrap)root@paladin-21:/opt/julia-master#

I've also tried about a dozen other combinations, but none were quite as promising as these, so I'll spare you the details. 

Is there any hope? Do any of these results suggest that this might be possible? I've got a fair amount of experience debugging and developing, but this is getting beyond my ability to proceed. I'd definitely appreciate any advice, help, or gentle suggestions that what I'm trying to do is impossible ;-)

Thanks,

-Robin

[maleadt]

The build error on master should be fixed now that #14154 has been merged.
Can't really help on the debugging though. Did you try a debug build (i.e. make debug)? I assume it might give you some line number information for the libjulia.so frame.

Op vrijdag 27 november 2015 06:42:44 UTC+1 schreef Robin Deits:

[Viral Shah]

Definitely not try using OpenBLAS - I believe it requires armv7. Using the approach you outline with using maximum external dependencies is the right one. As maleadt said, doing `make debug` seems like the best approach to figure out where things are failing.

Another thing to do would be to look in julia/src for arm specific ifdefs and seeing if things are being done with assumptions around a armv6 or v7.

-viral

[Viral Shah]

Could you try using these precompiled binaries? I'm not too optimistic, but this is the can't hurt category. They are for 0.4.0-rc3, and use openblas. You may have to set the CPU type for LLVM.

https://drive.google.com/open?id=0B0rXlkvSbIfhT1lTY0VVRTdvaVE

-viral

On Friday, November 27, 2015 at 11:12:44 AM UTC+5:30, Robin Deits wrote:

[Jameson Nash]

Hey Robin!

The soft-float ABI seems uncommon now for sure. We are overriding the processor detection for VFP and forcing it into hard-float mode for JIT (https://github.com/JuliaLang/julia/blob/e6459314b5c4cf2664097dbb5106cce543fb9c9a/src/codegen.cpp#L6240-L6244). We can probably remove that now, but the hard failure of specifying it and getting a quick SIGILL seemed better than failing to detect it and getting soft failures of garbage answers from the floating point routines.

The "Assembler messages" build errors on master should be fixed now (https://github.com/JuliaLang/julia/issues/14152).

-jameson

[Robin Deits]

@maleadt:

Thanks for pointing out that PR: it did indeed fix the compilation issues I was having. Using the pre-packaged dependencies, I was able to cleanly build julia-debug from master in brickstrap. Running julia-debug through gdb eventually failed with:

Program received signal SIGSEGV, Segmentation fault.

0x00000000 in ?? ()

(gdb) bt

#0  0x00000000 in ?? ()

#1  0xb5bb4654 in do_trycatch () at flisp.c:956

#2  0xb5bb4654 in do_trycatch () at flisp.c:956

#3  0xb5bbc23c in apply_cl (nargs=2) at flisp.c:1856

#4  0xb5bae204 in _applyn (n=0) at flisp.c:729

#5  0xb5bae60c in fl_applyn (n=0, f=3001047922) at flisp.c:774

#6  0xb5a57a54 in jl_parse_next () at /opt/julia/src/ast.c:576

#7  0xb5a90f44 in jl_parse_eval_all (fname=0x947a85d0 "/opt/julia/usr/etc/julia/juliarc.jl", len=35)

    at /opt/julia/src/toplevel.c:570

#8  0xb5a912bc in jl_load (fname=0x947a85d0 "/opt/julia/usr/etc/julia/juliarc.jl", len=35)

    at /opt/julia/src/toplevel.c:621

#9  0xb5a9136c in jl_load_ (str=0x93de0e00) at /opt/julia/src/toplevel.c:629

@Viral:

Thanks also! I'll stick to the system blas as you recommended. I tried the binaries you linked, but they don't seem to be compatible with my architecture (running ./bin/julia from the extracted folder just gives "-bash: ./bin/julia: No such file or directory")

@ Jameson

Hey! Long time no see :). Pulling in master fixed my build issues, as you suggested, thanks! And I've edited codegen.cpp to try to build with vfp off. I'm running a build now, and will let you know how it turns out when it finishes. 

Thanks everybody! I'll keep hacking and let you know if I make any progress. 

-Robin

[Robin Deits]

One additional piece of info, in case it's relevant:

The system install of git actually segfaults while unpacking some repositories, unless I set:

git config --global pack.threads 1

which suggests that there are some issues with threads on the EV3. Does setting JULIA_THREADS=0 seem like a good idea? 

-Robin

[Tony Kelman]

JULIA_THREADS is experimental and still off by default even on x86_64, getting it to work on arm (esp armv5) should wait until after a default build is at least getting to a repl.

[Robin Deits]

Gotcha. I was actually suggesting turning it *off* for armv5, but hadn't yet realized that it's already off by default.