Compiling libuv for android using mac os x

[Mark Simpson]

Hi,

I've got a few problems with Android, hopefully someone can give me some pointers. I've not done a huge amount of cross-compiling for android, but have built a few things successfully (libcurl with openSSL & C-ARES). Any help/pointers appreciated, cheers.

tl;dr summary

I can't get libuv compiled using gyp, and I can compile libuv using autotools, but make check fails and my test program doesn't function correctly if I use this binary.

Long version

I've been doing a bit of prototyping using libuv & libcurl. Thus far, I have my test program successfully running on iOS and I'm satisfied with it. I'm now trying to get it running on Android. I've run into problems, though.

Development environment

Firstly, my development / compile machine's env:

Mac mini, x86_64, OS X Yosemite 10.10.2 

Android NDK r10d, with native platform version=9

Gyp:

if I try to go the documented route and use gyp, the make stage fails.

$ source ./android-configure /Users/mark.simpson/Development/android-ndk-r10d gyp
Copying prebuilt binaries...
Copying sysroot headers and libraries...
Copying c++ runtime headers and libraries...
Copying files to: /Users/mark.simpson/built_software/libuv/android-toolchain
Cleaning up...
Done.
['-Dtarget_arch=arm', '-DOS=android', '/Users/mark.simpson/built_software/libuv/uv.gyp', '-I', '/Users/mark.simpson/built_software/libuv/common.gypi', '--depth=.', '-f', 'make', '-Goutput_dir=/Users/mark.simpson/built_software/libuv/out', '--generator-output', '/Users/mark.simpson/built_software/libuv/out', '-Dhost_arch=x64', '-Duv_library=static_library', '-Dcomponent=static_library']

The only modification I've made to the android-configure script is adding --stl=gnustil, and this is just so that the toolchain mirrors what I've used to build my other android libs. I've already cross-compiled libcurl & its dependencies and that's working fine. 

$ echo $PATH
/Users/mark.simpson/built_software/libuv/android-toolchain/bin:rest_of_my_path

 

$ make -C out
CC(target) /Users/mark.simpson/built_software/libuv/out/Debug/obj.target/libuv/src/fs-poll.o
arm-linux-androideabi-gcc: error: x86_64: No such file or directory
arm-linux-androideabi-gcc: error: unrecognized command line option '-fasm-blocks'
arm-linux-androideabi-gcc: error: unrecognized command line option '-mpascal-strings'
arm-linux-androideabi-gcc: error: unrecognized command line option '-arch'
make: *** [/Users/mark.simpson/built_software/libuv/out/Debug/obj.target/libuv/src/fs-poll.o] Error 1

After a brief chat on IRC, tjfontaine said it looks like gyp is getting confused and mixing in compile flags that aren't meant for this type of cross-compile. He also suggested that gyp sometimes struggles with cross-compile scenarios, and it would probably be a good bet to go the autotools route. Given that I know absolutely nothing about gyp, I went down the autotools route next.

Autotools:

When I go the autotools route, make runs without error, but make check fails and my test program doesn't function correctly.

$ sh autogen.sh
+ glibtoolize
+ aclocal -I m4
+ autoconf
+ automake --add-missing --copy 

$ ./configure --build=x86_64-apple-darwin --host=arm-linux-androideabi --with-sysroot=$ANDROID_SYSROOT --prefix /Users/mark.simpson/built_software/libuv_compile/
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for arm-linux-androideabi-strip... arm-linux-androideabi-strip
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking build system type... x86_64-apple-darwin
checking host system type... arm-unknown-linux-androideabi
checking for arm-linux-androideabi-gcc... arm-linux-androideabi-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether arm-linux-androideabi-gcc accepts -g... yes
checking for arm-linux-androideabi-gcc option to accept ISO C89... none needed
checking whether arm-linux-androideabi-gcc understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of arm-linux-androideabi-gcc... gcc3
checking if arm-linux-androideabi-gcc supports -fvisibility=hidden flag... yes
checking if arm-linux-androideabi-gcc supports -g flag... yes
checking if arm-linux-androideabi-gcc supports -std=gnu89 flag... yes
checking if arm-linux-androideabi-gcc supports -pedantic flag... yes
checking if arm-linux-androideabi-gcc supports -Wall flag... yes
checking if arm-linux-androideabi-gcc supports -Wextra flag... yes
checking if arm-linux-androideabi-gcc supports -Wno-unused-parameter flag... yes
checking for arm-linux-androideabi-ar... arm-linux-androideabi-ar
checking the archiver (arm-linux-androideabi-ar) interface... ar
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by arm-linux-androideabi-gcc... /Users/mark.simpson/built_software/libuv/android-toolchain/arm-linux-androideabi/bin/ld
checking if the linker (/Users/mark.simpson/built_software/libuv/android-toolchain/arm-linux-androideabi/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /Users/mark.simpson/built_software/libuv/android-toolchain/bin/arm-linux-androideabi-nm -B
checking the name lister (/Users/mark.simpson/built_software/libuv/android-toolchain/bin/arm-linux-androideabi-nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 196608
checking how to convert x86_64-apple-darwin file names to arm-unknown-linux-androideabi format... func_convert_file_noop
checking how to convert x86_64-apple-darwin file names to toolchain format... func_convert_file_noop
checking for /Users/mark.simpson/built_software/libuv/android-toolchain/arm-linux-androideabi/bin/ld option to reload object files... -r
checking for arm-linux-androideabi-objdump... arm-linux-androideabi-objdump
checking how to recognize dependent libraries... pass_all
checking for arm-linux-androideabi-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for arm-linux-androideabi-ar... (cached) arm-linux-androideabi-ar
checking for archiver @FILE support... @
checking for arm-linux-androideabi-strip... (cached) arm-linux-androideabi-strip
checking for arm-linux-androideabi-ranlib... arm-linux-androideabi-ranlib
checking command to parse /Users/mark.simpson/built_software/libuv/android-toolchain/bin/arm-linux-androideabi-nm -B output from arm-linux-androideabi-gcc object... ok
checking for sysroot... /Users/mark.simpson/Development/android-ndk-r10d/platforms/android-9/arch-arm
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for arm-linux-androideabi-mt... no
checking for mt... no
checking if : is a manifest tool... no
checking how to run the C preprocessor... arm-linux-androideabi-gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if arm-linux-androideabi-gcc supports -fno-rtti -fno-exceptions... no
checking for arm-linux-androideabi-gcc option to produce PIC... -fPIC -DPIC
checking if arm-linux-androideabi-gcc PIC flag -fPIC -DPIC works... yes
checking if arm-linux-androideabi-gcc static flag -static works... yes
checking if arm-linux-androideabi-gcc supports -c -o file.o... yes
checking if arm-linux-androideabi-gcc supports -c -o file.o... (cached) yes
checking whether the arm-linux-androideabi-gcc linker (/Users/mark.simpson/built_software/libuv/android-toolchain/arm-linux-androideabi/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... Android linker
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking whether make supports nested variables... (cached) yes
checking for dlopen in -ldl... yes
checking for kstat_lookup in -lkstat... no
checking for kvm_open in -lkvm... no
checking for gethostbyname in -lnsl... no
checking for perfstat_cpu in -lperfstat... no
checking for pthread_mutex_init in -lpthread... no
checking for clock_gettime in -lrt... no
checking for sendfile in -lsendfile... no
checking for socket in -lsocket... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... unknown
checking for _LARGE_FILES value needed for large files... unknown
checking for pkg-config... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: executing depfiles commands
config.status: executing libtool commands

$ make
  CC       src/libuv_la-fs-poll.lo
  CC       src/libuv_la-inet.lo
  CC       src/libuv_la-threadpool.lo
  CC       src/libuv_la-uv-common.lo
  CC       src/libuv_la-version.lo
  CC       src/unix/libuv_la-async.lo
  CC       src/unix/libuv_la-core.lo
  CC       src/unix/libuv_la-dl.lo
  CC       src/unix/libuv_la-fs.lo
  CC       src/unix/libuv_la-getaddrinfo.lo
  CC       src/unix/libuv_la-getnameinfo.lo
  CC       src/unix/libuv_la-loop-watcher.lo
  CC       src/unix/libuv_la-loop.lo
  CC       src/unix/libuv_la-pipe.lo
src/unix/pipe.c: In function 'uv__pipe_getsockpeername':
src/unix/pipe.c:237:15: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (addrlen > *size) {
               ^
  CC       src/unix/libuv_la-poll.lo
  CC       src/unix/libuv_la-process.lo
  CC       src/unix/libuv_la-signal.lo
  CC       src/unix/libuv_la-stream.lo
  CC       src/unix/libuv_la-tcp.lo
  CC       src/unix/libuv_la-thread.lo
  CC       src/unix/libuv_la-timer.lo
  CC       src/unix/libuv_la-tty.lo
  CC       src/unix/libuv_la-udp.lo
  CC       src/unix/libuv_la-android-ifaddrs.lo
src/unix/android-ifaddrs.c: In function 'getNetlinkResponse':
src/unix/android-ifaddrs.c:139:9: warning: implicit declaration of function 'uv__free' [-Wimplicit-function-declaration]
         uv__free(l_buffer);
         ^
src/unix/android-ifaddrs.c:140:9: warning: implicit declaration of function 'uv__malloc' [-Wimplicit-function-declaration]
         l_buffer = uv__malloc(l_size);
         ^
src/unix/android-ifaddrs.c:140:18: warning: assignment makes pointer from integer without a cast [enabled by default]
         l_buffer = uv__malloc(l_size);
                  ^
src/unix/android-ifaddrs.c: In function 'newListItem':
src/unix/android-ifaddrs.c:185:27: warning: initialization makes pointer from integer without a cast [enabled by default]
     NetlinkList *l_item = uv__malloc(sizeof(NetlinkList));
                           ^
src/unix/android-ifaddrs.c: In function 'interpretLink':
src/unix/android-ifaddrs.c:351:13: warning: assignment makes pointer from integer without a cast [enabled by default]
     l_entry = uv__malloc(sizeof(struct ifaddrs) + sizeof(int) + l_nameSize + l_addrSize + l_dataSize);
             ^
src/unix/android-ifaddrs.c: In function 'interpretAddr':
src/unix/android-ifaddrs.c:480:13: warning: assignment makes pointer from integer without a cast [enabled by default]
     l_entry = uv__malloc(sizeof(struct ifaddrs) + l_nameSize + l_addrSize);
             ^
  CC       src/unix/libuv_la-pthread-fixes.lo
  CC       src/unix/libuv_la-linux-core.lo
  CC       src/unix/libuv_la-linux-inotify.lo
  CC       src/unix/libuv_la-linux-syscalls.lo
  CC       src/unix/libuv_la-proctitle.lo
  CCLD     libuv.la

$ make check
lots of warnings like: 

test/task.h:80:13: warning: anonymous variadic macros were introduced in C99 [-Wvariadic-macros] (should I be specifying C99 somewhere?)

then...

test/test-signal-multiple-loops.c:248: error: undefined reference to 'uv__pthread_sigmask'

collect2: error: ld returned 1 exit status

make[1]: *** [test/run-tests] Error 1

make: *** [check-am] Error 2

So basically, I'm a bit stumped now. Hopefully I'm just making some sort of newbie CC error. Any help appreciated.

[Saúl Ibarra Corretgé]

Hi,

On 03/23/2015 01:56 PM, Mark Simpson wrote:
> Hi,
>
> I've got a few problems with Android, hopefully someone can give me some
> pointers. I've not done a huge amount of cross-compiling for android,
> but have built a few things successfully (libcurl with openSSL &
> C-ARES). Any help/pointers appreciated, cheers.
>

> *tl;dr summary*

> I can't get libuv compiled using gyp, and I can compile libuv using
> autotools, but make check fails and my test program doesn't function
> correctly if I use this binary.
>

> *Long version*

> I've been doing a bit of prototyping using libuv & libcurl. Thus far, I
> have my test program successfully running on iOS and I'm satisfied with
> it. I'm now trying to get it running on Android. I've run into problems,
> though.
>

> *Development environment*

> Firstly, my development / compile machine's env:
> Mac mini, x86_64, OS X Yosemite 10.10.2
> Android NDK r10d, with native platform version=9
>

> *Gyp:*

> if I try to go the documented route and use gyp, the make stage fails.
>

> *$ source ./android-configure
> /Users/mark.simpson/Development/android-ndk-r10d gyp*

> Copying prebuilt binaries...
> Copying sysroot headers and libraries...
> Copying c++ runtime headers and libraries...
> Copying files to:
> /Users/mark.simpson/built_software/libuv/android-toolchain
> Cleaning up...
> Done.
> ['-Dtarget_arch=arm', '-DOS=android',
> '/Users/mark.simpson/built_software/libuv/uv.gyp', '-I',
> '/Users/mark.simpson/built_software/libuv/common.gypi', '--depth=.',
> '-f', 'make',
> '-Goutput_dir=/Users/mark.simpson/built_software/libuv/out',
> '--generator-output',
> '/Users/mark.simpson/built_software/libuv/out', '-Dhost_arch=x64',
> '-Duv_library=static_library', '-Dcomponent=static_library']
>
>
> The only modification I've made to the android-configure script is
> adding --stl=gnustil, and this is just so that the toolchain mirrors
> what I've used to build my other android libs. I've already
> cross-compiled libcurl & its dependencies and that's working fine.
>

> *$ echo $PATH
> */Users/mark.simpson/built_software/libuv/android-toolchain/bin:*rest_of_my_path*
>
>
>
> *$ make -C out
> * CC(target)

> /Users/mark.simpson/built_software/libuv/out/Debug/obj.target/libuv/src/fs-poll.o
> arm-linux-androideabi-gcc: error: x86_64: No such file or directory
> arm-linux-androideabi-gcc: error: unrecognized command line option
> '-fasm-blocks'
> arm-linux-androideabi-gcc: error: unrecognized command line option
> '-mpascal-strings'
> arm-linux-androideabi-gcc: error: unrecognized command line option
> '-arch'
> make: ***
> [/Users/mark.simpson/built_software/libuv/out/Debug/obj.target/libuv/src/fs-poll.o]
> Error 1
>
>
> After a brief chat on IRC, tjfontaine said it looks like gyp is getting
> confused and mixing in compile flags that aren't meant for this type of
> cross-compile. He also suggested that gyp sometimes struggles with
> cross-compile scenarios, and it would probably be a good bet to go the
> autotools route. Given that I know absolutely nothing about gyp, I went
> down the autotools route next.
>

There was a hack somewhere in joyent/libuv to make it work on OSX, but
IIRC this was a problem with GYP on OSX, it does work on Linux.

> *Autotools:*

> When I go the autotools route, make runs without error, but make check
> fails and my test program doesn't function correctly.
>

> *$ sh autogen.sh
> *+ glibtoolize

> + aclocal -I m4
> + autoconf
> + automake --add-missing --copy
>

> *$ ./configure --build=x86_64-apple-darwin
> --host=arm-linux-androideabi --with-sysroot=$ANDROID_SYSROOT
> --prefix /Users/mark.simpson/built_software/libuv_compile/*

> *$ make*

I submitted https://github.com/libuv/libuv/pull/285 to fix these warnings.

> *$ make check*

> lots of warnings like:
> test/task.h:80:13: warning: anonymous variadic macros were introduced in
> C99 [-Wvariadic-macros] (should I be specifying C99 somewhere?)
>
> then...
>

> *test/test-signal-multiple-loops.c:248: error: undefined reference to
> 'uv__pthread_sigmask'*

> collect2: error: ld returned 1 exit status
> make[1]: *** [test/run-tests] Error 1
> make: *** [check-am] Error 2
>

A quick search shows that pthread_sigmask might not be supported
proberly on Andoird. We might need to skip the test there.

> So basically, I'm a bit stumped now. Hopefully I'm just making some sort
> of newbie CC error. Any help appreciated.
>

Well, the fact that tests don't run doesn't mean that libuv doesn't
work. Have you tried using the resulting library from the autoconf
build? Any problems there?


Cheers,

--
Saúl Ibarra Corretgé
bettercallsaghul.com

[Mark Simpson]

Hi, 

Firstly, thanks for the reply & for fixing those warnings. 

In response to your question:

I have tried to use the lib (I mentioned this in the original post, but should've stated it more prominently) and not much is working. I've tried three things so far:

1. Running an idle loop that spams to the console

Result: I can run the idle loop & print stuff just fine.

 

2. The libuv socket example (http://nikhilm.github.io/uvbook/networking.html). I'm just creating a socket server then connecting to it via my mac's terminal & netcat (this is something we do for our development builds with a non-libuv socket server, so I'm fairly sure the connection/network configuration is correct when using nc).

Result: I can't connect. uv_listen returns -13 (Listen error permission denied). 

 

3. A libcurl / libuv test program (basically this: https://raw.githubusercontent.com/bagder/curl/master/docs/examples/multi-uv.c but using callbacks for the write -- I have the same example working on iOS) 

Result: Some of the plumbing appears to trigger. The curl socket callbacks don't trigger, but my requests come through as completing, but with no valid information (e.g. after I receive a CURLMSG_DONE message, curl_easy_getinfo(easy_handle, ....) doesn't tell me anything, as the out pointers are left unchanged). I'll dig into debugging this a bit more tomorrow and let you know what is/isn't triggered.

Tomorrow, I will try to cut down my example programs to a bare minimum then attach them to this thread. Also, do you have any suggestions for simple scenarios that would aid in diagnosing the problem? If there's anything I can do to provide better information, I will get onto that tomorrow.

Thanks for the help -- I appreciate it.

-Mark

[Saúl Ibarra Corretgé]

Hi Mark!

On 03/23/2015 07:56 PM, Mark Simpson wrote:
> Hi,
>

> Firstly, thanks for the reply & for fixing those warnings.
>
> In response to your question:
> I have tried to use the lib (I mentioned this in the original post, but
> should've stated it more prominently) and not much is working. I've
> tried three things so far:
>

Oh, looks like I missed that, sorry!

> 1. Running an idle loop that spams to the console
>
> Result: I can run the idle loop & print stuff just fine.
>

Ok, basics work then.

>
> 2. The libuv socket example
> (http://nikhilm.github.io/uvbook/networking.html). I'm just creating a
> socket server then connecting to it via my mac's terminal & netcat (this
> is something we do for our development builds with a non-libuv socket
> server, so I'm fairly sure the connection/network configuration is
> correct when using nc).
>

> Result: I can't connect. uv_listen returns -13 (*Listen error
> permission denied*).
>

Are you using a port lower than 1024? Errno 13 is EACCESS.

>
> 3. A libcurl / libuv test program (basically
> this: https://raw.githubusercontent.com/bagder/curl/master/docs/examples/multi-uv.c but
> using callbacks for the write -- I have the same example working on iOS)
>
> Result: Some of the plumbing appears to trigger. The curl socket
> callbacks don't trigger, but my requests come through as completing,
> but with no valid information (e.g. after I receive

> a CURLMSG_DONE message, curl_easy_getinfo(easy_handle, *....*)

> doesn't tell me anything, as the out pointers are left unchanged).
> I'll dig into debugging this a bit more tomorrow and let you know
> what is/isn't triggered.
>
>
> Tomorrow, I will try to cut down my example programs to a bare minimum
> then attach them to this thread. Also, do you have any suggestions for
> simple scenarios that would aid in diagnosing the problem? If there's
> anything I can do to provide better information, I will get onto that
> tomorrow.
>

I'd first try to get a basic TCP server working on Android. AFAIK, libuv
does work on it so it might just be a tiny thing we need to tweak.

[Mark Simpson]

Hi again Saúl,

In response to your question: I'm using port 12345.

To ensure that I'm not doing anything obviously wrong with my sample code, I've created an .h/.cpp containing a tiny amount of socket functionality. I call uv_tcp_connect from my main function on iOS/Android. I can compile & run it on iOS and connect using netcat. Running the same code on android results in uv_listen returning -13 ("Listen error permission denied").

My colleague and I are going to spin up a linux box and compile using gyp on linux. If this doesn't work, I'll get back to you with more details & a mini project that exhibits the errors I'm seeing. 

Thanks

[Mark Simpson]

Here's a tiny Android project that exhibits the problem (see attachment). It's the android native-plasma sample with everything ripped out and a handful of libuv calls grabbed from the examples link I posted earlier. This code works fine on iOS (I can trigger the connect callback). On android, I just get the -13 return code from uv_tcp_connect.

Probably worth double-checking the Android.mk, but we use similar settings & linker options for our other projects without an issue. Also, I have noticed rare SIGILLs in the debugger, but wasn't able to see anything incriminating on the stack / in the locals pane. I don't use the Android debugger much and don't tend to trust it a great deal, so not sure how significant this is. I have checked the libuv.a using readelf to ensure the format is correct and it looks fine, too.

$ readelf -h ./libuv.a | grep -i 'class\|machine' | head -2
  Class:                             ELF32
  Machine:                           ARM

My colleague and I are going to do the cross compile using linux & gyp this afternoon, will let you know how we get on.

Thanks,

-Mark

[Mark Simpson]

Sorry, I mean to say that my example returns -13 from uv_listen (line 32 of uv_tcp_server.cpp), not uv_tcp_connect.

   31 tty("uv tcp listen\n");

   32 int r = uv_listen((uv_stream_t*) &server, 10, on_new_connection); // r = -13

[Mark Simpson]

I've now cross-compiled using a linux machine & gyp. I dropped in the binary and have exactly the same outcome as my autotools / os x generated binary :-(

My steps were basically:

- Spin up an aws linux box

- Pull down the NDK & Android SDK

- Add them to the path

- Update the Android SDK

- Pull down the libuv repo & gyp

- source ./android-configure /home/ec2-user/downloads/android-ndk-r10d gyp

  make -C out

I've attached the linux & gyp compiled binary in case it's of any use. 

Anyway, either I've made mistakes using both os x / linux compile methods, or there's a problem in here somewhere. Not quite sure where to go from here. Any ideas? If anyone has the hardware to hand, it would be great if I could get some independent verification that there's a problem (or I'm doing something wrong).

Cheers,

-Mark

[Mark Simpson]

Mea culpa. Got the socket example working.

When setting up my project, I hadn't set <uses-permission android:name="android.permission.INTERNET"/> in the AndroidManifest.xml (the dangers of making the most stripped-down example possible ... stripped down too far!)

Sorry for the wild goose chase on that. I'm going to get back to my original problem and see if the curl/libuv combo works with this permission set. I will report back on that later.

Regards,

-Mark