Making NDK builded libs available in unit-tests for faster, device-independent testing

[Ivan K]

Already a long time i'm trying to compile native lib that way, to make it possible to use it inside unit-test project in linux env. But currently still can't. It's failes on load every time. But it's really interesting for me why and I want to deeply undesrtand the whole process. So I have several questions:

1. Is it actually possible to compile native lib with android ndk that way, to make it possible to load it from unit-test env in linux (of course in perfect case all major platforms: linux/macos/windows).

• if it's possible - what's steps is required, at least in general
• if it's not possible - than why (detailed answer very appriciated)

       2. In every example and also in the boilerplate project which generated via Android Studio actual JNI connector component (native-lib) buil as shared library. And this dummy stub compile and works well. However if inside cmake change this lib settings from SHARED to STATIC all        things brokes. Looks like it's not possible to build this lib as static. Why?         

[mike digioia]

Where is it located

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk+unsubscribe@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/21f8be1e-ac49-409c-b47a-825cebff5eec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ivan K]

@mpd What is "it" in your question? Sorry, that's not clear for me.

[AppCoder]

With the assumption that you are happy with your JNI connector and just want to test the code it's hooking up (unit test) I'd look at using CMake to just make libraries of the stuff you want to test.  I'm using the following to do boost and aws c++ sdk and my library components to build static libraries for android and my local OSX and build server linux boxes.  ANDROID_NDK_HOME is where your ndk lives (if you use AS, it should be ~/Library/Android/sdk/ndk-bundle/ )   I point my Android Studio build to pick up the shared libraries from the below build.   (If I were cool I'd have gradle to kick this off, but not there yet.)

#!/bin/sh
#
# This is a shell build script to build aws-sdk-cpp
#    and any aditional components using cmake
#    on for the local host and cross compiled for Android
#
#
# NDK locaiton
export ANDROID_NDK=$ANDROID_NDK_HOME
#
# Ordered list of component directories to build
if [ $# -gt 0 ]
then
export COMPONENTS=$@
else
export COMPONENTS="hello-world some-other-lib"
fi
#
# Here (for finding stuff)
export PROJECTHOME=`pwd`
#
# All the ABIs
# Have to build armeabi/Android first so the
# boost jam file can find the standalone tools
export ABI_LIST="armeabi `uname`"
#
# Make do the various compiles
for BUILD_ABI in $ABI_LIST
do
export BUILD_ABI
#
echo "Doing $BUILD_ABI"
# Special extras for Android stuff
if [ $BUILD_ABI = `uname` ]
then
export EXTRA_AWS_OPTIONS=""
export EXTRA_CMAKE_OPTIONS=""
else
# @TODO find way to make AWS to other ABIs
export EXTRA_AWS_OPTIONS=" -DTARGET_ARCH=ANDROID -DANDROID_ABI=$BUILD_ABI "
export EXTRA_CMAKE_OPTIONS="
    -DCMAKE_SYSTEM_NAME=Android \
    -DCMAKE_SYSTEM_VERSION=21 \
    -DCMAKE_ANDROID_NDK=$ANDROID_NDK \
    -DCMAKE_ANDROID_NDK_TOOLCHAIN_VERSION=clang \
    -DCMAKE_ANDROID_ARCH_ABI=$BUILD_ABI \
    -DCMAKE_ANDROID_STL_TYPE=c++_shared "
fi
#
# set up install dir; @TODO evaluate value of install/$BUILD_ABI/$comp
export INSTALL_DIR_BASE=$PROJECTHOME/install/$BUILD_ABI
mkdir -p $INSTALL_DIR_BASE
echo "We have $INSTALL_DIR_BASE as INSTALL_DIR_BASE"
#
# set up build dir
export CMAKE_PREFIX_PATH=$PROJECTHOME/build/$BUILD_ABI
mkdir -p $CMAKE_PREFIX_PATH
cd $CMAKE_PREFIX_PATH
#
# do AWS specific build block
mkdir -p aws-sdk-cpp
cd aws-sdk-cpp
export INSTALL_DIR=$INSTALL_DIR_BASE/aws-sdk-cpp
mkdir -p $INSTALL_DIR
if [ $# -lt 1 ]
then
export AWS_CMAKE_COMMAND=" \
cmake \
    -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
    -DCMAKE_RULE_MESSAGES:BOOL=OFF \
    -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
    $EXTRA_AWS_OPTIONS \
    -DBUILD_ONLY='logs;core;iam;access-management;cognito-identity;s3;kinesis' \
    $PROJECTHOME/aws-sdk-cpp "
$AWS_CMAKE_COMMAND || $AWS_CMAKE_COMMAND
# do it twice in case stand alone tool chain needs to be created for android...
make
make install
fi
cd ..

# dangerous assumption about innards of aws-sdk-cpp internal interactions with
# setting up android standalone NDK toolchain.  Will need to add more of these if
# we support more than one $BUILD_ABI
#
# Note the echo is required to force the file glob
#
export NDK_STANDALONE=`echo $PROJECTHOME/aws-sdk-cpp/toolchains/android/armeabi*`

if [ $# -lt 1 ]
then
# would really like boost to make with CMake...
cd $PROJECTHOME/boost
./build-boost.sh
cd $CMAKE_PREFIX_PATH
fi

export BOOST_ROOT=$INSTALL_DIR_BASE/boost

for comp in $COMPONENTS
do
export INSTALL_DIR=$INSTALL_DIR_BASE/$comp
mkdir -p $INSTALL_DIR
mkdir -p $comp
cd $comp
#
# @TODO should we read dependency lists from somewhere
#     or research letting cmake find stuff or provide
#    our INSTALL_DIR_BASE to cmake for find_package
#    and do it in $comp/CMakeLists.txt
#    e.g. -Daws-sdk-cpp_DIR won't be needed for luaJIT
cmake  \
        -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
        -DCMAKE_PREFIX_PATH=$INSTALL_DIR_BASE \
    $EXTRA_CMAKE_OPTIONS \
        -DBOOST_ROOT=$BOOST_ROOT \
    $PROJECTHOME/$comp
make VERBOSE=1
make install
cd ..
done
cd $PROJECTHOME
done

my hello-wolrd CMakeList.txt is:

# minimal CMakeLists.txt for Android NDK support
cmake_minimum_required(VERSION 3.0)

set(CMAKE_VERBOSE_MAKEFILE ON)

project(hello-world)

# Locate the AWS SDK for C++ package.
#   CMAKE_PREFIX_PATH=/path/to/installed/stuff
find_package(aws-sdk-cpp)

find_package(avro)

# use boost that we installed, not the system
set(Boost_USE_STATIC_LIBS ON)
set(Boost_INCLUDE_DIR ${CMAKE_INSTALL_PREFIX}/../boost/include)
set(Boost_LIBRARY_DIR ${CMAKE_INSTALL_PREFIX}/../boost/lib)

find_package(Boost 1.62 REQUIRED COMPONENTS chrono system)
include_directories(${Boost_INCLUDE_DIR})

# Force C++11
# Note that 3.7.1 CMake gives us rtti and exceptions by default
#     and the ndk r13 cmake files do not
add_definitions(-std=c++11)

file(GLOB HELLO_WORLD_SOURCE "${CMAKE_CURRENT_SOURCE_DIR}/*.cpp")
file(GLOB HELLO_WORLD_HEADERS "${CMAKE_CURRENT_SOURCE_DIR}/*.h")

# Make library and something we can run on android
add_library(${PROJECT_NAME} ${HELLO_WORLD_SOURCE})
add_executable(${PROJECT_NAME}_bin ${HELLO_WORLD_SOURCE})

target_link_libraries(hello-world aws-cpp-sdk-core ${Boost_LIBRARIES} avro::avrocpp )
target_link_libraries(hello-world_bin aws-cpp-sdk-core ${Boost_LIBRARIES} avro::avrocpp )

# This is weak:
# We should do the the config stuff to let find_package work
# as in https://github.com/forexample/package-example/blob/master/Foo/CMakeLists.txt
install (TARGETS ${PROJECT_NAME} ${PROJECT_NAME}_bin
     EXPORT ${PROJECT_NAME}
         ARCHIVE DESTINATION lib
         LIBRARY DESTINATION lib
         RUNTIME DESTINATION bin)

install (FILES ${HELLO_WORLD_HEADERS} DESTINATION include)

install (FILES ${HELLO_WORLD_HEADERS} DESTINATION lib)

target_include_directories(hello-world PUBLIC include)

export(PACKAGE hello-world)



With this I get build/install directories for each architecture and each project, libraries and binaries I can use on each architecture.
The hello-world binary runs on the local machine and I can push my unit tests to device with

#!/bin/sh
adb push install/armeabi/aws-sdk-cpp/lib/libaws-cpp-sdk-core.so /data/local/tmp/
#  WARNING, hard coded hack to find ASW toolchain
adb push aws-sdk-cpp/toolchains/android/armeabi-standalone-clang-android-21-libc++_shared-/arm-linux-androideabi/lib/libc++_shared.so /data/local/tmp/
adb push install/armeabi/hello-world/bin/hello-world_bin /data/local/tmp/
adb shell env LD_LIBRARY_PATH=/data/local/tmp /data/local/tmp/hello-world_bin

I don't know if this works on windows 10 with it's shell implementation.

And the haxy build-boost.sh is:

#!/bin/sh
export BOOST_BUILD_DIR=`pwd`/../build/$BUILD_ABI/boost
export BOOST_INSTALL_DIR=`pwd`/../install/$BUILD_ABI/boost
export HOST_ABI=`uname`

if [ "$BUILD_ABI" = "$HOST_ABI" ];
then
export EXTRA_B2_OPTIONS=""
else
# Note that gcc-android here is a lie, the user-config.jam replaces the compiler with clang
# We have to lie because boost's clang-darwin.jam hard codes ranlib (so we don't get the cross
# compile version)
export EXTRA_B2_OPTIONS="target-os=android toolset=gcc-android include=$NDK_STANDALONE/include/c++/4.9.x "
fi

if [ ! -e boost_1_62_0 ];
then
tar zxf boost_1_62_0.tar.gz
cd boost_1_62_0
cp -f ../clang-android-from-NDK_STANDALONE.jam tools/build/src/user-config.jam
./bootstrap.sh
else
cd boost_1_62_0
fi

./b2 \
   -d+2 \
   -j 4 \
   --reconfigure \
   $EXTRA_B2_OPTIONS \
   link=static \
   variant=release \
   threading=multi \
   --with-system \
   --with-thread \
   --with-chrono \
   --with-timer \
   --with-filesystem \
   --with-system \
   --with-program_options \
   --with-iostreams \
   --prefix=${BOOST_INSTALL_DIR} \
   --build-dir=${BOOST_BUILD_DIR} \
   install


It feels like a horrid hack of competing build systems (gradle, bjam, CMake, aws CMake to do Android) but it seems to be the best I've found for building/testing modern portable C++ code (I could almost be talked out of Boost if I had C++14, but Boost::Asio is so nice.)

[Alex Cohn]

I have no experience in running unit tests of Android native libs, but it is possible (http://stackoverflow.com/a/18137842/192373 uses ndk-build, not CMake). Googletest is distributed with NDK these days (see http://stackoverflow.com/a/22075341/192373). Whether your dev PC runs macOS, Linux, or Windows, you must run the tests on a device or emulator, so forget 'device-independent testing'. Note that behavior the same C code may behave differently on ARM devices and x86 devices; real testing must involve all target platforms: the C runtime on Marshmellow is different from Gingerbread; 64-bit build can expose bugs that are hidden in 32-bit version.

You can use the same source code and compile it with native Android/Linux/macOS/iOS/Windows toolchains, if you take special precautions. The major barrier is that the runtime C libraries on these platforms are not exactly compatible on the source level. But you should also take into account CPU differences, STL variations, compiler-specific pragmas and attributes, etc. CMake can help a lot, but I find it easier to have a separate CMakeLists.txt for Android Studio, rather than trying to match all platforms together.

My native code must run on Android and iOS, so I prefer to run my unit tests in XCode (with googletest) for iOS, and only run integration tests on Android.

Regarding the STATIC libs, you don't detail what things actually break. But maybe this sample https://github.com/alexcohn/android.mk-inner-library-sample/commit/7b980de3a38be29ae9aea294682ac50830a391ca will help you.

BR,

Alex Cohn