basic Android class library port merged

[Joel Dice]

Hi all,

I recently merged basic support for using the Android class library as an
alternative to the Avian and OpenJDK libraries. There are instructions
for using it in README.md. There are a few things to note:

* only tested on x86(_64) Linux so far

* The VM test suite passes and the avian-swt-examples build and work, but
I haven't tried anything beyond that yet; there will definitely be more to
do as we test other applications

* upstream libcore.git is not 64-bit friendly yet, but the Android devs
are receptive to patches and will even consider merging ports to e.g.
Windows if they're clean:
https://groups.google.com/group/android-contrib/browse_thread/thread/b71d884e9b17e236.
Meanwhile, I've created a temporary fork with 64-bit support added:
https://github.com/dicej/android-libcore64

Let me know if you have any questions or see problems with the changes.

[Pablo Guerrero]

Hi Joel, 

those are really good news. I've tried to compile it on mac, and here are the results.

You need to configure openssl for 64bit, otherwise doesn't link: './Configure darwin64-x86_64-cc' instead of './config'

On the makefile, there is a missing '-e' after '-i' for the sed command on line 1444

I had to remove -Werror when compiling android native code, otherwise I get: 

cc1plus: warnings being treated as errors

build/darwin-x86_64-android/android-src/libcore_io_Posix.cpp: In function ‘void Posix_setsockoptGroupReq(JNIEnv*, _jobject*, _jobject*, jint, jint, _jobject*)’:

build/darwin-x86_64-android/android-src/libcore_io_Posix.cpp:1177: warning: ‘Posix_setsockoptGroupReq(JNIEnv*, _jobject*, _jobject*, jint, jint, _jobject*)::group_req64’ declared with greater visibility than the type of its field ‘Posix_setsockoptGroupReq(JNIEnv*, _jobject*, _jobject*, jint, jint, _jobject*)::group_req64::gr_group’

make: *** [build/darwin-x86_64-android/libcore_io_Posix.o] Error 1

There were many errors compiling  libcore_net_RawSocket.cpp, but I managed to make it compile. I tried my best to find correct replacements without changing much code, but I've never played with sockets in C at this level before, so it's probably wrong. Also, I just modified it to compile on Mac, as I don't know what are the best practices for doing this with defines. I hope at least helps pointing in the right direction. The changes made to libcore_net_RawSocket.cpp at the end of the email.

Finally, during test, Buffers, Misc and Strings failed. I attach the log.

That's all for now, but it looks that porting it to Mac should be easy.

Cheers,

Pablo

############################

@@ -25,20 +25,36 @@

 #include "jni.h" 

 #include <sys/types.h>

 #include <sys/socket.h>

-#include <linux/rtnetlink.h>

+//#include <linux/rtnetlink.h>

 #include <net/if.h>

-#include <linux/if_ether.h>

-#include <linux/if_packet.h>

+//#include <linux/if_ether.h>

+//#include <linux/if_packet.h>

 #include <arpa/inet.h>

 #include <errno.h>

 #include <fcntl.h>

 #include <poll.h>

 #include <netinet/ip.h>

-#include <linux/udp.h>

+//#include <linux/udp.h>

+#include <netinet/udp.h>

+

+typedef uint16_t __be16;

+

+struct sockaddr_ll {

+    unsigned short  sll_family;

+    __be16          sll_protocol;

+    int             sll_ifindex;

+    unsigned short  sll_hatype;

+    unsigned char   sll_pkttype;

+    unsigned char   sll_halen;

+    unsigned char   sll_addr[8];

+};

+

+#define PF_PACKET AF_INET

+#define AF_PACKET AF_INET 

 union sockunion {

     sockaddr sa;

     sockaddr_ll sll;

 };

@@ -185,20 +201,20 @@ static jint RawSocket_recvPacket(JNIEnv* env, jclass, jobject fileDescriptor,

   }

   if (port != -1) {

     // quick check for UDP type & UDP port

     // the packet is an IP header, UDP header, and UDP payload

-    if ((size < (sizeof(struct iphdr) + sizeof(struct udphdr)))) {

+    if ((size < (sizeof(struct ip) + sizeof(struct udphdr)))) {

       return 0;  // runt packet

     }

-    u_int8_t ip_proto = ((iphdr *) packetData)->protocol;

+    u_int8_t ip_proto = ((ip *) packetData)->ip_p;

     if (ip_proto != IPPROTO_UDP) {

       return 0;  // something other than UDP

     }

-    __be16 destPort = htons((reinterpret_cast<udphdr*>(packetData + sizeof(iphdr)))->dest);

+    __be16 destPort = htons((reinterpret_cast<udphdr*>(packetData + sizeof(ip)))->uh_dport);

     if (destPort != port) {

       return 0; // something other than requested port

     }

   }

On Thu, Feb 28, 2013 at 7:53 AM, <simon.och...@gmail.com> wrote:

Very impressive!

I don't have much time currently but when the dust settles, I'll test the Scala/REPL stuff on top of it.

--
You received this message because you are subscribed to the Google Groups "Avian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to avian+un...@googlegroups.com.
To post to this group, send email to av...@googlegroups.com.
Visit this group at http://groups.google.com/group/avian?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Joel Dice]

On Thu, 28 Feb 2013, Pablo Guerrero wrote:

> those are really good news. I've tried to compile it on mac, and here are
> the results.

Thanks so much for doing that. I've updated android-libcore64 and avian,
and now all the tests except Datagrams pass on OS X. Datagrams fails due
to a NPE in NetworkInterface.getNetworkInterfacesList because it tries to
open /sys/class/net, which doesn't exist on OS X. Unfortunately, the
networking code is pretty Linux-specific, so it's going to take some
effort to get it working elsewhere.

[Pablo Guerrero]

Hi Joel,

I've just tried it, and it works perfectly. Datagrams is working for me, I don't know if you committed something after your mail, or if it's a random error, but I have run the tests multiple times and they all pass. 

I've noticed that the binary it's now 30MB, which is pretty big. The main problem is ICU that takes around 25MB (17 of data). Android builds ICU his own way, and gets smaller sizes, but doesn't work out of the box.

Here you can see the sizes for libicuXXX of a custom android build: http://o2droid.phj.hu/trac/browser/img/beta2/system/lib

They are really small, probably in part because they are for ARM and dynamic libraries.

I've created a minimal makefile to build the same files as Android does, and for the code the size reduces to half. Still double than the previous link. Maybe we could try to remove all the dead code when linking, but I didn't manage to do it yet (it's related with  -DU_STATIC_IMPLEMENTATION as show on a link below).

About the data, it's possible to create custom builds here: http://apps.icu-project.org/datacustom/

And there are instructions for custom builds here: http://userguide.icu-project.org/icudata

General instructions to reduce the size here: http://userguide.icu-project.org/packaging

So it seems that it's possible to reduce the size, but it's not easy. To have a classpath of 5MB but need 25MB for ICU it's an overkill.

Do you have any idea how openjdk is doing it? What is it supported in Avian? 

Cheers,

Pablo

--
You received this message because you are subscribed to the Google Groups "Avian" group.

To unsubscribe from this group and stop receiving emails from it, send an email to avian+unsubscribe@googlegroups.com.

[Joel Dice]

On Fri, 1 Mar 2013, Pablo Guerrero wrote:

> Hi Joel,
> I've just tried it, and it works perfectly. Datagrams is working for me, I
> don't know if you committed something after your mail, or if it's a random

> error, but I have run the tests multiple times and they all pass.�

Odd; it was still broken last time I tried it. I'll take a closer look
when I get a chance to see why it might work on some systems but not
others.

> I've noticed that the binary it's now 30MB, which is pretty big. The main
> problem is ICU that takes around 25MB (17 of data). Android builds ICU his
> own way, and gets smaller sizes, but doesn't work out of the box.
>
> Here you can see the sizes for libicuXXX of a custom android
> build:�http://o2droid.phj.hu/trac/browser/img/beta2/system/lib
> They are really small, probably in part because they are for ARM and dynamic
> libraries.
>
> I've created a minimal makefile to build the same files as Android does, and
> for the code the size reduces to half. Still double than the previous link.
> Maybe we could try to remove all the dead code when linking, but I didn't
> manage to do it yet (it's related with��-DU_STATIC_IMPLEMENTATION as show on
> a link below).
>
> About the data, it's possible to create custom builds
> here:�http://apps.icu-project.org/datacustom/
> And there are instructions for custom builds
> here:�http://userguide.icu-project.org/icudata
> General instructions to reduce the size here:
> http://userguide.icu-project.org/packaging
>
> So it seems that it's possible to reduce the size, but it's not easy. To
> have a classpath of 5MB but need 25MB for ICU it's an overkill.

I agree. I've had good luck shrinking C++ projects in the past by using
-fvisibility=hidden and -flto flags to GCC/clang. It might be worth
trying that with ICU, OpenSSL, and Expat in addition to the techniques you
found.

>
> Do you have any idea how openjdk is doing it? What is it supported in

> Avian?�

Avian's library just does UTF-8 and has minimal localization and timezone
support. I haven't looked closely at OpenJDK, but I suspect it relies
more heavily on the OS for help (e.g. for loading timezone info) rather
than do everything itself.

[Pablo Guerrero]

Hi Joel,

I tried building ICU and Avian with LLVM-GCC-4.2, and lto, but I couldn't make it work. I tried also with GCC-4.7 and I got the same error. Maybe these GCC builds doesn't enable LTO.

I also tried building both with clang and lto, and I cannot link the static library with avian. The problem seems to be that once the .o are linked into the .a, the LTO information is lost/unusable.

This kind of optimizations across libraries/build systems are harder than I expected.

Cheers,

Pablo

[Joel Dice]

On Fri, 1 Mar 2013, Pablo Guerrero wrote:

> Hi Joel,

> I tried building ICU and Avian with LLVM-GCC-4.2, and lto, but I couldn't
> make it work. I tried also with GCC-4.7 and I got the same error. Maybe
> these GCC builds doesn't enable LTO.

GCC 4.7 should work, but you have to make sure you use the same
optimization flags (including -flto) for both compiling and linking.

> I also tried building both with clang and lto, and I cannot link the static
> library with avian. The problem seems to be that once the .o are linked into
> the .a, the LTO information is lost/unusable.

If you specify -flto when compiling with clang, it will produce an LLVM
bitcode file instead of a Mach-O file, from which it can retrieve what it
needs to do optimization at link time. However, that's probably a dead
end anyway, because, as Josh Warner reported, clang's LTO doesn't preserve
exported symbols when generating an executable (as opposed to a shared
library) any more.

> This kind of optimizations across libraries/build systems are harder than I
> expected.

Yeah, LTO is kind of a new thing in general, and unless an app or library
is already designed to support it, it can be hard to get it working.
Anyway, it's not really a prerequisite for shrinking the code -- we should
be able to do dead code elimination without enabling full LTO. The key
from my perspective is to convince these libraries to build with
-fvisibility=hidden, and then use the -dead_strip linker command.

I'm going to have to rework the Avian/Android build system a bit anyway to
get iOS working, so I'll see if I can make progress on code size at the
same time.

[Pablo Guerrero]

Hi Joel,

Thanks for the info.

I don't have time to explain now, but I'm working on a different idea based on the Android.mk makefiles, I will send an email when I have something.

Cheers,

Pablo

[Pablo Guerrero]

Hi Joel,

I've tried a couple of things but I'm getting an Abort when executing Avian.

The first option is to configure icu with ./configure --enable-static --with-data-packaging=archive

And then link against $(android)/icu4c/stubdata/libicudata.a instead of $(android)/icu4c/lib/libicudata.a

That shrinks avian to about 12MB(4.5MB compressed with Zip), and in theory, we should be able to provide a custom .dat file that an avian's user could download here: http://apps.icu-project.org/datacustom/ We could provide a really minimum .dat for english, or whatever it's decided. The problem is I get "Abort trap: 6" when executing. The same happens without using --with-data-packaging=archive (bigger size of about 15MB)

As that didn't work, I tried to compile ICU using only the sources that Android is using in it's Android.mk files and integrate it into avian's build. That was harder than I expected, because it took a while to understand in depth avian's makefile, but in the end, the size it's similar (~15MB), and I also get the same Abort.

That leads me to think that we cannot shrink avian much more only with icu because it already includes only what it needs. To shrink it more, we would need to look to the other libraries, or maybe that's it.

The problem now is that I have no idea how to debug the Abort. 

I copy the changes I did to the makefile to build from Android.mk sources just in case they are useful.

Cheers,

Pablo

########

185a181,193

> get-icu4c-src = $(shell \

> cat $(android)/icu4c/$(1)/Android.mk | \

> awk "{if (sub(/\\\\$$/,\"\")) printf \"%s\", \$$0; else print \$$0}" | \

>                 grep "src_files .=" | \

>                 tr -d '\n' | \

>                 sed -e 's/src_files .=//g' | \

>                 awk '{gsub(/[[:space:]]+/,"\n$(android)/icu4c/$(1)/");print}' | \

>                 awk /./ | \

>                 sed -e 's/[^\/]*\/\.\.\///g' | \

> sed -e 's/[^ ]*\.h//g' )

> icu4c-src := $(call get-icu4c-src,'common') 

> icu4c-src += $(call get-icu4c-src,'i18n') 

> classpath-objects += $(shell echo $(icu4c-src) | sed -e 's%[^ ]*/\([^/]*\)\.cp*%$(build)/icu4c/\1.o%g')

1430a1442,1466

> $(build)/icu4c/%.o: $(android)/icu4c/common/%.c

> @echo "compiling $(@)"

> @mkdir -p $(dir $(@))

> $(cc) -DPIC -D_REENTRANT -DU_COMMON_IMPLEMENTATION -fvisibility=hidden $(android-cflags) -c $$($(windows-path) $(<)) $(call output,$(@))

> 

> $(build)/icu4c/%.o: $(android)/icu4c/i18n/%.c

> @echo "compiling $(@)"

> @mkdir -p $(dir $(@))

> $(cc) -DPIC -D_REENTRANT -D_REENTRANT -DU_I18N_IMPLEMENTATION -fvisibility=hidden $(android-cflags) -c $$($(windows-path) $(<)) $(call output,$(@))

> 

> $(build)/icu4c/%.o: $(android)/icu4c/stubdata/%.c

> @echo "compiling $(@)"

> @mkdir -p $(dir $(@))

> $(cc) -DPIC -D_REENTRANT -DU_COMMON_IMPLEMENTATION -fvisibility=hidden $(android-cflags) -c $$($(windows-path) $(<)) $(call output,$(@))

> 

> $(build)/icu4c/%.o: $(android)/icu4c/common/%.cpp

> @echo "compiling $(@)"

> @mkdir -p $(dir $(@))

> $(cxx) -DPIC -D_REENTRANT -DU_COMMON_IMPLEMENTATION -fvisibility=hidden $(android-cflags) -c $$($(windows-path) $(<)) $(call output,$(@))

> 

> $(build)/icu4c/%.o: $(android)/icu4c/i18n/%.cpp

> @echo "compiling $(@)"

> @mkdir -p $(dir $(@))

> $(cxx) -DPIC -D_REENTRANT -DU_I18N_IMPLEMENTATION -fvisibility=hidden $(android-cflags) -c $$($(windows-path) $(<)) $(call output,$(@))

> 

[Joel Dice]

On Wed, 6 Mar 2013, Pablo Guerrero wrote:

> Hi Joel,

> I've tried a couple of things but I'm getting an Abort when executing Avian.
>
> The first option is to configure icu with�./configure --enable-static
> --with-data-packaging=archive
> And then link against $(android)/icu4c/stubdata/libicudata.a instead
> of�$(android)/icu4c/lib/libicudata.a
>
> That shrinks avian to about 12MB(4.5MB compressed with Zip), and in theory,
> we should be able to provide a custom .dat file that an avian's user could
> download here:�http://apps.icu-project.org/datacustom/�We could provide a
> really minimum .dat for english, or whatever it's decided. The problem is I
> get "Abort trap: 6" when executing. The same happens without using
> --with-data-packaging=archive (bigger size of about 15MB)
>
> As that didn't work, I tried to compile ICU using only the sources that
> Android is using in it's Android.mk files and integrate it into avian's
> build. That was harder than I expected, because it took a while to
> understand in depth avian's makefile, but in the end, the size it's similar
> (~15MB), and I also get the same Abort.
>
> That leads me to think that we cannot shrink avian much more only with icu
> because it already includes only what it needs. To�shrink�it more, we would
> need to look to the other libraries, or maybe that's it.
>

> The problem now is that I have no idea how to debug the Abort.�

Thanks for working on this, Pablo. Your best bet for debugging it is to
disable optimization and enable debug symbols (e.g. build ICU with
CFLAGS="-O0 -g3" CXXFLAGS="-O0 -g3" and Avian with mode=debug) and then
run Avian in gdb (e.g. gdb --args build/linux-x86_64-debug-android/avian
-cp build/linux-x86_64-debug-android/test Hello). You can use the run
command to start it and the backtrace command to get a stack trace when it
aborts.

> I copy the changes I did to the makefile to build from Android.mk sources
> just in case they are useful.

Thanks. I'll give it a try myself when I have some time.

[Pablo Guerrero]

Thanks Joel, I'll try to debug it with this.

Cheers,

Pablo

[Joshua Warner]

It may also be helpful to do a bootimage build.  All of the methods in the bootimage build will show up as symbols in gdb. (as opposed to the usual anonymous jitted addresses).

-Joshua

To unsubscribe from this group and stop receiving emails from it, send an email to avian+un...@googlegroups.com.

[Pablo Guerrero]

Hi,

I finally managed to reduce the size of ICU without errors (at least on a HelloWorld). I have a working Avian vm with the full android classpath on 14.3MB (macosx) and 3.5MB compressed with lzma or upx.

I have the impression that before I had different errors masking each other, because I tried some of this things before and it didn't work.

Actually it's quite simple. Just go to http://apps.icu-project.org/datacustom/ICUData50.html (The 50 version is important, it defaults to 51) and leave just Base Data enabled. Then, remove everything under icu4c/data/in and the icudt50l.dat file there (the name is also important). Finally build as usual.

Of course, you can keep whatever you need, but this demonstrates that is able to work with just Base Data. I would like to test if it's possible to provide other dat files on top of this one using the env variables, but for that I will need a more realistic example.

I had to modify the file icu4c/commom/unicode/uvernum.h to be able to compile because a version with 4 parts is invalid:
-#define U_ICU_VERSION "50.1.1." U_ICU_STRINGIFY(U_ICU_VERSION_BUILDLEVEL_NUM)
+#define U_ICU_VERSION "50.1.1"

I also had to modify some things in libcore too in order to compile on macosx:

diff --git a/luni/src/main/native/libcore_io_OsConstants.cpp b/luni/src/main/native/libcore_io_OsConstants.cpp
index 56a159f..a34e0d2 100644
--- a/luni/src/main/native/libcore_io_OsConstants.cpp
+++ b/luni/src/main/native/libcore_io_OsConstants.cpp
@@ -359,8 +359,12 @@ static void OsConstants_initConstants(JNIEnv* env, jclass c) {
     initConstant(env, c, "SO_KEEPALIVE", SO_KEEPALIVE);
     initConstant(env, c, "SO_LINGER", SO_LINGER);
     initConstant(env, c, "SO_OOBINLINE", SO_OOBINLINE);
+#if defined(SO_PASSCRED)
     initConstant(env, c, "SO_PASSCRED", SO_PASSCRED);
+#endif
+#if defined(SO_PEERCRED)
     initConstant(env, c, "SO_PEERCRED", SO_PEERCRED);
+#endif
     initConstant(env, c, "SO_RCVBUF", SO_RCVBUF);
     initConstant(env, c, "SO_RCVLOWAT", SO_RCVLOWAT);
     initConstant(env, c, "SO_RCVTIMEO", SO_RCVTIMEO);
diff --git a/luni/src/main/native/libcore_io_Posix.cpp b/luni/src/main/native/libcore_io_Posix.cpp
index 1daa83d..b2c627c 100644
--- a/luni/src/main/native/libcore_io_Posix.cpp
+++ b/luni/src/main/native/libcore_io_Posix.cpp
@@ -278,10 +278,14 @@ static jobject makeStructTimeval(JNIEnv* env, const struct timeval& tv) {
             static_cast<jlong>(tv.tv_sec), static_cast<jlong>(tv.tv_usec));
 }
 
+#if defined(__APPLE__)
+static jobject makeStructUcred(JNIEnv* env, const struct ucred& u) { abort(); }
+#else
 static jobject makeStructUcred(JNIEnv* env, const struct ucred& u) {
   static jmethodID ctor = env->GetMethodID(JniConstants::structUcredClass, "<init>", "(III)V");
   return env->NewObject(JniConstants::structUcredClass, ctor, u.pid, u.uid, u.gid);
 }
+#endif
 
 static jobject makeStructUtsname(JNIEnv* env, const struct utsname& buf) {
     TO_JAVA_STRING(sysname, buf.sysname);
diff --git a/luni/src/main/native/org_apache_harmony_xnet_provider_jsse_NativeCrypto.cpp b/luni/src/main/native/org_apache_harmony_xnet_provider_jsse_NativeCrypto.cpp
index ee6a57e..80588e5 100644
--- a/luni/src/main/native/org_apache_harmony_xnet_provider_jsse_NativeCrypto.cpp
+++ b/luni/src/main/native/org_apache_harmony_xnet_provider_jsse_NativeCrypto.cpp
@@ -4460,15 +4460,15 @@ static void NativeCrypto_X509_REVOKED_print(JNIEnv* env, jclass, jlong bioRef, j
         return;
     }
 
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wwrite-strings"
+//#pragma GCC diagnostic push
+//#pragma GCC diagnostic ignored "-Wwrite-strings"
     BIO_printf(bio, "Serial Number: ");
     i2a_ASN1_INTEGER(bio, revoked->serialNumber);
     BIO_printf(bio, "\nRevocation Date: ");
     ASN1_TIME_print(bio, revoked->revocationDate);
     BIO_printf(bio, "\n");
-    X509V3_extensions_print(bio, "CRL entry extensions", revoked->extensions, 0, 0);
-#pragma GCC diagnostic pop
+    X509V3_extensions_print(bio, (char *)"CRL entry extensions", revoked->extensions, 0, 0);
+//#pragma GCC diagnostic pop
 }
 
 static jbyteArray NativeCrypto_get_X509_CRL_crl_enc(JNIEnv* env, jclass, jlong x509CrlRef) {

[Joel Dice]

On Wed, 3 Jul 2013, Pablo Guerrero wrote:

> I finally managed to reduce the size of ICU without errors (at least on a
> HelloWorld). I have a working Avian vm with the full android classpath on
> 14.3MB (macosx) and 3.5MB compressed with lzma or upx.

Awesome! Thanks for working on that.

[Pablo Guerrero]

I've tried to run a simple helloworld in scala, and I'm getting a crash. I've tried using plain Avian putting the scala library in the cp and also using scala-avian and I get the same error on both.

When I run it on GDB I get this error:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x000000000600000c
0x00000001000cae19 in NativeBN_putULongInt (env=0x101815600, unnamed_arg=0x7fff5fbfc078, a=0x6000000, dw=0, neg=0 '\0') at java_math_NativeBN.cpp:108
108    java_math_NativeBN.cpp: No such file or directory.
    in java_math_NativeBN.cpp
(gdb) bt
#0  0x00000001000cae19 in NativeBN_putULongInt (env=0x101815600, unnamed_arg=0x7fff5fbfc078, a=0x6000000, dw=0, neg=0 '\0') at java_math_NativeBN.cpp:108
#1  0x00000001000c00bf in .Lcall ()
#2  0x00000001000040a2 in vm::dynamicCall (function=0x1000cadb0, arguments=0x7fff5fbfbf40, argumentTypes=0x7fff5fbfbf30 "\a\a\003\004\001", argumentCount=5, unnamed_arg=48, returnType=1) at x86.h:197
#3  0x0000000100001b51 in call (this=0x101013fa0, function=0x1000cadb0, arguments=0x7fff5fbfbf40, types=0x7fff5fbfbf30 "\a\a\003\004\001", count=5, size=48, returnType=1) at posix.cpp:782
#4  0x000000010007795e in invokeNativeSlow (t=0x101815600, method=0x105252af8, function=0x1000cadb0) at compile.cpp:7778
#5  0x00000001000784bd in invokeNative2 (t=0x101815600, method=0x105252af8) at compile.cpp:7850
#6  0x000000010005f666 in invokeNative (t=0x101815600) at compile.cpp:7882
#7  0x000000010200007b in ?? ()
#8  0x00000001000763d6 in invoke (thread=0x101815600, method=0x1058d96e8, arguments=0x7fff5fbfc4f0) at compile.cpp:8669
#9  0x000000010005ee72 in invokeList (this=0x1010147b0, t=0x101815600, method=0x1058d96e8, this_=0x0, indirectObjects=false, arguments=0x7fff5fbfc748) at compile.cpp:9170
#10 0x0000000100039569 in vm::Processor::invoke (this=0x1010147b0, t=0x101815600, method=0x1058d96e8, this_=0x0) at processor.h:188
#11 0x0000000100028740 in vm::initClass (t=0x101815600, c=0x1058d9908) at machine.cpp:4559
#12 0x0000000100075029 in compile (t=0x101815600, allocator=0x101014830, bootContext=0x0, method=0x1058d5dc0) at compile.cpp:10460
#13 0x00000001000758e5 in compileMethod2 (t=0x101815600, ip=0x10203e86c) at compile.cpp:9549
#14 0x000000010005f430 in compileMethod (t=0x101815600) at compile.cpp:7615
#15 0x000000010200001b in ?? ()
#16 0x00000001000763d6 in invoke (thread=0x101815600, method=0x1019eb5c8, arguments=0x7fff5fbfcf20) at compile.cpp:8669
#17 0x000000010005ee72 in invokeList (this=0x1010147b0, t=0x101815600, method=0x1019eb5c8, this_=0x0, indirectObjects=false, arguments=0x7fff5fbfd178) at compile.cpp:9170
#18 0x0000000100039569 in vm::Processor::invoke (this=0x1010147b0, t=0x101815600, method=0x1019eb5c8, this_=0x0) at processor.h:188
#19 0x0000000100028740 in vm::initClass (t=0x101815600, c=0x1019eb608) at machine.cpp:4559
#20 0x0000000100058403 in tryInitClass (t=0x101815600, class_=0x1019eb608) at compile.cpp:2314
#21 0x000000010203db69 in ?? ()
#22 0x00000001000763d6 in invoke (thread=0x101815600, method=0x105232d58, arguments=0x7fff5fbfd580) at compile.cpp:8669
#23 0x000000010005ee72 in invokeList (this=0x1010147b0, t=0x101815600, method=0x105232d58, this_=0x0, indirectObjects=false, arguments=0x7fff5fbfd7d8) at compile.cpp:9170
#24 0x0000000100039569 in vm::Processor::invoke (this=0x1010147b0, t=0x101815600, method=0x105232d58, this_=0x0) at processor.h:188
#25 0x0000000100028740 in vm::initClass (t=0x101815600, c=0x105226e50) at machine.cpp:4559
#26 0x0000000100075029 in compile (t=0x101815600, allocator=0x101014830, bootContext=0x0, method=0x1052327d8) at compile.cpp:10460
#27 0x00000001000758e5 in compileMethod2 (t=0x101815600, ip=0x10203c6c0) at compile.cpp:9549
#28 0x000000010005f430 in compileMethod (t=0x101815600) at compile.cpp:7615
#29 0x000000010200001b in ?? ()
#30 0x00000001000763d6 in invoke (thread=0x101815600, method=0x104e224c0, arguments=0x7fff5fbfdfb0) at compile.cpp:8669
#31 0x000000010005ee72 in invokeList (this=0x1010147b0, t=0x101815600, method=0x104e224c0, this_=0x0, indirectObjects=false, arguments=0x7fff5fbfe208) at compile.cpp:9170
#32 0x0000000100039569 in vm::Processor::invoke (this=0x1010147b0, t=0x101815600, method=0x104e224c0, this_=0x0) at processor.h:188
#33 0x0000000100028740 in vm::initClass (t=0x101815600, c=0x104e1ea80) at machine.cpp:4559
#34 0x0000000100058403 in tryInitClass (t=0x101815600, class_=0x104e1ea80) at compile.cpp:2314
#35 0x0000000102031d39 in ?? ()
#36 0x00000001000763d6 in invoke (thread=0x101815600, method=0x101a04148, arguments=0x7fff5fbfe610) at compile.cpp:8669
#37 0x000000010005ee72 in invokeList (this=0x1010147b0, t=0x101815600, method=0x104ede0f8, this_=0x0, indirectObjects=false, arguments=0x7fff5fbfe868) at compile.cpp:9170
#38 0x0000000100039569 in vm::Processor::invoke (this=0x1010147b0, t=0x101815600, method=0x101a04148, this_=0x0) at processor.h:188
#39 0x0000000100028740 in vm::initClass (t=0x101815600, c=0x104e4bec0) at machine.cpp:4559
#40 0x0000000100058403 in tryInitClass (t=0x101815600, class_=0x101a05b78) at compile.cpp:2314
#41 0x0000000102031189 in ?? ()
#42 0x00000001000763d6 in invoke (thread=0x101815600, method=0x1019f6818, arguments=0x7fff5fbfec70) at compile.cpp:8669
#43 0x000000010005ee72 in invokeList (this=0x1010147b0, t=0x101815600, method=0x104f18d10, this_=0x0, indirectObjects=false, arguments=0x7fff5fbfeec8) at compile.cpp:9170
#44 0x0000000100039569 in vm::Processor::invoke (this=0x1010147b0, t=0x101815600, method=0x1019f6818, this_=0x0) at processor.h:188
#45 0x0000000100028740 in vm::initClass (t=0x101815600, c=0x104eff650) at machine.cpp:4559
#46 0x0000000100058403 in tryInitClass (t=0x101815600, class_=0x1019fdce0) at compile.cpp:2314
#47 0x0000000102030ff6 in ?? ()
#48 0x00000001000763d6 in invoke (thread=0x101815600, method=0x101c6a328, arguments=0x7fff5fbff480) at compile.cpp:8669
#49 0x000000010005ee72 in invokeList (this=0x1010147b0, t=0x101815600, method=0x104fa8fb0, this_=0x0, indirectObjects=true, arguments=0x7fff5fbff8b0) at compile.cpp:9170
#50 0x000000010008b6c7 in callStaticVoidMethodV (t=0x101815600, arguments=0x7fff5fbff700) at jnienv.cpp:1434
#51 0x00000001000c0130 in vmRun () at type-declarations.cpp:2532
#52 0x0000000100039260 in vm::runRaw (t=0x101815600, function=0x10008b620 <callStaticVoidMethodV>, arguments=0x7fff5fbff700) at machine.h:1950
#53 0x00000001000392d2 in vm::run (t=0x101815600, function=0x10008b620 <callStaticVoidMethodV>, arguments=0x7fff5fbff700) at machine.h:1957
#54 0x0000000100093f55 in CallStaticVoidMethodV (t=0x101815600, unnamed_arg=0x1010199d0, m=9, a=0x7fff5fbff8b0) at jnienv.cpp:1444
#55 0x0000000100119b40 in JNIEnv_::CallStaticVoidMethod (this=0x101815600, cls=0x1010199d0, methodID=0x9) at jni.h:1516
#56 0x00000001001192f7 in main (ac=4, av=0x7fff5fbffaa8) at main.cpp:282

Any idea on how to make it work?

Thanks,
Pablo

[Pablo Guerrero]

Hi again,

It's not a scala related problem. I can reproduce it with this simple java program:

import java.math.BigInteger;

public class BigIntegerError {

    public static void main(String[] args) {
        BigInteger i = BigInteger.valueOf(1234L);
        System.out.println(i);
    }

}

I get the same error, but this time on register_java_math_NativeBN, but I think it's going to crash on any function that uses it because the problem is the memory address.

I also confirm that with the steps for ICU provided before, it's possible to provide extra data files setting the env variable ICU_DATA.

Cheers,
Pablo

[Pablo Guerrero]

I finally found the bug, but it's a know one, xD. They are passing around int as a pointer instead of long in 64bit.

Fortunately there is a recent patch to libcore that fix this:
https://android.googlesource.com/platform/libcore/+/a125dded8ab0490d05e2fa9ec2e821ef1ae6facd%5E!/

And more general 64bit fixes:
https://android.googlesource.com/platform/libcore/+/1e5d730e58d94c3bfa14b7dde5ab3981fe5a170b

Has been any more work on this after this thread?
https://groups.google.com/forum/#!searchin/android-contrib/64/android-contrib/tx2ITpsX4jY/CcSeYWFme_gJ

If they are serious about porting it to 64bits, maybe we can just push patches to them and rely on the official one. And use libcore64 for the windows and iOS port.

Cheers,
Pablo

[Joel Dice]

On Thu, 4 Jul 2013, Pablo Guerrero wrote:

> Has been any more work on this after this thread?
> https://groups.google.com/forum/#!searchin/android-contrib/64/android-contr
> ib/tx2ITpsX4jY/CcSeYWFme_gJ
>
> If they are serious about porting it to 64bits, maybe we can just push
> patches to them and rely on the official one. And use libcore64 for the
> windows and iOS port.

Yes, I pushed a bunch of patches upstream to fix problems like this, and
they were all quickly accepted. The remaining non-upstreamed stuff in my
libcore64 repo is just for Windows and iOS/OSX, despite the name.

[Pablo Guerrero]

Good. I've updated libcore64 from upstream and I'm fixing the compilation problems to make it work again.

I'll send an update when it works.

Cheers,
Pablo

--
You received this message because you are subscribed to the Google Groups "Avian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to avian+unsubscribe@googlegroups.com.
To post to this group, send email to av...@googlegroups.com.

Visit this group at http://groups.google.com/group/avian.

[Pablo Guerrero]

Hi,

I finally managed to build it, many things have been restructured on libcore, but it looks better now. Also, I've checked the java functions calling native code and it looks like there are no more int pointers. That doesn't mean that it's 64bit capable, but it's a start.

The problem I have now is that DalvikVM is calling "JniConstants::init(env);" at some point before starting but Avian is not doing it, and Crypto is crashing because of some uninitialized variables.

( why is Dalvik registering all the native methonds on startup ? crypto and many others isn't used in the application. )

Before, this was called from Register.cpp inside libcore, but now it's out, so I don't think is a good idea to put it back again.
My impression is that classpath-android.cpp on preeBoot is the best place to put it, but I would need to call something to get env from the thread (as in Register.cpp).

What do you think?

Also, I have the impression that classpath-android.cpp overlaps with some code from libcore and libnativehelper, but this seems to work.

We are getting closer to have this working !

Cheers,
Pablo

[Joel Dice]

On Thu, 4 Jul 2013, Pablo Guerrero wrote:

> ( why is Dalvik registering all the native methonds on startup ? crypto and
> many others isn't used in the application. )

Yeah, I noticed that. They don't seem to want to use the traditional
dlopen/dlsym style of looking up JNI methods lazily for some reason. I
don't know why.

> Before, this was called from Register.cpp inside libcore, but now it's out,
> so I don't think is a good idea to put it back again.
> My impression is that classpath-android.cpp on preeBoot is the best place to
> put it, but I would need to call something to get env from the thread (as in
> Register.cpp).
>
> What do you think?

A vm::Thread* is also a JNIEnv* (see the "typedef Thread JNIEnv" in
machine.h), so you should be able to just pass the thread pointer to
JniConstants::init.

> Also, I have the impression that classpath-android.cpp overlaps with some
> code from libcore and libnativehelper, but this seems to work.

Yeah, I had to replace the implementations of a few functions because the
ones supplied by libcore etc. were not appropriate for Avian or for all
the platforms Avian supports. As libcore becomes more VM- and platform-
independent, we may be able to reduce this.

[Joel Dice]

Hi Pablo,

I just updated android-libcore64 to the latest from upstream and fixed a
few things in Avian to get everything working again (at least on Linux).
I've also updated README.md to specify commit hashes for each dependency
to ensure that the build instructions continue to work regardless of
future upstream changes.

Please let me know what other changes you'd like to see added prior to the
0.7 release (which I'd like to do soon).

Thanks.

[Pablo Guerrero]

Hi Joel,

Today I'm not going to stay on the computer, but I'll check tomorrow if I have something more than your changes and commit it.

Cheers,
Pablo

[Pablo Guerrero]

Hi Joel,

Wow! That was really fast. The core of your commit is almost line by line what I had, it just took me much longer to do it, xD, but I've learnt a lot.

I think just one thing it's missing, correct me if I'm wrong. In the file libcore/crypto/src/main/native/org_conscrypt_NativeCrypto.cpp you want CONSCRYPT_UNBUNDLED
to be unset so that it gets initialized correctly. For that, I've added -DJNI_JARJAR_PREFIX="" to android-cflags.

I attach also a diff on libcore to make it work on macosx. The definition of OS_SHARED_LIB_FORMAT_STR probably should go in other place, but it was just to make it work.

I also tried to run scala on it and it seems to work for basic examples, but I couldn't run the test suite, as I commented on the other thread, because it seems to rely on some behavior of OpenJDK. Maybe it would be nice to run the relevant tests that are part of libcore.

Cheers,
Pablo

[Joel Dice]

On Sun, 7 Jul 2013, Pablo Guerrero wrote:

> Hi Joel,
>

> Wow! That was really fast. The core of your commit is almost line by line
> what I had, it just took me much longer to do it, xD, but I've learnt a lot.
>
> I think just one thing it's missing, correct me if I'm wrong. In the file
> libcore/crypto/src/main/native/org_conscrypt_NativeCrypto.cpp you want
> CONSCRYPT_UNBUNDLED
> to be unset so that it gets initialized correctly. For that, I've added
> -DJNI_JARJAR_PREFIX="" to android-cflags.

I added -DJNI_JARJAR_PREFIX= in my patch, which should have the same
effect. Was it not working for you?

> I attach also a diff on libcore to make it work on macosx. The definition of
> OS_SHARED_LIB_FORMAT_STR probably should go in other place, but it was just
> to make it work.

I had put it in the makefile, which should ensure that it works on Windows
(haven't tried it there, though). I've combined your diff with mine, plus
a few more changes I needed to get it to build on my OS X 10.8 machine.
When you have a chance, please try it and let me know if it works for you.

[Pablo Guerrero]

My fault, I missed those two lines on your patch.

I'll give it a try, but I'm pretty sure it will work as it's basically the same.