[libcore64/android 4.4_r07] Do we need openssl assembly files?

[Carl Perfect]

Hello,

I'm updating robovm libcore to libcore64 that is based on android 4.4r07.
Meaning, I have to upgrade the external libraries like openssl too.

I noticed that openssl 4.4r07 comes with assembly files.
Especially in the crypto folder, and crypto/rc4, and crypto/des and crypto/aes.

I don't see those files in the current robovm tree.
Do we need those?

Carl.

[Carl Perfect]

On Thursday, June 5, 2014 12:09:26 PM UTC+2, Carl Perfect wrote:

Do we need those?

According to openssl used for android 4.1.1 r3, we can delete those asm files.
Good.

 

[Pablo Guerrero]

Not related to your question, but last time I compiled OpenSSL it generated big blobs that need to be included on the final binary.

I've found this option that might help reducing this size: -DOPENSSL_SMALL_FOOTPRINT

I don't have the time to do it now, but it would be nice to see if it works with libcore.

Cheers,

Pablo

[Carl Perfect]

On Friday, June 6, 2014 8:30:41 AM UTC+2, Pablo Guerrero wrote:

Not related to your question, but last time I compiled OpenSSL it generated big blobs that need to be included on the final binary.

I've found this option that might help reducing this size: -DOPENSSL_SMALL_FOOTPRINT

That's interesting.
For the time being, I can't get openssl 1.0.1e to build on my windows box (Robovm is using 1.0.1c)
The error is:

In file included from \android\external\openssl\crypto\er
\err_all.c:60:0:
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:24: error: pasting ")" and "_new" does not give a valid preprocessing token
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
                        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:333:8: note: in definition of macro 'DECLARE_ASN1_ALLOC_FUNCTIONS_name'
  type *name##_new(void); \
        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
 ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:24: error: pasting ")" and "_free" does not give a valid preprocessing token
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
                        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:334:7: note: in definition of macro 'DECLARE_ASN1_ALLOC_FUNCTIONS_name'
  void name##_free(type *a);
       ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
 ^
/android/external/openssl/crypto/../include/openssl/asn1.
:316:8: error: pasting "d2i_" and "(" does not give a valid preprocessing token
  type *d2i_##name(type **a, const unsigned char **in, long len); \
        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:309:2: note: in expansion of macro 'DECLARE_ASN1_ENCODE_FUNCTIONS'
  DECLARE_ASN1_ENCODE_FUNCTIONS(type, name, name)
  ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
 ^
/android/external/openssl/crypto/../include/openssl/asn1.
:317:6: error: pasting "i2d_" and "(" does not give a valid preprocessing token
  int i2d_##name(type *a, unsigned char **out); \
      ^
/android/external/openssl/crypto/../include/openssl/asn1.
:309:2: note: in expansion of macro 'DECLARE_ASN1_ENCODE_FUNCTIONS'
  DECLARE_ASN1_ENCODE_FUNCTIONS(type, name, name)
  ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
 ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:24: error: pasting ")" and "_it" does not give a valid preprocessing token
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
                        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:413:33: note: in definition of macro 'DECLARE_ASN1_ITEM'
  OPENSSL_EXTERN const ASN1_ITEM name##_it;
                                 ^
/android/external/openssl/crypto/../include/openssl/asn1.
:309:2: note: in expansion of macro 'DECLARE_ASN1_ENCODE_FUNCTIONS'
  DECLARE_ASN1_ENCODE_FUNCTIONS(type, name, name)
  ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:539:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_RESPONSE)
 ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:24: error: pasting ")" and "_new" does not give a valid preprocessing token
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
                        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:333:8: note: in definition of macro 'DECLARE_ASN1_ALLOC_FUNCTIONS_name'
  type *name##_new(void); \
        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
 ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:24: error: pasting ")" and "_free" does not give a valid preprocessing token
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
                        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:334:7: note: in definition of macro 'DECLARE_ASN1_ALLOC_FUNCTIONS_name'
  void name##_free(type *a);
       ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
 ^
/android/external/openssl/crypto/../include/openssl/asn1.
:316:8: error: pasting "d2i_" and "(" does not give a valid preprocessing token
  type *d2i_##name(type **a, const unsigned char **in, long len); \
        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:309:2: note: in expansion of macro 'DECLARE_ASN1_ENCODE_FUNCTIONS'
  DECLARE_ASN1_ENCODE_FUNCTIONS(type, name, name)
  ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
 ^
/android/external/openssl/crypto/../include/openssl/asn1.
:317:6: error: pasting "i2d_" and "(" does not give a valid preprocessing token
  int i2d_##name(type *a, unsigned char **out); \
      ^
/android/external/openssl/crypto/../include/openssl/asn1.
:309:2: note: in expansion of macro 'DECLARE_ASN1_ENCODE_FUNCTIONS'
  DECLARE_ASN1_ENCODE_FUNCTIONS(type, name, name)
  ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
 ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:24: error: pasting ")" and "_it" does not give a valid preprocessing token
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
                        ^
/android/external/openssl/crypto/../include/openssl/asn1.
:413:33: note: in definition of macro 'DECLARE_ASN1_ITEM'
  OPENSSL_EXTERN const ASN1_ITEM name##_it;
                                 ^
/android/external/openssl/crypto/../include/openssl/asn1.
:309:2: note: in expansion of macro 'DECLARE_ASN1_ENCODE_FUNCTIONS'
  DECLARE_ASN1_ENCODE_FUNCTIONS(type, name, name)
  ^
/android/external/openssl/crypto/../include/openssl/asn1.
:302:38: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS_name'
 #define DECLARE_ASN1_FUNCTIONS(type) DECLARE_ASN1_FUNCTIONS_name(type, type)
                                      ^
/android/external/openssl/crypto/../include/openssl/ocsp.
:543:1: note: in expansion of macro 'DECLARE_ASN1_FUNCTIONS'
 DECLARE_ASN1_FUNCTIONS(OCSP_REQUEST)
 ^

I had to undefine DSO_DLFCN HAVE_DLFCN_H: the new version of openssl uses a dlfcn's apis that are not ported on windows.
Do you happen to know what the consequences of this are ?

For the sake of completion, the following flags has been added in the android 4.4_r0.7 build

• MONOLITH
• OPENSSL_NO_EC_NISTP_64_GCC_128
• OPENSSL_NO_HEARTBEATS
• OPENSSL_NO_SCTP
• TERMIO

Note:  ZLIB is not defined anymore.

I don't know what is the correct course here:

• try with a more recent version (current is 1.0.1h)
• use prebuilt openssl binaries for the moment
• reading more documetation (brrr) :)
  

[Pablo Guerrero]

Sorry, no idea about the error.

I would go with the prebuilt binaries for the moment, as this is
something pretty standard, and can be fixed in the future.

Cheers,
Pablo

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "RoboVM" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/robovm/GGTubkAT2vg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> robovm+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Carl Perfect]

On Friday, June 6, 2014 1:39:02 PM UTC+2, Pablo Guerrero wrote:

I would go with the prebuilt binaries for the moment, as this is
something pretty standard, and can be fixed in the future.

Or I can use Robovm current openssl.
Which would count as a downgrade for 4.4_r03
I hope the computer gods will forgive me for this...

[Carl Perfect]

For the sake of completion, I use gcc.exe (GCC) 4.8.2

[Carl Perfect]

On Friday, June 6, 2014 1:47:49 PM UTC+2, Carl Perfect wrote:

On Friday, June 6, 2014 1:39:02 PM UTC+2, Pablo Guerrero wrote:

I would go with the prebuilt binaries for the moment, as this is
something pretty standard, and can be fixed in the future.

The best course seems to be to follow this or that procedure.
But now I need msys and perl.
(sigh)

[Carl Perfect]

On Friday, June 6, 2014 2:07:23 PM UTC+2, Carl Perfect wrote:

The best course seems to be to follow this or that procedure.
But now I need msys and perl.
(sigh)

Okay, I built openssl from the official sources (latest version)
But... some functions like SSL_SESSION_get_version are not available.
It seems that Android is running some special version of openssl.

More at 9.

[Carl Perfect]

On Friday, June 6, 2014 4:53:05 PM UTC+2, Carl Perfect wrote:

More at 9.

Android's openssl  doesn't work with clang either

[Carl Perfect]

Newer versions have the same issue

[Carl Perfect]

Okay, this is working now.

I downloaded openssl 1.0.1e applied the Android's openssl patches:

• channelid.patch             
• alpn.patch                  
• jsse.patch                  
• handshake_cutthrough.patch  

and build the library.
It seems to work

[Pablo Guerrero]

Nice. Did you try also with -DOPENSSL_SMALL_FOOTPRINT?

[Carl Perfect]

On Saturday, June 7, 2014 9:15:56 AM UTC+2, Pablo Guerrero wrote:

Nice. Did you try also with -DOPENSSL_SMALL_FOOTPRINT?

Not yet.
To be fair, I am rushing to get a compiling/linking application.
I am eager to test the new two phases compiler that Jan released.

So to get there, I'm commenting code left and right.
While leaving explanation in the code and documenting the process.
I have a list of high level issues ie issues that are not related to port luni to windows.

If you have any idea, don't hesitate to chime.

Nevertheless, as you would imagine porting C code to windows is a pretty thankless task.
And I'm not afraid to say that reading your feedback/encouragement on the group is the best of it.
Until I get a working prototype that is :)

Carl.

[Pablo Guerrero]

Yes, it's not really encouraging until you get something basic working.

Cheers,
Pablo

[Carl Perfect]

On Saturday, June 7, 2014 7:15:34 PM UTC+2, Pablo Guerrero wrote:

Yes, it's not really encouraging until you get something basic working.

Issue fixed here : https://github.com/PerfectCarl/robovm/commit/f5eb475dcc41ee896031e2abd88919bea697be03

This is not perfect as I had to update one openssl.
I have a hunch that I could achieve the same with a windows specific include file that I could add in the cmakefile.

I'll revisit this one day.
Promise :)

And try your compilation flag, Pablo!

[Pablo Guerrero]

Yes, I think the most importante is to keep moving, and maintain a
list of things to improve in the future as you are doing.

[Carl Perfect]

On Sunday, June 8, 2014 8:45:13 PM UTC+2, Pablo Guerrero wrote:

Yes, I think the most importante is to keep moving, and maintain a
list of things to improve in the future as you are doing.

Indeed.

Keep moving. Keep committing :)

[Niklas Therning]

I'm currently looking at fixing the ICU mess in RoboVM. One thing I'd like to do before I start to mess with ICU is to sync with the latest Android sources which is android-4.4.3_r1. I'm afraid I'm not prepared yet to base RoboVM on Avian's libcore6, just wanted to give you a heads up. I suspect the diffs between android-4.4_r0.7 and android-4.4.3_r1 are less than between the old version RoboVM is currently based on (android-4.1.1_r3) so this is probably a good thing for you, right?

--
You received this message because you are subscribed to the Google Groups "RoboVM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robovm+un...@googlegroups.com.

[Carl Perfect]

On Monday, June 16, 2014 9:02:37 AM UTC+2, Niklas Therning wrote:

 I'm afraid I'm not prepared yet to base RoboVM on Avian's libcore6,

Well, to be fair the Robovm on Windows task has not produced
any workable results yet (despite all the progress we made).

So my heart is not too much bleeding at the news.
(but I hope that at some deep down level, you are feeling
very bad about it :) )

On a serious note, even if working off libcore64 is a huge step
forward as windows build is concerned, there is still one thing
that concerns me: the changes that you made in luni.
Should they be integrated to libcore64 to have one super libcore64
that supports 64bits and a great array of platforms?
Or should they stay in Robovm?
Would those changes break avian?

Though one.

 

so this is probably a good thing for you, right?

Indeed. Furthermore, I have the feeling that this is a good thing
for all Robovm users.

[Niklas Therning]

Actually, the 4.4.3 release I've been looking at for the past couple of days seems to be fully 64-bit capable so far. No more ints passed around between Java and native code to hold pointers, everything is long as far as I have seen. No Windows support yet though. :-)

I don't know about the RoboVM-specific changes. I think we should keep those to ourselves.

BTW, in a very distant future I would like to switch to OpenJDK. OpenJDK obviously supports Windows out of the box. That would make all your hard work be in vain. I hope you're ok with that. :-)

[Pablo Guerrero]

Hi Niklas,

The 64bit changes on libcore64 were integrated upstream (+ more
changes directly done upstream), so it's normal that current libcore
works for 64bits.

In my opinion, the android advantages over OpenJDK are the license,
that is much more permissive, and the classpath size.

The size issue can be solved on 1.8 using the compact1 profile, or
using proguard by default, but not the licensing one.

Anyway, I still cannot figure out what are the exact limitations of
GPL + Classpath exception in practice, as I'm not a lawyer.

Cheers,
Pablo

> You received this message because you are subscribed to a topic in the
> Google Groups "RoboVM" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/robovm/GGTubkAT2vg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Carl Perfect]

On Tuesday, June 17, 2014 5:22:59 PM UTC+2, Niklas Therning wrote:

That would make all your hard work be in vain. I hope you're ok with that. :-)

Are you implying that dealing with linking errors, cabalistic #define and deciphering
makefiles is not fun?
That all this is not a reward in itself?

[Niklas Therning]

Correction, the Android fixes to make it work on windows would be in vain. There's a lot more to it that you're already aware of course. :-) And as Pablo pointed out there are things that need to be clarified when it comes to the licensing of OpenJDK.

[Jan Blok]

Any idea's about timeline for integration of 4.4.3 release and overhaul of ICU?

Adding the windows compatibiliy diff as patch on top of these 2 major changes is considered easy... 

[Niklas Therning]

I have 4.4.3 working in a local branch. I'm aiming at pushing it today or tomorrow. The ICU situation is still horrible I'm afraid but it only affects iOS/OSX. On other platforms one has always had to provide a custom ICU data file. I will see if it is possible to create a minimal ICU data file (max a few hundred kB) and bundle that with RoboVM. Then let the user override it with a custom file if more locales are needed. This is the most stable solution I can think of but it sucks that we cannot utilize that ICU data file which is already available on iOS/OSX. Droid 4.4.3 uses ICU 51 while the iOS 8 data file has been built with ICU 53 so they are not compatible.

[Pablo Guerrero]

Hi Niklas,

I did that for Avian, and it worked perfectly.

https://groups.google.com/d/msg/avian/guZyk5Rmw5Y/JEDPIg3yAekJ

I'm sure you know about it, but you can use
http://apps.icu-project.org/datacustom/ICUData51.html with just base
data. It did work for me. I also used a basic english only version
(2.7MB) as the default, that you can later override with env
variables.

Cheers,
Pablo

> You received this message because you are subscribed to a topic in the
> Google Groups "RoboVM" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/robovm/GGTubkAT2vg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Niklas Therning]

That's what I want to do, thanks! Yes, I'm aware of the data customizer. However, 2.7 MB is still too much. Did you skip all charset mapping tables? One idea I have is to skip all that charset data and instead build a CharsetProvider which uses iconv. The Android Charset class supports custom CharsetProviders and iconv is available on iOS/OSX and most other platforms. Just need to investigate exactly what iconv can do and if it supports everything required by the CharsetProvider API.

BTW, I've patched the Android JNI code in similar ways to work on Darwin. We should sync this somehow (and try to get it into upstream of course). It's a pain to maintain. I have most of the libcore tests running successfully. Do you run those tests?

[Pablo Guerrero]

Yes, 2.7 is still a lot, but it compresses pretty well. I'm sure this
can be improved easily, as I didn't put much effort on the dataset
itself.

For the changes on the JNI, I didn't run the tests, and AFAIK they are
not run on Avian on Android, but I worked on this long time ago, and
now there is more people interested on Avian on Android, so maybe they
have run the tests.

[Niklas Therning]

I managed to produce a very small ICU data file by hacking the ICU make files a bit. The result is here: https://github.com/robovm/icu4c-51_2-data.

This data file is only about 250 kB. It only contains English and doesn't include and charset conversion tables, no break iterators and no collators. I also replaced the root locale with the English locale and deleted everything from the English locale. That way any locale one tries to use always falls back on the English symbols and names. So if you format a Date using "de" like this:

  DateFormat.getDateTimeInstance(DateFormat.LONG, DateFormat.LONG, new Locale("de")).format(new Date())

It returns the exact same string as if using English instead of German and doesn't throw an exception.

Furthermore the data file is gzipped down to about 98 kB and embedded into the executable. The user can provide a custom icudt51l.dat file if desired by placing it into the root of the app. If no such custom data file is found RoboVM will fall back to the embedded minimal ICU data file.

Feel free to use the same approach with Avian.

[Pablo Guerrero]

That's great, I'll have a look.

Cheers,
Pablo