Android build with prebuilt libcrypto.so

[Android Jeff]

Hi there,

I am not good at makefiles, I followed link below to build SQLCipher from source for Android, all are working fine:

http://sqlcipher.net/sqlcipher-for-android/

And I also noticed how to use an prebuilt libcrypto.so from here:

http://sqlcipher.net/introduction/

what I want to do is building SQLCipher from source for Android, but need to link against a prebuilt libcrypto.so. My lib name is "libcrypto.so.1.0.0", and the soname inside the lib is also   "libcrypto.so.1.0.0". My question is how should I make it work, should following steps be correct (I have done "make init"):

copy both of libcrypto.so.1.0.0 and its symbolic link named libcrypto.so to  android-database-sqlcipher/external/android-libs/

cd android-database-sqlcipher

./configure --enable-tempstore=yes CFLAGS="-DSQLITE_HAS_CODEC" LDFLAGS="-lcrypto"

make

Where should I put all header files of the prebuilt libcrypto? My libcrypto.so.1.0.0 and all header files are in attachment.

Thanks

[Nick Parker]

Hi Jeff,

We are dynamically linking to libcrypto during the build, so we ultimately end up using the version of libcrypto provided on the Android device.  Early on we tried to modify the LD_PATH such that we could replace the library used from the system library without success.  To use your version of libcrypto, you would need to perform a static build which could be included in the binary.  As an example, we do this with the ICU project.  The modifications we made to the build are highlighted here [1] and becomes included here [2].  You would want to do something similar for libcrypto.

1.  https://github.com/sqlcipher/android-database-sqlcipher/blob/master/external/Android.mk#L107-226
2.  https://github.com/sqlcipher/android-database-sqlcipher/blob/master/external/Android.mk#L76

-- 
Nick Parker

[Android Jeff]

Thank you Nick for your quick and authoritative response.

Now I understood why "libcrypto.so" was not required by the code that uses SQLCipher. The one under Android system lib folder is being used.

Actually the libcrypto.so.1.0.0 that I want to use is a prebuilt FIPS capable OpenSSL shared library, which is required. As you said I may build SQLCipher link against libcrypto.a, but as per the building process of FIPS module, I have to modify SQLCipher makefile to use fipsld to replace gcc linker, I have tried this on other project, to me it was like a nightmare. So I prefer to use the prebuilt libcrypto.so. Do you think following is doable:

1. following the doc do a full SQLCipher for android build.

2. remove the Android.mk under android-database-sqlcipher/external/openssl, so that the build of openssl won't happen.

3. under android-database-sqlcipher/external/android-libs, make a symbolic link named "libcrypto.so" to replace the solid libcrypto.so, the link points to the real "libcrypto.so.1.0.0", and make them have a newer timestamp.

4. replace android-database-sqlcipher/external/openssl/include with my header files.

5. then run "make" again. So during this make, openssl won't be built, but SQLCipher will link against newly added "libcrypto.so.1.0.0". In my java code, before loading SQLCipher's libxxxx.sos, load "libcrypto.so.1.0.0".

Should this be possible?

Thank you very much Nick

[Nick Parker]

Hi Jeffrey,

While you would be able to build and link against your version of libcrypto.so, assuming it was built for the ARM architecture, it may not be made available at runtime on the Android devices.  Previous attempts to modify the LD_LIBRARY_PATH to provide a system library override were unsuccessful.  The calls to System.loadLibrary within SQLCipher for Android is only providing a JNI bridge allowing Java to call into native C functions.  A static build of libcrypto is likely the solution in this case, although I believe this would require a separate FIPS validation process due to the change in the build process.

-- 
Nick Parker

[Android Jeff]

Thank you Nick for your reply. Finally I did it. I would not be able to do it without your help. Here are my steps:

1. follow the “SQLCipher for Android Build Tutorial” in following link to do a full SQLCipher build for android. http://sqlcipher.net/sqlcipher-for-android/

2. remove the Android.mk under android-database-sqlcipher/external/openssl, so that the build of openssl won't happen.

3. under android-database-sqlcipher/external/android-libs, remove the existing libcrypto.so, create a symbolic link named "libcrypto.so" that points to the solid library file "libcrypto.so.1.0.0".
4. replace android-database-sqlcipher/external/openssl/include with the header file folder from FIPS OpenSSL build, by default it’s at /usr/local/openssl/include on Linux and Mac OS X.
5. do search under android-database-sqlcipher and remove all copied of libdatabase_sqlcipher.so and libsqlcipher_android.so. These two libs rely on libcrypto.so, check Android.mk to know this.
6. then run "make" again. gcc will complain a duplicated java class, haven't got time to check why it happens, but the two removed .so are created, they are exactly what we need. Reuse all other .so and .jar files.
[dx] UNEXPECTED TOP-LEVEL EXCEPTION:
[dx] java.lang.IllegalArgumentException: already added: Lexample/EventDataSQLHelper;
so during this make, openssl won't be built, but SQLCipher will link against newly added "libcrypto.so.1.0.0". In java code, before loading SQLCipher's libxxxx.sos, do System.load("libcrypto.so.1.0.0"). All are working well.

Why it works in my case: my libcrypto.so has a different gcc soname from the one comes in SQLCipher package. So the actual lib name that libdatabase_sqlcipher.so links against is "libcrypto.so.1.0.0" instead of "libcrypto.so", so there is no confliction. 

Thank you again Nick for your help.

[Nick Parker]

Hi Jeffrey,

That's great new to hear about your build.  How are you verifying that the FIPS version is actually being used in your current scenario, given that SQLCipher would operate identically either way?

-- 
Nick Parker

[Jeffrey Cui]

Sorry didn't reply it to the group, reply to all for this time.

Hi Nick,

1. Because libdatabase_sqlcipher.so and libsqlcipher_android.so are linked against "libcrypto.so.1.0.0" (there was no other libcrypto.so files under android-database-sqlcipher when building them, if I remove the only libcrypto.so.1.0.0, build will fail) so I know  libdatabase_sqlcipher.so and libsqlcipher_android.so are linking against libcryptoso.1.0.0.

2. In Java, if libcrypto.so.1.0.0 is not loaded ahead of loading  libdatabase_sqlcipher.so and libsqlcipher_android.so, I failed in System.loadLibrary("libdatabase_sqlcipher.so") with error "libcryptoso.1.0.0 can not be found".

So I am sure the replaced lib crypto.so.1.0.0 is being used.

Thanks

Jeffrey

[Kondlada]

Hi Jeffry,

                  Hope you doing well. I am trying to use another authenticated openssl version , your way of approach may really help me out. Can you please take some time and explain in steps how you have actually changed the symbolic link against your own libcrypto.so.1.0.0  and the fifth step where you actually removed some cypher files. I will be really very thankful for your reply.

Thanks & Regards,
karthik

[Jeffrey Cui]

no problem, i will get back to you when i am in office today.
thanks
jeffrey

[Kondlada]

Hi Nick,

              I am able to follow jeffrey steps and quiet interestingly able to build sqlcipher. I am wondering why should we built different binaries for x86 as the same armeabi build would work seamlessly.

             Can you please help me to understand for new build support for x86 and surprisingly it fails if we follow the same steps as Jeffrey explained for new build.


Thanks & Regards,
kondlada

[Stephen Lombardo]

Hello Kondlada,

The reason there was change in the build process for x86 is that ARM and x86 are different processor architectures. You can't take native libraries, like SQLCipher, for the ARM architecture and use them under x86. Thus, in order to support newer x86 devices and native emulator, minor changes to the build scripts were required. However, these changes only facilitated the creation of separate copies of the native libraries by architecture.

That said, I don't think we have enough information on what you are attempting to confidently say what is not working. To hazard a guess, it seems most likely that your OpenSSL build is causing problems, for example, trying to use an ARM OpenSSL lib with the x86 build or vice versa.

If you'd like to continue this thread I'd suggest you post at gist.github.com with a complete log of every step of your build process, including your custom OpenSSL, source code checkouts, etc, under 2.0.8, and then again with the latest version 2.1.1 so we can see the differences, what errors you are receiving etc.

Cheers,
Stephen

> > *copy both of libcrypto.so.1.0.0 and its symbolic link named
> > libcrypto.so to android-database-sqlcipher/external/android-libs/*
> >
> > *cd android-database-sqlcipher*
> >
> > *./configure --enable-tempstore=yes CFLAGS="-DSQLITE_HAS_CODEC" LDFLAGS=
> > "-lcrypto"*
> >
> > *make*

[Lamchith M C]

Are these steps given by Jef to make FIPS compliant sql cypher still valid ? I am trying the same and i did not find Android.mk file under android-database-sqlcipher/external/openssl folder

[Nick Parker]

Hi,

At the end of August we released SQLCipher for Android 2.2.2 which
modified the build process to statically link OpenSSL into the native
library to address the Android security issue. More information about
the change can be found here:

http://sqlcipher.net/blog/2013/8/28/sqlcipher-for-android-222-release.html

On 10/16/13 2:01 AM, Lamchith M C wrote:
> Are these steps given by Jef to make FIPS compliant sql cypher still
> valid ? I am trying the same and i did not find Android.mk file
> under android-database-sqlcipher/external/openssl folder
>
> On Monday, January 7, 2013 8:52:33 PM UTC+5:30, Stephen Lombardo wrote:
>
> Hello Kondlada,
>
> The reason there was change in the build process for x86 is that ARM
> and x86 are different processor architectures. You can't take native
> libraries, like SQLCipher, for the ARM architecture and use them
> under x86. Thus, in order to support newer x86 devices and native
> emulator, minor changes to the build scripts were required. However,
> these changes only facilitated the creation of separate copies of
> the native libraries by architecture.
>
> That said, I don't think we have enough information on what you are
> attempting to confidently say what is not working. To hazard a
> guess, it seems most likely that your OpenSSL build is causing
> problems, for example, trying to use an ARM OpenSSL lib with the x86
> build or vice versa.
>
> If you'd like to continue this thread I'd suggest you post at

> gist.github.com <http://gist.github.com> with a complete log of

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "SQLCipher Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlcipher+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Nick Parker

[Lamchith M C]

Nick , 

As you have mentioned here how to set this ldlibs of mk ?
http://stackoverflow.com/questions/12596901/how-to-build-fips-compliant-sqlcipher-to-call-fips-mode-set1

I have followed the steps given and getting undefined reference to `FIPS_mode_set'

[Lamchith M C]

I have added following and it seems to be working LOCAL_LDLIBS += -L../external/openssl/include/openssl/fips

and $(EXTERNAL_PATH)/openssl/include/openssl/fips \ to LOCAL_C_INCLUDES +

is this correct ?

[Lamchith M C]

On test android app i am getting , while using the generated so files , along with jar file from the site

10-17 14:07:31.200: I/Database(2339): JNI_OnLoad called

10-17 14:07:31.200: I/Database(2339): JNI_OnLoad register methods 

10-17 14:07:31.200: E/dalvikvm(2339): ERROR: couldn't find native method

10-17 14:07:31.200: E/dalvikvm(2339): Requested: Lnet/sqlcipher/database/SQLiteDatabase;.fips_set_mode:(I)V

10-17 14:07:31.200: E/JNIHelp(2339): RegisterNatives failed for 'net/sqlcipher/database/SQLiteDatabase', aborting

10-17 14:07:31.200: A/libc(2339): Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1), thread 2339 (pppforsqlcipher)

Any help for this ?

[Jeffrey Cui]

Your fips openssl build has problem, follow the answer of brewphone in following link
http://stackoverflow.com/questions/11091905/android-build-openssl-fips-2-0
to generate the libcrypto.a.
With current sqlcipher release u need .a instead of .so.


Lamchith M C <lamc...@gmail.com> wrote:

You received this message because you are subscribed to a topic in the Google Groups "SQLCipher Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlcipher/Ldy_vJgrp9k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlcipher+...@googlegroups.com.

[Lamchith M C]

Got it , but the above mentioned error was after i tried with libcrypto.a file only. Then i generated the sqlcipher.jar (i had issues to generating sqlcipher.jar )  file and the error mentioned below got resolved, but FIPS_mode_set(enable) was returning 0 value , which means the FIPS mode were not being set.

[Lamchith M C]

Following is the current log, FIPS mode is not being set

10-17 18:50:12.410: D/dalvikvm(6902): Added shared lib /data/data/com.example.testapppforsqlcipher/lib/libstlport_shared.so 0x411f7908

10-17 18:50:12.420: D/dalvikvm(6902): Added shared lib /data/data/com.example.testapppforsqlcipher/lib/libsqlcipher_android.so 0x411f7908

10-17 18:50:12.420: D/dalvikvm(6902): Added shared lib /data/data/com.example.testapppforsqlcipher/lib/libdatabase_sqlcipher.so 0x411f7908

10-17 18:50:12.420: D/Database(6902): Enable FIPS mode ...

10-17 18:50:12.580: D/Database(6902): error is 2d06b06f.

10-17 18:50:12.580: D/Database(6902): Failed to enable FIPS mode.

2d06b06f is FIPS_check_incore_fingerprint:fingerprint does not match after googling

[Lamchith M C]

On giving it another try , using Linking with the Static Archive  section on http://wiki.openssl.org/index.php/FIPS_Library_and_Android#Linking_with_the_Static_Archive

make is failing. Any suggestion?

i have tried with ./configure --build=x86_64-unknown-linux-gnu --enable-tempstore=yes CFLAGS="-DSQLITE_HAS_CODEC"  with same result. Is not possible to compile with fipsld.

[Nick Parker]

Hi,

Please post the error message you receive from make, that will be
helpful in diagnosing the problem.

Also, you may consider reviewing the build script we use for preparing
static OpenSSL libraries here:

https://github.com/sqlcipher/android-database-sqlcipher/blob/master/build-openssl-libraries.sh

On 10/22/13 11:46 AM, Lamchith M C wrote:
> On giving it another try , using _*Linking with the Static Archive
> *_ section

> http://stackoverflow.com/questions/11091905/android-build-openssl-fips-2-0 <http://stackoverflow.com/questions/11091905/android-build-openssl-fips-2-0>

> <https://groups.google.com/groups/opt_out>.

>
> --
> Nick Parker
>
> --
>
> ---
> You received this message because you are subscribed to a
> topic in the Google Groups "SQLCipher Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sqlcipher/Ldy_vJgrp9k/unsubscribe

> <https://groups.google.com/d/topic/sqlcipher/Ldy_vJgrp9k/unsubscribe>.

> To unsubscribe from this group and all its topics, send an
> email to sqlcipher+...@googlegroups.com.
> For more options, visit
> https://groups.google.com/groups/opt_out

> <https://groups.google.com/groups/opt_out>.

[Lamchith M C]

Attached the make error. Previously attached config.log was more descriptive. These happens after setting 

$ export CC=`find /usr/local/ssl/$ANDROID_API -name fipsld`
$ echo $CC 
/usr/local/ssl/android-14/bin/fipsld
$ export FIPSLD_CC="$ANDROID_TOOLCHAIN/arm-linux-androideabi-gcc" 
$ echo $FIPSLD_CC 
/opt/android-ndk-r8e/toolchains/arm-linux-androideabi-4.6/prebuilt/linux-x86/bin/arm-linux-androideabi-gcc

 as per trying the open ssl wiki.

[Nick Parker]

Hi,

I believe the issue you are seeing is due to redefining CC when building
FIPS, however it is being picked up when the ./configure step is run to
prepare the SQLCipher source in SQLCipher for Android. Can you reset CC
before attempting to build SQLCipher for Android?

> > an email to sqlcipher+...@googlegroups.com <javascript:>.

> > For more options, visit https://groups.google.com/groups/opt_out
> <https://groups.google.com/groups/opt_out>.
>
> --
> Nick Parker
>
> --
>
> ---

[scott coleman]

I've got a similar issue,

I'm successfully compiling my own fipscanister.o and libcrypto.a (in their own directories)
as in : http://stackoverflow.com/questions/11091905/android-build-openssl-fips-2-0

Then I'm copying libcrypto.a into the SQLCipher library at /external/android-libs/armeabi

I'm trying to update SQLiteDatabase.java and net_sqlcipher_database_SQLiteDatabase.cpp to add fips_set_mode() as described in
http://stackoverflow.com/questions/12596901/how-to-build-fips-compliant-sqlcipher-to-call-fips-mode-set1

I get a fatal error: openssl/fips.h: no such file or directory

There is no fips.h in the SQLCipher distribution

[Nick Parker]

Hello Scott,

SQLCipher does not include the FIPS Object Module v2.0 source, it uses
the standard OpenSSL source tree as a submodule. Building with FIPS
would require further customizations to the build process.

> <https://groups.google.com/groups/opt_out>.

>
> --
> Nick Parker
>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "SQLCipher Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlcipher+...@googlegroups.com

> <mailto:sqlcipher+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Nick Parker

[scott coleman]

OK, thanks anyway.

Scott..