Building SQLCipher from sources for Android using LibTomCrypt

[M. Mosed]

Hi, 

What I am trying to do is building SQLCipher from sources for Android, but need to link against LibTomCrypt library(libtomcrypt.so or libtomcrypt.a) instead of Openssl. 

I need to use the generated library (libsqlite3.so or as it is called libsqlcihper.so now) from command-line to test different encryption algorithms programmatically on my Galaxy tab device. 

No application integration is required for now, just I need the libsqlite3.so with Sqlcipher encryption support using LibTomCrypt.

My requirements: SQLCipher community edition source code for Android, Android ICS 4.0.4, SQLite 3.7.4 Amalgamation, 

and LibTomCrypt 1.17 with full featured algorithms and modes.

How can I achieve that and where to find the proper source code for that? Does the Sqlcipher community edition provide these sources?

What about if I compile a stock SQLite 3.7.4 amalgamation file (sqlite3.c) to use SQLCipher and link to libtomcrypt.so? 

Can that work and provide an  libsqlite3.so with Libtomcrypt encryption support?

All hints are appreciated,
Thanks, 
Mosed

[Nick Parker]

Hello Mosed,

SQLCipher for Android [1] is configured to be built with OpenSSL
statically linked. You can certainly use libtomcrypt in place of
OpenSSL, however there would be some work required to complete that.
Specifically you would need to build static libtomcrypt libraries for
required platform you desire to target, armeabi likely being the
minimum. You would also need to change the NDK build to link against
those instead of OpenSSL as well as have SQLCipher configured with
-DSQLCIPHER_CRYPTO_LIBTOMCRYPT. If I may ask, is there a reason you are
interested in using libtomcrypt over OpenSSL?

[1] https://github.com/sqlcipher/android-database-sqlcipher/

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "SQLCipher Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlcipher+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Nick Parker

[M. Mosed]

Hello Nick,
Thanks for the info and sorry for late reply since I was out for a few days. 
I will give it a try to compile SQLCipher with the help of your hints then I will keep you informed.

The reason of using Libtomcrypt over OpenSSL is that I am experimenting on some algorithms that 
are not available in OpenSSL (such as Twofish, RC6, and others) to study their performance aspects 
in mobile devices. 

“Specifically you would need to build static libtomcrypt libraries” 
What about building and linking against dynamic Libtomcrypt (libtomcrypt.so)? 
Will that work, since I already compiled dynamic Libtomcrypt for Android but still have 
problems with static one.

When running make init command I get this error:

android-database-sqlcipher$ make init
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
git submodule update --init
fatal: Not a git repository (or any of the parent directories): .git
make: *** [init] Error 128

Any hint about this error?

Thanks, 
Mosed

[Nick Parker]

Hello Mosed,

You could dynamically link libtomcrypt, however that would require
further modifications to the build process. It does not appear that you
have cloned the android-database-sqlcipher project. Could you try this
instead:

> git clone https://github.com/sqlcipher/android-database-sqlcipher.git
> make init
> make

[M. Mosed]

Hi Nick,
I followed the points you provided to build Sqlcipher with Libtomcrypt as follow:

• I compiled Libtomcrypt statically for Android (armeabi) 

• I modified the ndk build to link with Libtomcrypt instead of Openssl by modifying the Android.mk file

• I have SQLCipher configured with -DSQLCIPHER_CRYPTO_LIBTOMCRYPT and also with the help of --with-crypto-lib=libtomcrypt

All seems to be ok and linking to Libtomcrypt is progressing instead of Openssl but some errors occurred: 

/android-database-sqlcipher/obj/local/armeabi/objs/sqlcipher/sqlcipher/sqlite3.o: In function `sqlcipher_ltc_add_random':

/android-database-sqlcipher/external/sqlcipher/sqlite3.c:15420: undefined reference to `fortuna_add_entropy'

/android-database-sqlcipher/external/sqlcipher/sqlite3.c:15420: undefined reference to `fortuna_add_entropy'

/android-database-sqlcipher/external/sqlcipher/sqlite3.c:15429: undefined reference to `fortuna_ready'

/android-database-sqlcipher/obj/local/armeabi/objs/sqlcipher/sqlcipher/sqlite3.o: In function `sqlcipher_ltc_random':

/android-database-sqlcipher/external/sqlcipher/sqlite3.c:15498: undefined reference to `fortuna_read'

/android-database-sqlcipher/obj/local/armeabi/objs/sqlcipher/sqlcipher/sqlite3.o: In function `sqlcipher_ltc_deactivate':

/android-database-sqlcipher/external/sqlcipher/sqlite3.c:15474: undefined reference to `fortuna_done'

/android-database-sqlcipher/obj/local/armeabi/objs/sqlcipher/sqlcipher/sqlite3.o: In function `sqlcipher_ltc_activate':

/android-database-sqlcipher/external/sqlcipher/sqlite3.c:15449: undefined reference to `fortuna_start'

/android-database-sqlcipher/external/sqlcipher/sqlite3.c:15466: undefined reference to `fortuna_desc'

collect2: ld returned 1 exit status

make[1]: *** [/android-database-sqlcipher/obj/local/armeabi/libsqlcipher.so] Error 1

make[1]: Leaving directory `/android-database-sqlcipher/external'

make: *** [build-external] Error 2

What do you think the reasons behind these errors?
They looks related to Libtomcrypt but how to remove them and continue to build 
libsqlcipher (.so or .a) which is the library I need.

Thank you.
Mosed

[Nick Parker]

Hello Mosed,

It sounds as if the linker can not find your static build of
libtomcrypt. You will likely need to adjust the Android.mk file,
specifically this [1] section to properly reference your libtomcrypt build.

[1]
https://github.com/sqlcipher/android-database-sqlcipher/blob/master/external/Android.mk#L54-L56

On 3/10/14, 9:47 AM, M. Mosed wrote:
> Hi Nick,
> I followed the points you provided to build Sqlcipher with Libtomcrypt as follow:
>

> *

>
> I compiled Libtomcrypt statically for Android (armeabi)
>

> *

>
> I modified the ndk build to link with Libtomcrypt instead of Openssl by modifying the Android.mk file
>

> *

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "SQLCipher Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlcipher+...@googlegroups.com

> <mailto:sqlcipher+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Nick Parker

[M. Mosed]

Hello Nick, 

I did all the necessary changes to Android.mk to link to Libtomcrypt based on the link you 

provided but I still have same "fortuna_" errors as before.

The modifications to Android.mk file are attached.

The important point is that when I try to compile libsqlcipher as static library (libsqlcipher.a) 

it compiles successfully to produce the static library libsqlcipher.a.

But when I change the Android.mk file to build same library as shared (libsqlcipher.so), it will 

give the mentioned "foruna_" errors. Strange! What do you think makes the difference?

When building with Openssl before, it was compiling successfully for both static and shared without any errors.

Any suggestions?

Thank you,

Mosed

[Nick Parker]

Hello Mosed,

Based on your errors you describe, I would gather there is a problem
with the static libtomcrypt library you have built, or an issue relating
to how you are attempting to link that library during the build of
SQLCipher using it. You may want to verify you have properly configured
the header search path as well.

We generally do not provide free support for custom builds of SQLCipher
via the mailing list. If this is a commercial endeavor, please consider
contacting us off list at sup...@zetetic.net to discuss our commercial
support options which include custom SQLCipher builds.

> <https://github.com/sqlcipher/android-database-sqlcipher/blob/master/external/Android.mk#L54-L56>

[M. Mosed]

Hello Nick, 

Thank you very much for your time and important helping points.

Using you build problem hint, I succeeded to build both the static and 

shared Sqlcipher with Libtomcrypt support since there was a file missing (fortuna.c) during the

libtomcrypt static compile. Now it is compiling properly. 

I will start experimenting on the new library on my device.

Actually my work is not a commercial one, but an academic try to test more encryption algorithms with

Sqlcipher.

Thank you again,

Mosed

[M. Mosed]

Hello Nick, 

It looks that Sqlcipher build with Libtomcrypt does not support PRAGMA cipher

to switch the cipher/length/mode combination. 

I tried it but it always encrypt and decrypt ignoring all ciphers/modes settings I used in the pragma.

I am trying to use different ciphers from Libtomcrypt but it looks that this flexibility is 

not yet supported by Sqlcipher. Am I missing something?

Is there any way to change the cipher/length/mode either by PRAGMA way or by 

rebuilding the library after modifying some code (such as crypto_libtomcrypt.c)?

Any hints how to do that? 

Thank you,

Mosed

[Nick Parker]

Hello Mosed,

Yes, the ltc provider only supports AES at the moment. This [1] would
require changes to allow for adjusting the cipher.

[1]
https://github.com/sqlcipher/sqlcipher/blob/master/src/crypto_libtomcrypt.c

[M. Mosed]

Hello Nick,

Thank you man for your super fast answer.

I will give it a try and keep you posted.

Thanks again,

Mosed 

[M. Mosed]

Hi Nick,

SQLCipher with different ciphers and modes is working fine now. 

To confirm the type of cipher used, I added a simple printf in the crypto_libtomcrypt.c:

printf("%s\n",cipher_descriptor[cipher_idx].name);

It shows the cipher name properly but that is printed on screen multiple times instead of one time. 

I thought that is due to some threading/mutexing. My work now need single-threading mode. 

To stop any threading and go in single-thread mode, I tried to use many options:

• Rebuild libsqlcipher with compile option -DSQLITE_THREADSAFE=0

• Use the runtime option sqlite3_config(SQLITE_CONFIG_SINGLETHREAD); inside application code

• Compile with the following option: -DSQLITE_MUTEX_APPDEF=1

• Also this compile option SQLCIPHER_LTC_NO_MUTEX_RAND

All that has no benefit, still the printf message prints repeatedly (around 4 times with each db insert) 

which shows that some threading is still lurking somewhere.

My question is that does Sqlcipher has any built-in threading? If yes, how to stop it to go in single-thread mode?

If no, what do think is the reason behind this multi message printing. I would like just to confirm

that no thread are active in my current serial tests.

Thank you,

Mosed

[Nick Parker]

Hello Mosed,

Where did you add your printf statement? There are multiple functions
within a crypto provider that get executed multiple times as the
provider is used when operating on individual pages within the database.
You can use -DSQLITE_THREADSAFE=0 to disable all mutexes within SQLite,
making it unsafe to use more than one thread at a time.

On 4/9/14, 7:01 AM, M. Mosed wrote:
> Hi Nick,
>
> SQLCipher with different ciphers and modes is working fine now.
>
> To confirm the type of cipher used, I added a simple printf in the
> crypto_libtomcrypt.c:
>
> printf("%s\n",cipher_descriptor[cipher_idx].name);
>
>
> It shows the cipher name properly but that is printed on screen multiple
> times instead of one time.
>
> I thought that is due to some threading/mutexing. My work now need
> single-threading mode.
>
> To stop any threading and go in single-thread mode, I tried to use many
> options:
>
>

> *
>
> Rebuild libsqlcipher with compile option /-DSQLITE_THREADSAFE=0 /
>
> *
>
> Use the runtime option
> /sqlite3_config(SQLITE_CONFIG_SINGLETHREAD);/inside application code
>
> *
>
> Compile with the following option: /-DSQLITE_MUTEX_APPDEF=1/
>
> *
>
> Also this compile option /SQLCIPHER_LTC_NO_MUTEX_RAND/

>
>
> All that has no benefit, still the printf message prints repeatedly
> (around 4 times with each db insert)
>
> which shows that some threading is still lurking somewhere.
>
>
> My question is that does Sqlcipher has any built-in threading? If yes,
> how to stop it to go in single-thread mode?
>
> If no, what do think is the reason behind this multi message printing. I
> would like just to confirm
>
> that no thread are active in my current serial tests.
>
>
> Thank you,
>
> Mosed
>

[M. Mosed]

Hello Nick, 

I placed the printf statement in sqlcipher_ltc_cipher() function (inside crypto_libtomcrypt.c)

since the real encryption starts there. 

So do you suggest to add it in any other place which can show threading activity (in case I activated threading feature) 

while at same time avoid pager multiple calls to crypto functions.

Let me modify my question, is there any other way/tool to monitor threads activity in 

such kind of scenario (SQLcipher + Libtomcrypt + pthreads/OpenMP + Android device) better 

than using simple printf. There are many ways but they might not fit to this scenario.

Thank you,

Mosed

[Nick Parker]

Hello Mosed,

The threading modes supported within SQLite focus on the level of
isolation provided to protect internal resources within the library.
That said, you might find it beneficial to use a platform specific tool
to isolate a given threading scenario within your application. For
example, you mention Android below, you can use the DDMS monitor to
track object allocation by threads. Outside of that, I am not certain I
understand the problem you are trying to solve. Please feel free to
clarify further if you have unanswered questions.

[M. Mosed]

Hi Nick, 

I am trying now to build SQLCipher to work with multi-threading support using different 

parallel APIs such as OpenMP or Pthreads. The compile is successful and it shows, when debug,

that many threads are running during database encryption.  It is normal that two threads should give 

better performance than one thread but problem with me is that the performance with multiple

threads is not improved when compared to single-thread!

What do you think might be the reason behind that in SQLCipher/Sqlite? Can database locking be 

the cause to affect threads execution in someway? Or might be another reason behind.

Thanks, 

Mosed

[Nick Parker]

Hello Mosed,

I think it may be difficult to explain the behavior you are seeing without further detail in the scenario you are testing.  Could you share more information regarding your specific test scenario you are attempting?  Thanks!

--

---
You received this message because you are subscribed to the Google Groups "SQLCipher Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlcipher+...@googlegroups.com.

[M. Mosed]

Hello Nick,

Firstly, I would like to thank you for your continuous help in the last few months to work 

and build SQLCipher which greatly assisted me to proceed with my academic work.

Sorry for late response since I was working with other projects for a while.

In your last post you asked me for more details about the scenario I am testing. Here they are.

My main idea is to test the performance of SQLCipher encryption when using one core (one thread) and two 

cores or more (two threads or more). I use OpenMP to manage parallel threads. I do the tests on a real device.

What I did is that I compiled SQLCipher for Android with LibTomCrypt and I came up with libsqlcipher.so library 

which I will call from my native code through command line (I sent to your email a small C code with details).

I added OpenMP to the environment (with the help of NDK) to provide parallel processing functionality. 

Using OpenMP, I placed a parallel pragma inside the main encryption code that is available in the mode of 

operation (inside LibTomCrypt). This pragma will create two threads and divide the encryption work between them.

I did that to measure the performance of an SQLCipher database when using one thread (one core) and two threads (two cores). 

But unfortunately I was getting no improvement when using two threads. That looks strange to me since two threads 

should be much faster than one!

When I worked with the same scenario inside an Android device without SQLCipher databases, I got almost double of 

the speedup (less encryption time when I used two threads/cores). But when I tried to implement it with SQLCipher, I can not see 

any improvement in performance. I don’t know why? Any advice?

My questions:

Is it possible, theoretically, to get significant improvement in the db encryption time when using two threads (two cores) as compared to one thread (one core)?

What about practically? How to achieve that and get good speedup since my scenario is not able to do that? What is wrong with it?

What might prevent the two threads to achieve better than one thread?

Am I facing any kind of database file locks? Or other reasons?

Any hints from you will be highly helping and appreciated.

If you need any further details, please inform me.

Thank you,

Mosed

[Nick Parker]

Hello Mosed,

My apologizes for just responding. I think you would need to evaluate
the type of cipher block mode being used when considering multiple
threads with regard to performance improvements. For example, by
default SQLCipher uses AES in CBC mode, in which the encryption is not
parallelizable, however decryption is. Using ECB or CTR mode would be
options to look into and compare against.

On 10/1/14 11:17 PM, M. Mosed wrote:
> Hello Nick,
>
> Firstly, I would like to thank you for your continuous help in the last
> few months to work
>
> and build SQLCipher which greatly assisted me to proceed with my
> academic work.
>
> Sorry for late response since I was working with other projects for a
> while.
>
> In your last post you asked me for more details about the scenario I am
> testing. Here they are.
>

> My main ideais to test the performance of SQLCipher encryption when

> using one core (one thread) and two
>
> cores or more (two threads or more). I use OpenMP to manage parallel
> threads. I do the tests on a real device.
>
>
>

> What I didis that I compiled SQLCipher for Android with LibTomCrypt and

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "SQLCipher Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlcipher+...@googlegroups.com

> <mailto:sqlcipher+...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--
Nick Parker

[M. Mosed]

Hi Nick,

Thank you for your hint. I will play with different modes of operation and keep you posted.

What about using XTS mode? As I know it is a parallelizable mode.

Can it achieve good performance in both encryption and decryption when it is used in SQLCipher?

Thanks, 

Mosed

[Nick Parker]

Hello Mosed,

We look forward to hearing your results when you complete your
investigation.

> > an email to sqlcipher+...@googlegroups.com <javascript:>
> > <mailto:sqlcipher+...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
> --
> Nick Parker

[Stephen Lombardo]

Hello Mosed,

I'm not clear exactly how you are redefining the encryption inside the libtomcrypt code. While XTS is a parellelizable mode, it would depend on your implementation. SQLCipher breaks the database up into pages, and each is encrypted or decrypted serially. Thus, at most you would be parallelizing the encryption of a single page of data at a time.

Cheers,

Stephen

To unsubscribe from this group and stop receiving emails from it, send an email to sqlcipher+...@googlegroups.com.

[M. Mosed]

Hello Stephen,

Hello Nick,

Mosed is back :). Busy for long time with different projects.

Not feeling good to leave the work in this thread until finished and get fruitful results.

About “I'm not clear exactly how you are redefining the encryption inside the libtomcrypt code”

As you know that Sqlcipher does add ltc support to Sqlite using CBC mode. By changing that part of

code you can redefine other encryption modes, which are supported by Libtomcrypt, instead of CBC.

“Thus, at most you would be parallelizing the encryption of a single page of data at a time”

Yes you are right. What I try to do is divide the page data between threads to improve performance.

But how much do you think that can improve performance?

Although this work is an academic one, I am willing to contact your customer support service and

get a paid support to help complete this work.

Thank you,

Mosed