SQLCipher as dynamic library within Couchbase Lite

[Brendan Duddridge]

Hi,

So I've successfully managed to use SQLCipher from within Couchbase Lite for Mac and encrypt one of my databases. I compiled SQLCipher as a .dylib dynamic library and linked it in to CBL Mac in the Link Libraries and Frameworks area and added -DSQLITE_HAS_CODEC to Other C Flags.

Now, this isn't necessarily directly a CBL issue, other than the fact that I can't compile it as a static library into CBL due to some symbol naming conflicts, so I have to use a dynamic library.

How do I get the dynamic library linked into my application so that the library doesn't need to be installed at any specific location on a user's drive?

It seems that the .dylib needs to be sitting in the location where it's first installed when built otherwise the application can't find it. It's not finding it from within the Xcode project itself.

Maybe I'm going about it all wrong and it needs to be a framework instead of a single .dylib file?

Thanks,

Brendan

[Brendan Duddridge]

I think this Stack Overflow post will help me figure it out:

http://stackoverflow.com/questions/7562793/how-do-i-create-a-working-framework-with-dylib-files-in-xcode-4

[Brendan Duddridge]

Unfortunately after spending a couple of days building SQLCipher in a billion different ways, I'm no further ahead than I was at the start.

So I have no idea how to properly integrate SQLCipher with Couchbase Lite such that I don't have to have my libsqlcipher.dylib in a fixed location on my drive. I've tried all the suggestions in the SOF post, but have yet to find a proper solution.

I did eventually get libsqlcipher to load from a relative location (I think), but then it doesn't actually work. That is, the call to set the encryption key on the database returns NO. So I'm still at a loss of how to proceed. At this point in time I'm thinking my only solution is to just stick with ForestDB which doesn't require SQLCipher for encryption.  My only concern with that is I don't have any proper database admin tools to use with a ForestDB database. Perhaps there's some that I'm not aware of other than CouchbaseLite Viewer. With SQLite I've been using SQLite Studio and SQLite Manager.

Mind you, I am using SQLCipher in my app outside of Couchbase Lite access too, so that adds one more level of complexity. The reason for this is I have a migration routine where I read my old SQLCipher encrypted database file via FMDB and then convert the data and store it in Couchbase Lite.

I got libsqlcipher.dylib to load by setting its "Dynamic Library Install Name" property to "@rpath/libsqlcipher.dylib" and the "Dynamic Library Install Name Base" to just @"rpath". So according to the log output, it is loading:

dyld: loaded: /Users/brendan/Projects/xcode-build/Tap_Forms_Mac-gmfcnfvjlruxgkapuehplefkxifo/Build/Products/Debug/Tap Forms.app/Contents/MacOS/../Frameworks/libsqlcipher.dylib

I did tell my build to copy the libsqlcipher.dylib into my Frameworks directory just as a place to store it so it could find it. And I have my "Runpath search paths" set to find it there: @loader_path/../Frameworks and @executable_path/../Frameworks

If anyone has any further ideas of what I could try that would be greatly appreciated.

Or should I not bother with SQLCipher since I don't know when or if Couchbase will stop supporting SQLite and go with ForestDB support only.

Thanks,

Brendan

[Jens Alfke]

On Oct 5, 2015, at 11:31 AM, Brendan Duddridge <bren...@gmail.com> wrote:

So I have no idea how to properly integrate SQLCipher with Couchbase Lite such that I don't have to have my libsqlcipher.dylib in a fixed location on my drive. I've tried all the suggestions in the SOF post, but have yet to find a proper solution.

Sorry about this. :( You’re sort of being the guinea pig for using SQLCipher with CBL on Mac OS — or maybe someone else has used it, but they were already expert at setting up embedded binaries so they didn’t have any questions.

The good news is that when we release CBL 1.2 we will include SQLCipher binaries for all supported platforms, and build instructions. (Yes, we’re working on encryption support for Android/Java and Xamarin/.NET too.)

I did eventually get libsqlcipher to load from a relative location (I think), but then it doesn't actually work. That is, the call to set the encryption key on the database returns NO. So I'm still at a loss of how to proceed.

The dyld log message you quoted looks correct. Make sure there is no other log showing the system sqlite3.dylib being loaded.

You can also look at the line CBL logs at startup that includes the SQLite version info, to check whether it matches what you built.

Did you modify the CBL Mac target to link libsqlcipher.dylib instead of libsqlite.dylib?

Or should I not bother with SQLCipher since I don't know when or if Couchbase will stop supporting SQLite and go with ForestDB support only.

We’ll be supporting it for a while yet. At least through 1.3, maybe farther. (Actually, depending on how you interpret semantic versioning, we might have to bump our version to 2.0 if we remove support for it.)

—Jens

PS: Out of curiosity, how big is your libsqlcipher.dylib?

[Brendan Duddridge]

Thanks for your response Jens. That's good info to know. I don't mind being a guinea pig on this, so I'll keep persevering. I'm not super adept at building my own dynamic libraries or frameworks. I just do a lot of trial and error. I do use a lot of third party frameworks and libraries though.

I did notice (and this is probably my main problem), that libsqlite3.dylib is being loaded up near the top of the log:

dyld: loaded: /usr/lib/libsqlite3.dylib

I'm just trying to track down why it's loading. I use a bunch of third party frameworks and I suspect one of them is loading it in. I've removed any reference to sqlite3 that I could find, but so far nothing has prevented it from loading.

Even though it is loading, is there a way to have SQLCipher take priority?

Thanks,

Brendan

[Brendan Duddridge]

Oh, you had asked how big my libsqlcipher.dylib file was. It's 1285028 bytes. About 1.3 MB.

Thanks,

Brendan

[Jens Alfke]

On Oct 5, 2015, at 12:58 PM, Brendan Duddridge <bren...@gmail.com> wrote:

I did notice (and this is probably my main problem), that libsqlite3.dylib is being loaded up near the top of the log:

dyld: loaded: /usr/lib/libsqlite3.dylib

I'm just trying to track down why it's loading. I use a bunch of third party frameworks and I suspect one of them is loading it in.

Oh, it’s not a problem if some other framework is using it. The two libraries aren’t going to conflict since they have different names. I just thought it might be a good way to see if Couchbase Lite were loading it.

Again, look at the log message from CBL_SQLiteStorage when the first database opens; that will give the version info from the SQLite API, which should help you figure out which library it’s using.

Oh, you had asked how big my libsqlcipher.dylib file was. It's 1285028 bytes. About 1.3 MB.

Smaller than I thought. I’m thinking that for our 1.2 Mac build we may as well just always build in SQLCipher. That would be simpler than having two versions of the library (one for encryption, one not.)

—Jens

[Brendan Duddridge]

Well the strange thing is that the log message I get from CBL is:

14:33:09.559| Couchbase Lite using SQLite version 3.8.10.2 (2015-05-20 18:17:19 2ef4f3a5b1d1d0c4338f8243d40a2452cc1f7fe4)

It seems to stay the same all the time, no matter whether I link in SQLCipher or not.

However, I am now able to encrypt again with CBL. Unfortunately my own FMDB based code isn't working. I do get FMDB from cocoa pods, so maybe I'll need to just embed it within and not use the CBL version.

One of the differences I see in your copy of FMDB is you reference sqlite via #import <sqlite3.h> rather than the cocoa pod's version of #import "sqlite3.h". Quotes instead of angled brackets. But I'm not entirely sure if that makes a difference.

Maybe I can go back to using SQLCipher via cocoapods but still have CBL use the .dylib I built that appears to be working.

Thanks,

Brendan

[Jens Alfke]

On Oct 5, 2015, at 1:44 PM, Brendan Duddridge <bren...@gmail.com> wrote:

One of the differences I see in your copy of FMDB is you reference sqlite via #import <sqlite3.h> rather than the cocoa pod's version of #import "sqlite3.h". Quotes instead of angled brackets. But I'm not entirely sure if that makes a difference.

It’s only a compile-time difference; wouldn’t affect what happens during linking or loading. The angle-bracket include just means to look for the header file in the system include paths first.

—Jens

[Brendan Duddridge]

Phew! I finally got both my app using SQLCipher and CBL using it from the same shared libsqcipher.dylib.

In the end this is everything I did:

In SQLCipher's Xcode project on the sqlcipher target:

1. Change Mach-O Type to "Dynamic Library"
   
2. Set Dynamic Library Install Name to "@rpath/libsqlcipher.dylib"
   
3. Set Dynamic Library Install Name Base to "@rpath"
   
4. Change Executable Extension to "dylib"
   
5. Build project
   

In Couchbase Lite's CBL Mac target:

1. Delete reference to libsqlite.dylib in the Link Binary with Libraries Build Phase
   
2. Create an sqlcipher folder within the vendors folder
   
3. Copy the libsqlcipher.dylib file from the SQLCipher's build folder into the sqcipher folder in CBL.
   
4. Copy the sqlite3.h file into the same folder.
   
5. Add the following to the Other C Flags setting:
   
   -DSQLITE_HAS_CODEC -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61
   Now, not all of those may be required, but that's just what I have always had in my project, so I stuck it there.
   
   
6. Add the libsqlcipher.dylib to the Link Binary with Libraries Build Phase.
7. Clean and build the CBL Listener target (or just the CBL Mac target if that's all you need).

In your Mac app's target:

1. Copy the CouchbaseLite.framework and CouchbaseLiteListener.framework to my app's Xcode project.
2. Create an sqlcipher folder in my app's project and copy the libsqlcipher.dylib and sqlite3.h file into it.
3. Add libsqlcipher.dylib to the Copy Frameworks Build Phase.
4. Since I'm using cocoa pods to import FMDB in my app, you need to add this to your Podfile:
   
   
   post_install do |installer_representation|
      installer_representation.pods_project.targets.each do |target|
         if target.name == 'FMDB'
            target.build_configurations.each do |config|
               config.build_settings['OTHER_CFLAGS'] ||= ['$(inherited)']
               config.build_settings['OTHER_CFLAGS'] << '-DSQLITE_HAS_CODEC -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4_UNICODE61'
               end
            end
       end
   end
   
   
5. Build and run your app.

Hopefully I didn't miss anything and that this helps someone else get this working without too much trouble.

Thanks Jens for all your help!

Brendan