How to build sqlcipher using Visual C++?
Aleksey Kontsevich
improved following
http://www.wiki.crossplatform.ru/index.php/%D0%9F%D1%80%D0%B8%D0%B2%D1%8F%D0%B7%D0%BA%D0%B0_SQLCipher_%D0%BA_Qt
article and successfully built sqlcipher library and plugin with gcc
(Linux) and MinGW (Windows). But how to build it in Windows with
Visual C++ (may be even in Visual Studio)? I have found no
instructions about it.
Stephen Lombardo
Aleksey Kontsevich
On 14 дек, 21:51, Stephen Lombardo <sjlomba...@zetetic.net> wrote:
> Hi Aleksey,
>
> If you want to build a Visual Studio using SQLCipher you can either:
>
> 1. Build a DLL - Here are some relevant threads:
>
>
> http://groups.google.com/group/sqlcipher/browse_thread/thread/bab6ab5...
> 2. Generate an amalgamation - the dependencies and configure process is
> about the same as with #1, but you would make sqlite3.c to build the
> amalgamated source file
>
> In either case you'll need OpenSSL DLLs or static libraries to link against
> on windows.
>
> Another alternative is to purchase the pre-build win32 binary package we
> Cheers,
> Stephen
>
> On Sun, Dec 12, 2010 at 10:52 AM, Aleksey Kontsevich
> > How to build sqlcipher using Visual C++? I had read, extended and
> > improved following
>
Stephen Lombardo
- The first link provided contains an extensive discussion of the prerequisites for SQLCipher build on windows, including what prerequisites are required.
- The OpenSSL documentation also contains detailed information on how to build OpenSSL for windows.
- The second link also contains information on how to build the SQLCipher DLL.
- In addition, the original QT document you referred to generates a SQLCipher DLL which you should be able to use with Visual Studio. - http://www.wiki.crossplatform.ru/index.php/%D0%9F%D1%80%D0%B8%D0%B2%D1%8F%D0%B7%D0%BA%D0%B0_SQLCipher_%D0%BA_Qt
Aleksey Kontsevich
Only thing we can do is to attach dll built with MinGW to VS project?
> simplified build script to post here (the windows build has a number of
> prerequisites and command line steps) That said:
>
> are required.
> build OpenSSL for windows.
> 3. The second link also contains information on how to build the
> SQLCipher DLL.
> 4. In addition, the original QT document you referred to generates a
> The steps for using the generated DLL with Visual Studio is really
> an exercise for the developer, not something we are able to provide guidance
> on. There is plenty of information on how to include a DLL into a Visual
> Studio project available in the VS help documentation and on
> msdn.microsoft.com.
>
> As an aside, If you, or someone else who's using SQLCipher with Visual C++,
> would like to author a full tutorial we'd be happy to post the instructions
> on the SQLCipher site to help out other folks. If you have any other
> specific questions let us know and we'll do our best to help you out.
> Thanks!
>
> Cheers,
> Stephen
>
> On Wed, Dec 15, 2010 at 7:58 AM, Aleksey Kontsevich
Stephen Lombardo
make sqlite3.c
This will generate a large c source file named sqlite3.c in the sqlcipher source directory, which can then include directly in your Visual C++ project. Note that you will still need to link your project against either the OpenSSL dll or static library for SQLCipher to have access to the crypto functions.
Cheers,
Stephen
Michael Stephenson
Hi Stephen,
Thanks again for all the good stuff.
After I started working on a project using SqlCipher for sqlite encryption (on Windows, via MSVC), I realized some things:
**********************************************
1) I wanted to be able to rebuild the library myself when a new version of sqlite comes out, or with a version of sqlite that a SqlCipher version was not published against.
2) I wanted to be able to customize certain things, such as:
a. What crypto library I use. In my case, I wasn't happy adding 500K for openssl to the size of my executable when all I needed was a key derivation function and one or two encryption algorithms, or if I found an algorithm that I liked that had acceptable security but was faster than openssl's AES.
b. Whether I want to use a passphrase + kdf, versus just storing an encryption key in binary format or generating one at runtime from known values with the goal of obfuscating the binary. In my case, this is not a "user sets the passphrase" scenario, but rather a "use doesn't own the data" scenario. Storing a passphrase in plaintext makes it easier to reverse, so I don’t want to use a passphrase. Keep in mind here that in my use cases, reversing of the executable is by far the weaker vulnerability than reversing the encryption (even relatively weak encryption).
c. What random function I want to use to generate initialization vectors on the database pages, or whether I want to use them at all, or want to derive them from, say, the page number.
d. How big I want the encryption key, initialization vector, etc. to be.
3) I wanted to better understand how SqlCipher integrated with the sqlite code.
4) I wanted to be able to try different implementations and generate performance metrics.
**********************************************
So, I’ve restructured the code in the following ways…
**********************************************
1) I always now start with a plain sqlite amalgamated build.
2) To sqlite3.c, I:
a. Add #include “stdafx.h” at the very top of the file
b. Add #include “crypto.h” at the very end of the file
3) Create a new file, stdafx.h, in the same folder as sqlite3.c. (I actually just copy this file from a previous sqlite encryption project as a starting point). My base file is:
#pragma once
#define WIN32_LEAN_AND_MEAN
#define SQLITE_ENABLE_COLUMN_METADATA
#define SQLITE_API __declspec(dllexport)
#define SQLITE_DEFAULT_PAGE_SIZE 4096
#define SQLITE_HAS_CODEC
4) Place the attached crypto.h in the same folder as sqlite3.c.
a. This file is included at the bottom of sqlite3.c per #2b above.
b. This file provides implementation of the six or seven sqlite functions that must be defined when SQLITE_HAS_CODEC is defined during compilation.
c. The main features of this crypto.h are simplicity and dependency injection.
d. This is all boilerplate code that has been trimmed down to minimum requirements.
e. The things that can be done entirely boilerplate are implemented entirely in this file.
f. This file never changes from build to build of sqlite.
g. The dependency injection comes from the following functions that are declared in crypto.h but are not defined:
void CodecLibraryInitialize(void); // initialize crypto library, if needed
int CodecInitialize(struct Db*, codec_ctx* pCtx); // further initialize codec/cipher beyond defaults
int CodecKeySize(); // tell us the length of the encryption key (in bytes)
int CodecIvSize(); // tell us the size of the initialization vector on each database page
int CodecDeriveKey(const char* szPassphrase, int nPassphrase,
const unsigned char* pSalt, int nSalt,
unsigned char* pKey, int nKey); // generate an encryption key/length, returned in pKey, nKey
void CodecRandomness(int nBytes, void* pBuf); // generate pseudorandom bytes into buffer
void CodecFree(void* p); // free codec state data
int CodecEncryptDecrypt(codec_ctx* pCtx, Pgno pgNo, int mode,
int bufSize, unsigned char* in,
unsigned char* out); // encrypt/decrypt
h. At the bottom of this file is an include, #include “cryptoimpl.h”
5) I then create a file cryptoimpl.h in the same folder as sqlite3.c (again, copied from a base template or a previous project), and implement the 8 functions declared in 4f above (some of these may be empty implementations).
This approach has worked quite nicely for me. It gives me a clear definition of what functions I need to implement in what ways to enable/disable/alter behaviors. For example, CodecIvSize() can return 0 to decide that there is no initialization vector on the database pages. I can also use whatever crypto library or crypto functions I want to use to implement key derivation, random pseudobytes, and encryption/decryption. I can even use a C++ library if I handle the C/C++ linkage issues correctly. A sample cryptoimpl.h is attached to this e-mail along with crypto.h. I’ve also tried to use function names such that: a) All callbacks start with Codec. b) No functions except those that are part of sqlite (declared in sqlite) start with sqlite. c) Functions pointed to via the call to sqlite3PagerSetCodec start with “X” (XCodec and XCodecFree). d) Unnecessary functions that could be manually inlined were removed.
Some attempts at simplification have been made. For example, in sqliteCodecAttach, I assume that a negative nKey means that a binary encryption key is being passed rather than a passphrase. The passphrase itself is never stored anywhere in memory after a key is derived from it.
The one thing that is left out using this approach is the pragma support, which would require editing the sqlite3.c amalgamation at the location where pragma.c is spliced into the file. Since I don’t need pragma support (I am the only user of these databases; end users do not have the ability to run queries directly), I prefer to leave this out to help keep the build/implementation process simple.
During the work on this, I’ve learned a lot about sqlite and SqlCipher; it’s been fun.
Just throwing this out there in case there are any ideas you might want to integrate into SqlCipher. Anyone who finds anything here useful is free to use whatever they like completely free of any license or copyright and without any guarantees or warranties.
Thanks,
~Mike Stephenson
Aleksey Kontsevich
Thank You very much for Your answers. I think, I understand what is
amalgamation build now. Will try to include generated sqlite3.c file
into my sqlcipher Qt sql driver plugin project and in success will
update the documentation then -
http://www.wiki.crossplatform.ru/index.php/%D0%9F%D1%80%D0%B8%D0%B2%D1%8F%D0%B7%D0%BA%D0%B0_SQLCipher_%D0%BA_Qt
With best regards,
Aleksey.
> Visual C++ alone using an amalgamation build of SQLCipher.
> The prerequisites are the same, except, instead of running make dll, you'd
> run
>
> make sqlite3.c
>
> This will generate a large c source file named sqlite3.c in the sqlcipher
> source directory, which can then include directly in your Visual C++
> project. Note that you will still need to link your project against either
> the OpenSSL dll or static library for SQLCipher to have access to the crypto
> functions.
>
> Cheers,
> Stephen
>
> On Thu, Dec 16, 2010 at 4:55 AM, Aleksey Kontsevich
Aleksey Kontsevich
./configure --prefix=/QTPATH/src/plugins/sqldrivers/sqlcipher --
disable-tcl --disable-amalgamation CFLAGS="-DSQLITE_HAS_CODEC -I../
OpenSSL/include /c/Windows/System32/libeay32.dll"
With --disable-amalgamation or --enable-amalgamation?
Aleksey Kontsevich
libeay32mt.lib?
Stephen Lombardo
Based on the naming I believe libeay32mt.lib should be the multi threaded static lib and libeay32md.lib would be the import library for the dll.
Aleksey Kontsevich
====================================================
TARGET = qsqlcipher
HEADERS = ../../../sql/drivers/sqlite/qsql_sqlite.h
SOURCES = smain.cpp \
../../../sql/drivers/sqlite/qsql_sqlite.cpp
QT += sql
SQLITE_TEMP_STORE=2
!system-sqlite:!contains( LIBS, .*sqlite.* ) {
CONFIG(release, debug|release):DEFINES *= NDEBUG
DEFINES += SQLITE_OMIT_LOAD_EXTENSION SQLITE_OMIT_COMPLETE
INCLUDEPATH += include
win32 {
HEADERS += ./include/sqlite3.h ./include/sqlite3ext.h
SOURCES += sqlite3.c
LIBS += ./lib/libeay32MD.lib
}
unix {
QMAKE_RPATHDIR += lib
LIBS += -Llib -lsqlite3
}
} else {
LIBS *= $$QT_LFLAGS_SQLITE
QMAKE_CXXFLAGS *= $$QT_CFLAGS_SQLITE
}
include(../qsqldriverbase.pri)
====================================================
And got many errors during solution build like following:
>Error 1 error C2037: left of 'pBt' specifies undefined struct/union 'Btree' c:\Qt\4.7.0\src\plugins\sqldrivers\sqlcipher\sqlite3.c 44924 qsqlcipher
>Error 13 error C2198: 'sqlite3_mutex_try' : too few arguments for call c:\Qt\4.7.0\src\plugins\sqldrivers\sqlcipher\sqlite3.c 44991 qsqlcipher
What other sources or ./configure options I missed here? Need to
include something more to sqlite3.c?
Stephen Lombardo
Even when you are using the amalgamation, you still need to include the corresponding header file, sqlite3.h. Make sure you've included that in your project before building.
Michael Stephenson
Also check that you are compiling/linking sqlite3.c with C linkage and not C++.
Aleksey Kontsevich
On 17 дек, 04:44, Stephen Lombardo <sjlomba...@zetetic.net> wrote:
> Hi Aleksey,
>
> Even when you are using the amalgamation, you still need to include the
> corresponding header file, sqlite3.h. Make sure you've included that in your
> project before building.
>
> Cheers,
> Stephen
>
> On Thu, Dec 16, 2010 at 9:41 PM, Aleksey Kontsevich
Aleksey Kontsevich
Michael Stephenson
>Error 1 error C2037: left of 'pBt' specifies undefined struct/union 'Btree' c:\Qt\4.7.0\src\plugins\sqldrivers\sqlcipher\sqlite3.c 44924 qsqlcipher
means that the definition of struct Btree is missing. You could search sqlite3.c for "struct Btree". It should be typedef'd near the top of the file and then actually defined somewhere not too far above the line with the error.
In my plain sqlite amalgamation (not SqlCipher), the definition of struct Btree comes in from the file btreeInt.h. I see these comments in my plain sqlite3.c a couple hundred lines above the struct Btree definition:
/************** Include btreeInt.h in the middle of btmutex.c ****************/
/************** Begin file btreeInt.h ****************************************/
It sounds like btreeInt.h may not have been properly inserted into the amalgamation when it was built via configure.
If someone has a known-good sqlite3.c, a diff would show the problem quickly. I would try to provide a known-good, but at the moment cygwin and msys are crashing on my windows machine and are not functional.
-----Original Message-----
From: sqlc...@googlegroups.com [mailto:sqlc...@googlegroups.com] On Behalf Of Aleksey Kontsevich
Sent: Thursday, December 16, 2010 11:16 PM
To: SQLCipher Users
Subject: Re: How to build sqlcipher using Visual C++?
Michael Stephenson
-----Original Message-----
From: sqlc...@googlegroups.com [mailto:sqlc...@googlegroups.com] On Behalf Of Aleksey Kontsevich
Sent: Thursday, December 16, 2010 8:38 PM
To: SQLCipher Users
Aleksey Kontsevich
Aleksey Kontsevich
produces incorrect sqlite3.c?
Stephen Lombardo
Stephen Lombardo
Aleksey Kontsevich
and copied to project dir also. However I think sqlite3.c does not use
it as sqlite3.h included into it:
>** This file is all you need to compile SQLite. To use SQLite in other
>** programs, you need this file and the "sqlite3.h" header file that defines
>** the programming interface to the SQLite library. (If you do not have
>** the "sqlite3.h" header file at hand, you will find a copy embedded within
>** the text of this file. Search for "Begin file sqlite3.h" to find the start
>** of the embedded sqlite3.h header file.) Additional code files may be needed
>** if you want a wrapper to interface SQLite with your choice of programming
>** language. The code for the "sqlite3" command-line shell is also in a
>** separate file. This file contains only code for the core SQLite library.
Changed .pro file to:
HEADERS += sqlite3.h
SOURCES += sqlite3.c
Still have that errors just trying to compile generated sqlite3.cpp:
>Error 1 error C2037: left of 'pBt' specifies undefined struct/union 'Btree' c:\Qt\4.7.0\src\plugins\sqldrivers\sqlcipher\sqlite3.c 44924 qsqlcipher
>Error 60 error C2037: left of 'isExclusive' specifies undefined struct/union 'BtShared' c:\Qt\4.7.0\src\plugins\sqldrivers\sqlcipher\sqlite3.c 45498 qsqlcipher
>Error 79 error C2065: 'BtLock' : undeclared identifier c:\Qt\4.7.0\src\plugins\sqldrivers\sqlcipher\sqlite3.c 45549 qsqlcipher
I checked generated sqlite3.cpp and sqlite3.h - they do not have
definitions for Btree, BtShared, BtLock structure, for
SQLITE_FILE_HEADER definition, etc.
Do I need some more sources to compile or change some configure
options? What is wrong here?
On 17 дек 2010, 16:24, Stephen Lombardo <sjlomba...@zetetic.net>
wrote:
Aleksey Kontsevich
errors start from 44924 line. Seems it hidden by
#ifdef SQLITE_HAS_CODEC
pragma. Where need to specify it?
Stephen Lombardo
Are you defining SQLITE_HAS_CODEC when you build your application that includes the SQLCipher amalgamation?
Cheers,
Stephen
Aleksey Kontsevich
openssl/evp.h file - builds successfully now!!! Thank You very much
for all, Stephen and Michael!!! :)
Aleksey Kontsevich
Stephen Lombardo
Aleksey Kontsevich
#pragma once
#define SQLITE_ENABLE_COLUMN_METADATA
#ifdef Q_WS_WIN
#define WIN32_LEAN_AND_MEAN
#define SQLITE_API __declspec(dllexport)
#endif
#define SQLITE_DEFAULT_PAGE_SIZE 4096
#define SQLITE_HAS_CODEC