Using pysqlite2 with apsw

[Sriram Karra]

I am having a lot of trouble using apsw with pysqlite2. I downloaded the latest versions of both, and built them both against the latest amalgamation of sqlite3. Yet I am having trouble creating a dbapi2 connection on top of a apsw connection. 

Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05) 

[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> from pysqlite2 import dbapi2 as sqlite3

>>> print sqlite3.sqlite_version

3.7.15.2

>>> import apsw

>>> print apsw.sqlitelibversion()

3.7.15.2

>>> apsw_con = apsw.Connection(":memory:")

>>> apsw_con.createscalarfunction("times_two", lambda x: 2*x, 1)

>>> con = sqlite3.connect(apsw_con)

Segmentation fault: 11

As you can see the sqlite version is the same. I read in the docs that apsw and pysqlite have to be build against the same sqlite dynamic library. So I began to wonder if compiling them separately against the same verstion but different physical instances of the sqlite3.c file could be the problem. Then again, I could not find way of (a) building a standalone sqlite3.so and (b) an option in the build scripts of either apsw or pysqlite to use an external shared library for sqlite3. Can someone please help with this problem?

Thanks

Sriram

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/02/13 03:39, Sriram Karra wrote:
> I am having a lot of trouble using apsw with pysqlite2. I downloaded
> the latest versions of both,

I had thought pysqlite had removed the functionality. However it appears
the code is still there, but the documentation has been removed.

>>>> con = sqlite3.connect(apsw_con)
> Segmentation fault: 11

You'd need to use a debugger to find out which code is hitting the problem.

> I read in the docs that apsw and pysqlite have to be build against the
> same sqlite dynamic library.

The reason is that SQLite has various internal global variables. If you
have two copies of the amalgamation in the same program then you will have
two different copies of the global variables and SQLite will get very
confused if you pass things from one to the other.

> (a) building a standalone sqlite3.so

$ ./configure
$ make

You'll end up with .libs/libsqlite3.so

> (b) an option in the build scripts of either apsw or pysqlite to use
> an external shared library for sqlite3.

http://apidoc.apsw.googlecode.com/hg/build.html#finding-sqlite-3

Distutils build_ext command allows specifying include directories and
library directories.

$ python setup.py build_ext --help

This should work with both APSW and pysqlite:

$ python setup.py build_ext -I .../wherever -L .../wherever install

Roger

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlEUVsUACgkQmOOfHg372QSxGACgiV1N/ZBau72pKIZB5sbsZbCZ
T54An00EWV/ZLAzntFOXt9fEM2KOzGlM
=7MRR
-----END PGP SIGNATURE-----

[Sriram Karra]

On Friday, 8 February 2013 07:07:09 UTC+5:30, Roger Binns wrote:

 

> (a) building a standalone sqlite3.so

$ ./configure
$ make

You'll end up with .libs/libsqlite3.so

 I am on a Mac. I ended up with a libsqlite3.dylib

$ python setup.py build_ext -I .../wherever -L .../wherever install

I could specify my sqlite3 location in this manner, but it was not really linking well with apsw or pysqlite. I guess I am doing something a bit stupid. Can you spot anything? The error I get is:

llvm-gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -Wl,-F. -arch i386 -arch x86_64 build/temp.macosx-10.7-intel-2.7/src/apsw.o -L/Users/sriramkarra/Downloads/software/unix/sqlite-autoconf-3071502/.libs/ -L/Users/sriramkarra/Downloads/software/unix/sqlite-autoconf-3071502/.libs -lsqlite3 -o build/lib.macosx-10.7-intel-2.7/apsw.so

ld: warning: ignoring file /Users/sriramkarra/Downloads/software/unix/sqlite-autoconf-3071502/.libs//libsqlite3.dylib, file was built for unsupported file format which is not the architecture being linked (i386) 

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/02/13 03:39, Sriram Karra wrote:
> /Users/sriramkarra/Downloads/software/unix/sqlite-autoconf-3071502/.libs//libsqlite3.dylib,
>
>
file was built for unsupported file format which is not the architecture
> being linked (i386)

You have to give the relevant flags to SQLite's configure to make it build
for all architectures (aka universal binary). This or similar variant may
work (untested):

$ ./configure CFLAGS="-arch i386 -arch x86_64"
$ make

Roger

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlEapx0ACgkQmOOfHg372QT7NwCfeiTl+vlS9cOR6nQpsRMPPhOY
6YcAoMUjALCrZS70HTf3W2n5VeOfjSXH
=BC+H
-----END PGP SIGNATURE-----

[Sriram Karra]

On Wednesday, 13 February 2013 02:03:33 UTC+5:30, Roger Binns wrote:

>
file was built for unsupported file format which is not the architecture
> being linked (i386)

You have to give the relevant flags to SQLite's configure to make it build
for all architectures (aka universal binary).  This or similar variant may
work (untested):

  $ ./configure CFLAGS="-arch i386 -arch x86_64"
  $ make

Hm, I am not sure the latest releases are meant to be compiled. I get this error: llvm-gcc-4.2: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags"

That's a lot of flags that don't work with multiple -arch flags. Any clue what's the impact of nuking all of them?

Cardassia:sqlite-autoconf-3071502 sriramkarra$ make

if /bin/sh ./libtool --tag=CC --mode=compile gcc -DPACKAGE_NAME=\"sqlite\" -DPACKAGE_TARNAME=\"sqlite\" -DPACKAGE_VERSION=\"3.7.15.2\" -DPACKAGE_STRING=\"sqlite\ 3.7.15.2\" -DPACKAGE_BUGREPORT=\"http://www.sqlite.org\" -DPACKAGE_URL=\"\" -DPACKAGE=\"sqlite\" -DVERSION=\"3.7.15.2\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DHAVE_FDATASYNC=1 -DHAVE_USLEEP=1 -DHAVE_LOCALTIME_R=1 -DHAVE_GMTIME_R=1 -DHAVE_DECL_STRERROR_R=1 -DHAVE_STRERROR_R=1 -DHAVE_READLINE=1 -I. -I.    -D_REENTRANT=1 -DSQLITE_THREADSAFE=1  -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_RTREE -arch i386 -arch x86_64 -MT sqlite3.lo -MD -MP -MF ".deps/sqlite3.Tpo" -c -o sqlite3.lo sqlite3.c; \

then mv -f ".deps/sqlite3.Tpo" ".deps/sqlite3.Plo"; else rm -f ".deps/sqlite3.Tpo"; exit 1; fi

mkdir .libs

 gcc -DPACKAGE_NAME=\"sqlite\" -DPACKAGE_TARNAME=\"sqlite\" -DPACKAGE_VERSION=\"3.7.15.2\" "-DPACKAGE_STRING=\"sqlite 3.7.15.2\"" -DPACKAGE_BUGREPORT=\"http://www.sqlite.org\" -DPACKAGE_URL=\"\" -DPACKAGE=\"sqlite\" -DVERSION=\"3.7.15.2\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DHAVE_FDATASYNC=1 -DHAVE_USLEEP=1 -DHAVE_LOCALTIME_R=1 -DHAVE_GMTIME_R=1 -DHAVE_DECL_STRERROR_R=1 -DHAVE_STRERROR_R=1 -DHAVE_READLINE=1 -I. -I. -D_REENTRANT=1 -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_RTREE -arch i386 -arch x86_64 -MT sqlite3.lo -MD -MP -MF .deps/sqlite3.Tpo -c sqlite3.c  -fno-common -DPIC -o .libs/sqlite3.o

llvm-gcc-4.2: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags

make: *** [sqlite3.lo] Error 1

[Roger Binns]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28/02/13 23:43, Sriram Karra wrote:
> I get this error: llvm-gcc-4.2: -E, -S, -save-temps and -M options are
> not allowed with multiple -arch flags"

I googled that error and got this resolution:

./configure --disable-dependency-tracking

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlEwdiwACgkQmOOfHg372QSn5QCgujusqraFqbSZCLfOE5wEKnZq
S4MAoI/NclHmaVE6OJT4hXNnJ7AFY/SC
=EjdW
-----END PGP SIGNATURE-----

[Sriram Karra]

On Fri, Mar 1, 2013 at 3:04 PM, Roger Binns <rog...@rogerbinns.com> wrote:

 

On 28/02/13 23:43, Sriram Karra wrote:

> I get this error: llvm-gcc-4.2: -E, -S, -save-temps and -M options are
> not allowed with multiple -arch flags"

I googled that error and got this resolution:

  ./configure --disable-dependency-tracking

Thank you. That did fix it. I am now able to use apsw with pysqilte2. I now have a related problem for which I will start a new thread.