mod_python and a newer version of sqlite

[Nils Breunese]

Hello,

I'm trying to move two Trac environments from a Fedora Core 3 server
(Apache httpd 2.0.53, Trac 0.10.3, mod_python 3.1.3, python-sqlite
1.0.1, sqlite 2.8.16, PHP 4.3.11) to a new CentOS 4.4 server (Apache
httpd 2.0.52, Trac 0.10.3.1, mod_python 3.1.3, python-sqlite 1.1.7,
sqlite 3.3.3, PHP 5.1.6).

I'm seeing the infamous "PythonHandler trac.web.modpython_frontend:
DatabaseError: unsupported file format" error in my error_log. I have
had no luck so far getting it to work, even after reading through the
thread called "mod_python and sqlite3: unsupported file format" in
the archives and googling around.

I dumped the sqlite2 db's and imported those dumps into sqlite3
databases on the new server and those databases both seem to be OK. I
can access them just fine using trac-admin.

I also tried the steps on the Trac wiki and got the following results:

>>> import trac.db.sqlite_backend
>>> trac.db.sqlite_backend._ver
(3, 3, 3)
>>> trac.db.sqlite_backend.have_pysqlite
1
>>> trac.db.sqlite_backend.sqlite.version
'1.1.7'

This doesn't really tell me anything more than running 'rpm -q sqlite
python-sqlite', but at least both methods give me the same version
numbers. :o)

I have no idea what I still need to do now to get it to work. I found
a ticket (http://trac.edgewall.org/ticket/3112) saying that PHP5
might be interfering somehow and that disabling PHP5 fixes the
problem for some people, but I really cannot disable PHP5 on this
server. Any ideas?

Nils Breunese.

[Emmanuel Blot]

> >>> trac.db.sqlite_backend.sqlite.version
> '1.1.7'
> Any ideas?

From http://trac.edgewall.org/wiki/TracInstall#ForSQLite
To use SQLite 3 DB, you need PySQLite 2.x

Either stick with a SQLite 2 DB format and PySQLite 1.x (I don't think
it's recommended), or upgrade you PySQLite library.

Cheers,
Manu

[Nils Breunese]

Emmanuel Blot wrote:

I just got what yum installed from the CentOS 4 repositories. CentOS
4 has sqlite-3.3.3 and python-sqlite-1.1.7 in their repository (see
<http://mirror.centos.org/centos-4/4.4/os/i386/CentOS/RPMS/>) and my
system is completely updated. So, you're saying CentOS (and thus Red
Hat) are shipping a combination of sqlite and python-sqlite that
cannot work together?

Nils Breunese.

[Noah Kantrowitz]

Look for packages under "sqlite3" and "python-sqlite2".

--Noah

[Emmanuel Blot]

> So, you're saying CentOS (and thus Red
> Hat) are shipping a combination of sqlite and python-sqlite that
> cannot work together?

Nope, only that you don't have the appropriate packages installed on
your system ;-)

Cheers,
Manu

[Nils Breunese]

Noah Kantrowitz wrote:

Check the link I posted, there are no packages by those names in the
CentOS repository. Just sqlite (version 3.3.3) and python-sqlite
(version 1.1.7).

The python-sqlite-1.1.7 package even requires libsqlite3.so.0:

# rpm -q --requires python-sqlite
libc.so.6
libc.so.6(GLIBC_2.0)
libc.so.6(GLIBC_2.1.3)
libc.so.6(GLIBC_2.3)
libpthread.so.0
libsqlite3.so.0
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1

And the sqlite-3.3.3 package provides it:

# rpm -q --provides sqlite
libsqlite3.so.0
sqlite = 3.3.3-1.2

Is someone on this list running Trac on CentOS 4 using mod_python and
the standard CentOS sqlite and python-sqlite packages? Are python-
sqlite-1.1.7 and sqlite-3.3.3 really incompatible, even though Red
Hat and CentOS are shipping these versions together?

Nils Breunese.

[Emmanuel Blot]

> Is someone on this list running Trac on CentOS 4 using mod_python and
> the standard CentOS sqlite and python-sqlite packages? Are python-
> sqlite-1.1.7 and sqlite-3.3.3 really incompatible, even though Red
> Hat and CentOS are shipping these versions together?

There are not incompatible per se, from PySQLite web site, PySQLite
1.1.x is compatible with SQLite3 but uses a "legacy" API.
However, Trac needs the new API.

See also http://initd.org/tracker/pysqlite/wiki/PysqliteVersions

--
Manu

[Noah Kantrowitz]

Nils Breunese wrote:
> Is someone on this list running Trac on CentOS 4 using mod_python and
> the standard CentOS sqlite and python-sqlite packages? Are

> python-sqlite-1.1.7 and sqlite-3.3.3 really incompatible, even though

> Red Hat and CentOS are shipping these versions together?

Unless they (RH) just changed the names, yes.

--Noah

[Nils Breunese]

Emmanuel Blot wrote:

You mean Trac can't use the legacy API. ;o)

So it is not possible to use Trac using stock RHEL/CentOS 4 packages?
Do I need to go and compile my own stuff now? Are there any
instructions for getting Trac to run on RHEL/CentOS 4?

Thanks for all your replies so far,

Nils Breunese.

[Nils Breunese]

Noah Kantrowitz wrote:

Reading http://initd.org/tracker/pysqlite/wiki/PysqliteVersions it
seems that python-sqlite-1.1.x was especially created to support
sqlite 3.x, so I don't think they are incompatible at all. It just
seems that Trac is not able to use sqlite 3.x and python-sqlite-1.1.x.

If anyone has any pointers that might help me get Trac to work,
that'd be great. I'd rather just use yum to install stuff than go
compile my own (I've never had to do that in all my years of using
mainstream distro's), if only keeping stuff up2date is just a lot
more work when you need to build it yourself.

Thanks,

Nils Breunese.

[Noah Kantrowitz]

http://trac.edgewall.org/wiki/TracOnRhel4 perhaps ;-)

--Noah

[Nils Breunese]

Noah Kantrowitz wrote:

Hehe, I hadn't found that one yet. But my setup is exactly like
described there. That links doesn't mention anything special about
installing special version of sqlite or python-sqlite and I am using
the trac package from rpmforge.

Nils Breunese.

[Christian Boos]

Nils Breunese wrote:
> Noah Kantrowitz wrote:
>
>> Nils Breunese wrote:
>>> Emmanuel Blot wrote:
>>>
>>>> From http://trac.edgewall.org/wiki/TracInstall#ForSQLite
>>>> To use SQLite 3 DB, you need PySQLite 2.x

That information in the wiki page is a bit misleading... The Pysqlite2
1.1.x releases have always worked with SQLite. Only Pysqlite version
1.0.x works with SQLite 2.x and not 3.x.

> (snip)

>
> Is someone on this list running Trac on CentOS 4 using mod_python and
> the standard CentOS sqlite and python-sqlite packages? Are

> python-sqlite-1.1.7 and sqlite-3.3.3 really incompatible, even though

> Red Hat and CentOS are shipping these versions together?

So those packages are OK.
What probably happens is that there's some kind of binary
incompatibility within Apache, another SQLite library is loaded or
another module (mod_php5.so?) includes the SQLite objects directly, and
the sqlite-3.3.3 library can't be loaded or doesn't end up being used
(i.e. GCC's own interpretation of the dll hell).

Make sure only one SQLite version gets used, either by rebuilding php to
use 3.3.3 (if php is the culprit), or by rebuilding pysqlite so that it
links with the sqlite library already in use. You'll have to play a bit
with ldd and nm in order to identify the other libraries involved.

-- Christian

[Noah Kantrowitz]

RPMForge does seem to have the older 1.0.x pysqlite, which Trac can use.
This is probably your best bet.

--Noah

[Nils Breunese]

Christian Boos wrote:

> What probably happens is that there's some kind of binary
> incompatibility within Apache, another SQLite library is loaded or
> another module (mod_php5.so?) includes the SQLite objects directly,
> and
> the sqlite-3.3.3 library can't be loaded or doesn't end up being used
> (i.e. GCC's own interpretation of the dll hell).
>
> Make sure only one SQLite version gets used, either by rebuilding
> php to
> use 3.3.3 (if php is the culprit), or by rebuilding pysqlite so
> that it
> links with the sqlite library already in use. You'll have to play a
> bit
> with ldd and nm in order to identify the other libraries involved.

I just have stock CentOS 4.4 installs of httpd, python, mod_python,
sqlite and python-sqlite and this box is also running PHP5 from the
CentOSPlus repository (no mod_php5.so, but libphp5.so). Frankly I'm
not really prepared to go and rebuild some packages and "play a bit
with ldd and nm" to get Trac to work with sqlite. I think I might
just see if I can make the switch to MySQL or some other backend.

Thanks though,

Nils Breunese.

[Nils Breunese]

Noah Kantrowitz wrote:

Why would that help? From what I read pysqlite 1.0 is not compatible
with sqlite 3.x and sqlite 3.3.3 is what comes with CentOS 4.4.

Nils Breunese.

[Matthew Gillen]

Do you have a package installed called "python-sqlite2" ? If so, what's it's
version?

[Noah Kantrowitz]

You would also use FCGI, since Trac and PHP are in different memory
spaces with FCGI this won't be issue.

--Noah

[Noah Kantrowitz]

Because RPMForge also has sqlite2.

--Noah

[Emmanuel Blot]

> Reading http://initd.org/tracker/pysqlite/wiki/PysqliteVersions it
> seems that python-sqlite-1.1.x was especially created to support
> sqlite 3.x, so I don't think they are incompatible at all.
> It just seems that Trac is not able to use sqlite 3.x and python-sqlite-1.1.x.

Yeah, this is what I wrote, isn't it? ;-)
python-sqlite-1.1.x has been created to support Sqlite3, but it is not
the recommended way to access SQLite3: Pysqlite2 is the way to go.

Now if both your servers have a similar architecture, there is a
chance that you can simply copy the .db file from one machine to
another - rather than dump/loading it, and keep using SQLite2 format.
This would make it till CentOS decides to upgrade its packages:
Pysqlite2.1 is 1 year and a half old...

Cheers,
Manu

[Nils Breunese]

Matthew Gillen wrote:

No, I don't. These are the only sqlite packages I have installed:

# rpm -qa | grep sqlite
python-sqlite-1.1.7-1.2
sqlite-3.3.3-1.2

According to yum there's also no python-sqlite2 package available in
the repositories.

Nils Breunese.

[Nils Breunese]

Op 27-apr-2007, om 18:54 heeft Emmanuel Blot het volgende geschreven:

> Now if both your servers have a similar architecture, there is a
> chance that you can simply copy the .db file from one machine to
> another - rather than dump/loading it, and keep using SQLite2 format.

They're not similar enough, the old FC3 server runs SQLite2, CentOS 4
comes with SQLite3 and I have already experienced SQLite2 files don't
really work that well with SQLite3.

> This would make it till CentOS decides to upgrade its packages:
> Pysqlite2.1 is 1 year and a half old...

Because of stability distributions like RHEL and CentOS don't do
version upgrades during their lifecycle, they only backport security
fixes (<http://www.redhat.com/advice/speaks_backport.html>).

Nils Breunese.

[Nils Breunese]

Op 27-apr-2007, om 18:52 heeft Noah Kantrowitz het volgende geschreven:

>>> RPMForge does seem to have the older 1.0.x pysqlite, which Trac
>>> can use.
>>> This is probably your best bet.
>>
>> Why would that help? From what I read pysqlite 1.0 is not compatible
>> with sqlite 3.x and sqlite 3.3.3 is what comes with CentOS 4.4.
>
> Because RPMForge also has sqlite2.

Sadly both packages are called sqlite and the latest version at
RPMForge is also 3.x, so I'd have to install an old package and block
sqlite updates in my package manager. I like to run patched software,
so I'm not really up for that.

Nils Breunese.

[Emmanuel Blot]

> Because of stability distributions like RHEL and CentOS don't do
> version upgrades during their lifecycle, they only backport security
> fixes (<http://www.redhat.com/advice/speaks_backport.html>).

So you need to rebuild pysqlite2 on your own, or use PostgreSQL. MySQL
support is not as good.

Cheers,
Manu

[Nils Breunese]

Noah Kantrowitz wrote:

What exactly are you saying? That Trac and MySQL can't work using
mod_python when I'm running PHP? Why would that be a problem?

People, I'm off to have some dinner. Thanks for all your replies, I'm
goinig to blow off some steam and get back to this a little later.

Nils Breunese.

[Noah Kantrowitz]

It isn't a problem, I am just saying you could still use SQLite if you
switched from mod_python to FCGI.

--Noah

[Nils Breunese]

Nils Breunese wrote:

If Trac really is incompatible with SQLite 3, and PySQLite 1.1 using
mod_python then that TracOnRhel4 howto on the wiki should work for
nobody, because that's the setup you'd be installing if you follow
those instructions. I'm quite puzzled by this.

Nils Breunese.

[Aaron D. Marasco]

I have trac running fine on CentOS 4.4 and I don't think I did anything
special... however I have been known to forget everything after 48 hours.

I think packages with .rf at the end are from dag - see below

python-clearsilver-0.10.1-1.2.elf.rf
python-sqlite-1.1.7-1.2
mod_python-3.1.3-5.1
python-2.3.4-14.3
trac-0.10.3-1.el4.rf
sqlite-3.3.3-1.2
sqlite-devel-3.3.3-1.2
php-pecl-sqlite-1.0.3-1.2.el4.rf
sqlitemanager-1.2.0-1.el4.rf

...so it looks like the python-sqlite and sqlite are stock CentOS (could
be fasttrack). If there's any other packages you want me to confirm, let
me know.

- Aaron

http://blog.revragnarok.com/blog/tags/%27trac%27

/etc/yum.repos.d/dag.repo (copied by hand be careful):
[dag]
name=Dag RPM Repository
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=0
enabled=1

[Nils Breunese]

Aaron D. Marasco wrote:

> I have trac running fine on CentOS 4.4 and I don't think I did
> anything
> special... however I have been known to forget everything after 48
> hours.
>
> I think packages with .rf at the end are from dag - see below
>
> python-clearsilver-0.10.1-1.2.elf.rf
> python-sqlite-1.1.7-1.2
> mod_python-3.1.3-5.1
> python-2.3.4-14.3
> trac-0.10.3-1.el4.rf
> sqlite-3.3.3-1.2
> sqlite-devel-3.3.3-1.2
> php-pecl-sqlite-1.0.3-1.2.el4.rf
> sqlitemanager-1.2.0-1.el4.rf
>
> ...so it looks like the python-sqlite and sqlite are stock CentOS
> (could
> be fasttrack). If there's any other packages you want me to
> confirm, let
> me know.

So apparently it *is* possible to use Trac with sqlite 3 and python-
sqlite 1.1. Then what could be causing my problem? Would that be the
PHP5 module from CentOSPlus? I heard from Aaron he uses the stock PHP
4.3 package. Can anybody explain to me if/why PHP5 could cause
trouble (and not PHP 4?)?

Nils Breunese.

[Nils Breunese]

Nils Breunese wrote:

I created a fresh Subversion repository and Trac environment
following http://trac.edgewall.org/wiki/TracOnRhel4, but I'm getting
the "PythonHandler trac.web.modpython_frontend: DatabaseError:
unsupported file format" error again. Really the only difference I
see with my setup is that I have PHP 5.1.6 installed instead of RHEL/
CentOS 4's PHP 4.3. :o(

I'm pretty bummed that I can't seem to get Trac to work on CentOS 4
using mod_python, while it worked just fine on Fedora Core 3.

Nils Breunese.

[Aaron D. Marasco]

Nils Breunese wrote:
> Nils Breunese wrote:
>> Aaron D. Marasco wrote:
>>> I have trac running fine on CentOS 4.4 and I don't think I did anything
>>> special... however I have been known to forget everything after 48
>>> hours.
>>>
>>> I think packages with .rf at the end are from dag - see below
>>>
>>> python-clearsilver-0.10.1-1.2.elf.rf
>>> python-sqlite-1.1.7-1.2
>>> mod_python-3.1.3-5.1
>>> python-2.3.4-14.3
>>> trac-0.10.3-1.el4.rf
>>> sqlite-3.3.3-1.2
>>> sqlite-devel-3.3.3-1.2
>>> php-pecl-sqlite-1.0.3-1.2.el4.rf
>>> sqlitemanager-1.2.0-1.el4.rf
>>>
>>> ...so it looks like the python-sqlite and sqlite are stock CentOS
>>> (could
>>> be fasttrack). If there's any other packages you want me to confirm,
>>> let
>>> me know.
>>
>> So apparently it *is* possible to use Trac with sqlite 3 and

>> python-sqlite 1.1. Then what could be causing my problem? Would that

>> be the PHP5 module from CentOSPlus? I heard from Aaron he uses the
>> stock PHP 4.3 package. Can anybody explain to me if/why PHP5 could
>> cause trouble (and not PHP 4?)?
>
> I created a fresh Subversion repository and Trac environment following
> http://trac.edgewall.org/wiki/TracOnRhel4, but I'm getting the
> "PythonHandler trac.web.modpython_frontend: DatabaseError: unsupported
> file format" error again. Really the only difference I see with my

> setup is that I have PHP 5.1.6 installed instead of RHEL/CentOS 4's

> PHP 4.3. :o(
>
> I'm pretty bummed that I can't seem to get Trac to work on CentOS 4
> using mod_python, while it worked just fine on Fedora Core 3.
>

I dunno why PHP would have anything to do with it. I have
php-4.3.9-3.22.4, along with php-ldap, php-mysql, php-pear all matching,
and php-pecl-sqlite-1.0.3-1.2.el4.rf . But I don't see how PHP matters.

- Aaron

[Noah Kantrowitz]

Nils Breunese wrote:
> Nils Breunese wrote:
>
>> Aaron D. Marasco wrote:
>>
>>> I have trac running fine on CentOS 4.4 and I don't think I did anything
>>> special... however I have been known to forget everything after 48
>>> hours.
>>>
>>> I think packages with .rf at the end are from dag - see below
>>>
>>> python-clearsilver-0.10.1-1.2.elf.rf
>>> python-sqlite-1.1.7-1.2
>>> mod_python-3.1.3-5.1
>>> python-2.3.4-14.3
>>> trac-0.10.3-1.el4.rf
>>> sqlite-3.3.3-1.2
>>> sqlite-devel-3.3.3-1.2
>>> php-pecl-sqlite-1.0.3-1.2.el4.rf
>>> sqlitemanager-1.2.0-1.el4.rf
>>>
>>> ...so it looks like the python-sqlite and sqlite are stock CentOS (could
>>> be fasttrack). If there's any other packages you want me to confirm, let
>>> me know.
>>
>> So apparently it *is* possible to use Trac with sqlite 3 and

>> python-sqlite 1.1. Then what could be causing my problem? Would that

>> be the PHP5 module from CentOSPlus? I heard from Aaron he uses the
>> stock PHP 4.3 package. Can anybody explain to me if/why PHP5 could
>> cause trouble (and not PHP 4?)?

It isn't PHP 4 vs 5, it is the fact that another Apache module is linked
against a different version of sqlite than your pysqlite bindings.
If/when it gets loaded first, that version takes priority and confuses
pysqlite. This is contraint of the way Linux (and most other unixes)
handle dynamic linking, and the fact that all Apache modules run within
the same memory space.

--Noah

[Nils Breunese]

But I only have one version of sqlite as CentOS 4.4 comes with only one
version of sqlite (3.x) and I would think it would be very strange if
the PHP packages from the CentOSPlus repository (for CentOS 4.4) would
be linked against a version of sqlite that is not available on CentOS
4.4. How can I check which versions of sqlite PHP and python are linked
against? Like I mentioned in my original post starting this thread I
already ran the following:

>>> import trac.db.sqlite_backend
>>> trac.db.sqlite_backend._ver
(3, 3, 3)
>>> trac.db.sqlite_backend.have_pysqlite
1
>>> trac.db.sqlite_backend.sqlite.version
'1.1.7'

sqlite 3.3.3 and python-sqlite 1.1.7 are the only sqlite-related
packages I have installed, but how can I check if PHP might be interfering?

Nils Breunese.

[Nils Breunese]

Nils Breunese wrote:
> Noah Kantrowitz wrote:

>> It isn't PHP 4 vs 5, it is the fact that another Apache module is linked
>> against a different version of sqlite than your pysqlite bindings.
>> If/when it gets loaded first, that version takes priority and confuses
>> pysqlite. This is contraint of the way Linux (and most other unixes)
>> handle dynamic linking, and the fact that all Apache modules run within
>> the same memory space.
>
> But I only have one version of sqlite as CentOS 4.4 comes with only one
> version of sqlite (3.x) and I would think it would be very strange if
> the PHP packages from the CentOSPlus repository (for CentOS 4.4) would
> be linked against a version of sqlite that is not available on CentOS
> 4.4. How can I check which versions of sqlite PHP and python are linked
> against? Like I mentioned in my original post starting this thread I
> already ran the following:
>
> >>> import trac.db.sqlite_backend
> >>> trac.db.sqlite_backend._ver
> (3, 3, 3)
> >>> trac.db.sqlite_backend.have_pysqlite
> 1
> >>> trac.db.sqlite_backend.sqlite.version
> '1.1.7'
>
> sqlite 3.3.3 and python-sqlite 1.1.7 are the only sqlite-related
> packages I have installed, but how can I check if PHP might be interfering?

Well, it seems it was indeed PHP interfering. I disabled the pdo_sqlite
extension (commented it out in /etc/php.d/pdo_sqlite.ini), restarted
apache and all the Trac projects were working! We're not using sqlite
with PHP so it's not really a problem, but I wonder what the ultimate
solution would be. Would the PHP package have to be altered to somehow
use the same sqlite as Python?

Nils Breunese.