changed sqlite requirements in 0.12dev (#8625)

[Eli]

All,

In looking to apply a cleaner version of [8817] to 0.12dev, I noticed that we
appear to have completely dropped support for sqlite 1 from 0.12dev in [8571]
as part of http://trac.edgewall.org/ticket/8625

CentOS 5 (and therefore REL 5) have sqlite 1
CentOS 6 and REL 6 aren't out yet.

So the statement from that ticket that "I think there's no need for a
deprecation period, as hardly anyone would be using it for new installations
or even upgrades." is, sadly, not the case.

This is going to be a problem for me, personally, as I try to get my various
Trac instances up to date.

I strongly believe that Trac needs to be able to run on the current major
"enterprise" versions of Linux, and as 0.12dev stands, it won't.

My Trac development time has been very limited, but I can try to address this
next week. Are there objections to me re-adding sqlite1 support given that
CentOS and REL are still shipping sqlite1?

Eli
------------------. "If it ain't broke now,
Eli Carter \ it will be soon." -- crypto-gram
retr...@gmail.com `-------------------------------------------------

[John Hampton]

Eli wrote:
> CentOS 5 (and therefore REL 5) have sqlite 1

Are you sure about this? I just setup a centos5.3 vz, no updates, and
only the default yum repos enabled[1] and I have sqlite-3.3.6 installed

-John


[1] base, updates, addons, extras

[Eli]

On Wednesday 18 November 2009 7:28:48 pm John Hampton wrote:
> Eli wrote:
> > CentOS 5 (and therefore REL 5) have sqlite 1
>
> Are you sure about this? I just setup a centos5.3 vz, no updates, and
> only the default yum repos enabled[1] and I have sqlite-3.3.6 installed

I guess I should have been clearer; there are 2 sqlite versions involved.
There is the version of the sqlite library, and there is the version of the
sqlite python bindings.

$ rpm -qa | grep sqlite
sqlite-3.3.6-5
sqlite-devel-3.3.6-5
python-sqlite-1.1.7-1.2.1

It is support for the sqlite python bindings version 1 that was dropped from
trunk.

[Felix Schwarz]

Am 19.11.2009 02:11, schrieb Eli:
> CentOS 5 (and therefore REL 5) have sqlite 1

Fedora EPEL ships python-sqlite2. Most people will use that repository
anyway to get python-genshi I suppose.

fs

[Christian Boos]

Hi Eli,

Eli wrote:
> All,
>
> In looking to apply a cleaner version of [8817] to 0.12dev, I noticed that we
> appear to have completely dropped support for sqlite 1 from 0.12dev in [8571]
> as part of http://trac.edgewall.org/ticket/8625
>
> CentOS 5 (and therefore REL 5) have sqlite 1
> CentOS 6 and REL 6 aren't out yet.
>
> So the statement from that ticket that "I think there's no need for a
> deprecation period, as hardly anyone would be using it for new installations
> or even upgrades." is, sadly, not the case.
>
> This is going to be a problem for me, personally, as I try to get my various
> Trac instances up to date.
>
> I strongly believe that Trac needs to be able to run on the current major
> "enterprise" versions of Linux, and as 0.12dev stands, it won't.
>
> My Trac development time has been very limited, but I can try to address this
> next week. Are there objections to me re-adding sqlite1 support given that
> CentOS and REL are still shipping sqlite1?
>

Well, it's hard to believe that people wanting to install a recent Trac
version couldn't also take the extra 5 minutes needed to download and
install the pysqlite 2.x bindings, if they're using Python 2.4.
Concerning the other versions, Python 2.3 support has been dropped and
I'm sure you know that Python 2.5 and upward come with the "sqlite3"
package bundled, so this is only an issue for 2.4. Pysqlite can even
download for you the latest SQLite amalgamation source file for creating
a self-contained binary, when doing a build_static.

In case you want to rely on the platform packages exclusively and if you
can't install pysqlite 2.x that way, then you certainly won't be able to
install Trac 0.12, Genshi 0.6 and Babel 0.9.4 that way either.

Now why did we drop support for pysqlite 1.1.x?
The last version released on the pysqlite 1.1.x branch was 1.1.8, in
2006. That 1.1 branch is basically an adaptation to SQLite 3.x of the
original pysqlite 1.0 branch which worked with SQLite 2.x. As such, it
shares many of its drawbacks, like poor concurrency behavior. Pysqlite
2.x was a major rewrite and works much better for Trac. Also, whenever
we have a problem report involving pysqlite 1.0.x or 1.1.x, the first
thing we do is to ask them to upgrade to 2.x, so we can hardly say it's
"supported".

As the topic of performance improvement gradually became more and more
important, trying to avoid misconfiguration or using "known bad" support
packages is something we should do. Better tell the Trac administrator
upfront that there's some more installation work to do in case all the
prerequisites are not met, rather than let him do a quick install with
problematic packages. Otherwise there will be *more* work afterward when
troubleshooting performance, at which point the responsibility of the
pysqlite 1.1.x package will not be obvious (see e.g
http://trac.edgewall.org/ticket/7785#comment:29 ; ok, that was an
upgrade from 1.0.x but I hope you get the point).

-- Christian

[Eli]

Ah, I'd forgotten about EPEL. Wonder where my brain went. Ok, "it's in EPEL"
is sufficient for me.

[Eli]

On Thursday 19 November 2009 12:21:16 pm Christian Boos wrote:
> Well, it's hard to believe that people wanting to install a recent Trac
> version couldn't also take the extra 5 minutes needed to download and
> install the pysqlite 2.x bindings, if they're using Python 2.4.

It's yet another 5 minutes, and "it's hard to believe" that it would only be 5
minutes of additional work to figure out what was needed, determine that it
was not available for the distro, locate the correct website, download the
right version, hunt down the build dependencies, build and package the
dependency, and install it.

> Concerning the other versions, Python 2.3 support has been dropped and
> I'm sure you know that Python 2.5 and upward come with the "sqlite3"
> package bundled, so this is only an issue for 2.4. Pysqlite can even
> download for you the latest SQLite amalgamation source file for creating
> a self-contained binary, when doing a build_static.

If CentOS 5 was shipping python 2.3, I would have blocked dropping support for
Python 2.3. If you are running the current version of a major enterprise
Linux distro, you should not have to upgrade things like python and sqlite
bindings to run the current version of Trac.

> In case you want to rely on the platform packages exclusively and if you
> can't install pysqlite 2.x that way, then you certainly won't be able to
> install Trac 0.12, Genshi 0.6 and Babel 0.9.4 that way either.

I see those three packages as fundamentally different from sqlite. In
particular, those are all edgewall.org projects. My concern was that I want
people (including myself) to be able to run Trac on CentOS 5, and did not want
to force yet another package upgrade on the process. The more work it is, the
fewer people will use Trac.

That said, since pysqlite 2 is in EPEL (http://fedoraproject.org/wiki/EPEL),
that's enough for me on that score, so I drop my objection.

(EPEL also has genshi 0.5.1, babel 0.9.4, trac 0.10.5 for what it's worth.)

> Now why did we drop support for pysqlite 1.1.x?
> The last version released on the pysqlite 1.1.x branch was 1.1.8, in
> 2006. That 1.1 branch is basically an adaptation to SQLite 3.x of the
> original pysqlite 1.0 branch which worked with SQLite 2.x. As such, it
> shares many of its drawbacks, like poor concurrency behavior. Pysqlite
> 2.x was a major rewrite and works much better for Trac. Also, whenever
> we have a problem report involving pysqlite 1.0.x or 1.1.x, the first
> thing we do is to ask them to upgrade to 2.x, so we can hardly say it's
> "supported".
>
> As the topic of performance improvement gradually became more and more
> important, trying to avoid misconfiguration or using "known bad" support
> packages is something we should do. Better tell the Trac administrator
> upfront that there's some more installation work to do in case all the
> prerequisites are not met, rather than let him do a quick install with
> problematic packages. Otherwise there will be *more* work afterward when
> troubleshooting performance, at which point the responsibility of the
> pysqlite 1.1.x package will not be obvious (see e.g
> http://trac.edgewall.org/ticket/7785#comment:29 ; ok, that was an
> upgrade from 1.0.x but I hope you get the point).

Since pysqlite 2 is in EPEL, I'm ok with dropping pysqlite 1 support. That
said, these arguments would convince me to _deprecate_ pysqlite 1, explicitly
stating in an ERROR-level deprecation message that upgrading to pysqlite 2 is
strongly encouraged, and stating that pysqlite 1 support would be dropped as
soon as pysqlite 2 was available in CentOS.


Fundamentally, for our 3rd-party dependencies, we must support the versions
available in the major enterprise distributions' latest release. Failure to
do so causes our users pain.

[Christian Boos]

Eli wrote:
> On Thursday 19 November 2009 12:21:16 pm Christian Boos wrote:
>
>> Well, it's hard to believe that people wanting to install a recent Trac
>> version couldn't also take the extra 5 minutes needed to download and
>> install the pysqlite 2.x bindings, if they're using Python 2.4.
>>
>
> It's yet another 5 minutes, and "it's hard to believe" that it would only be 5
> minutes of additional work to figure out what was needed, determine that it
> was not available for the distro,

If it's more than 5 minutes, then our installation documentation has a
problem, and that's *not* hard to believe :-)
So up to this point I agree, but not for the following:

> locate the correct website, download the right version,

A link to http://code.google.com/p/pysqlite/downloads/list in the
INSTALL file
and TracInstall page would be enough. Did that in [8830].

> hunt down the build dependencies, build and package the
> dependency, and install it.
>

No, as that's a simple matter of doing:

$ wget http://pysqlite.googlecode.com/files/pysqlite-2.5.6.tar.gz
$ tar xvfz pysqlite-2.5.6.tar.gz
$ cd pysqlite-2.5.6
$ python setup.py build_static install

No extra dependencies. As I said before, the above will automatically
fetch the latest SQLite and bundle it within the _pysqlite.so module, so
there's no interference with the "system" sqlite.

(easy_install pysqlite doesn't work well right now, I've e-mailed
Gerhard about that; if someone knows how to clear the old links in
http://pypi.python.org/simple/pysqlite/, please let us know)

>> Concerning the other versions, Python 2.3 support has been dropped and
>> I'm sure you know that Python 2.5 and upward come with the "sqlite3"
>> package bundled, so this is only an issue for 2.4. Pysqlite can even
>> download for you the latest SQLite amalgamation source file for creating
>> a self-contained binary, when doing a build_static.
>>
>
> If CentOS 5 was shipping python 2.3, I would have blocked dropping support for
> Python 2.3. If you are running the current version of a major enterprise
> Linux distro, you should not have to upgrade things like python and sqlite
> bindings to run the current version of Trac.
>

CentOS is listed 12th on DistroWatch.com and from the data I see there,
it roughly seems to be lagging 2 years behind the others from the top
10. More specifically, CentOS 5.4 (2009-10-21) seems to be based on
Fedora "zod" (2006-10-24), judging by the versions of the packages.

So no, I wouldn't have accepted that our decision of dropping the
support for Python 2.3 in Trac 0.12 could have been vetoed on the ground
that such a "major enterprise Linux distribution" was stuck in the past.

I expect a lively discussion next year, when I'll propose dropping 2.4
support for Trac 0.13 :-)

>> In case you want to rely on the platform packages exclusively and if you
>> can't install pysqlite 2.x that way, then you certainly won't be able to
>> install Trac 0.12, Genshi 0.6 and Babel 0.9.4 that way either.
>>
>
> I see those three packages as fundamentally different from sqlite. In
> particular, those are all edgewall.org projects. My concern was that I want
> people (including myself) to be able to run Trac on CentOS 5, and did not want
> to force yet another package upgrade on the process. The more work it is, the
> fewer people will use Trac.
>
> That said, since pysqlite 2 is in EPEL (http://fedoraproject.org/wiki/EPEL),
> that's enough for me on that score, so I drop my objection.
>
> (EPEL also has genshi 0.5.1, babel 0.9.4, trac 0.10.5 for what it's worth.)
>

Yeah, that's what I would expect. People wanting to use "enterprise"
Linux distributions will want to use stable versions of their software,
and usually just stick with what the packaging system proposes. If they
want a newer version of Trac, they could already have a go and install
Trac *0.11*. That will still work for them, it'll even presumably work
with their packaged version of Genshi (0.5.1).

Remember 0.12 is futureware at this point, not expected to ship before
Christmas 2009 ;-)

My point is that people who already crossed the line and installed
things outside of their packaging system and far too "recent" compared
to the baseline they have, wouldn't mind doing an extra installation
steps if: 1/ it's really needed, 2/ it's simple enough, 3/ it pays off
immediately by saving them the pain to fix it afterwards. Remember,
we're not talking here about enforcing an install of a newer version of
Python, or something painful to install like Clearsilver was.

Also what about setuptools? Do your enterprise Linux distribution have
it packaged? That would be surprising, knowing the reluctance of
distribution packagers to support language specific packaging
solutions... So that's also likely an extra install step anyway.

And what about Genshi? Could a Genshi package installed from the EPEL be
used by Trac, which has the setuptools "install_requires" thing?

-- Christian

[Felix Schwarz]

Am 20.11.2009 18:19, schrieb Christian Boos:
> Also what about setuptools? Do your enterprise Linux distribution have
> it packaged? That would be surprising, knowing the reluctance of
> distribution packagers to support language specific packaging
> solutions... So that's also likely an extra install step anyway.

Fedora EPEL (which produces packages to complement the basic RHEL/CentOS
offering) provides indeed packages for setuptools (0.6c7). And all
Python packages in Fedora contain egg info metadata so that
pkg_resources/setuptools will work with custom software as well.

In RHEL5/EPEL only selected packages ship with metadata, mostly
depending on the individual maintainer (I updated zope.interface some
weeks ago, some others feel that this change is not worth the potential
breakage).

RHEL6 (based on Fedora 11/12) will ship Python 2.6 and will have egg
info for all Python packages.

Generally speaking Fedora/EPEL has a collection of hard-working package
maintainers that try to provide a very good platform for Python
development (including setuptools support which you can use for your own
out-of-tree projects).

I just became co-maintainer for trac in Fedora and hope I can fix some
quirks there. So if you have any questions/suggestions about
Fedora/EPEL, don't hesitate to contact me or the Fedora python-sig.

> And what about Genshi? Could a Genshi package installed from the EPEL be
> used by Trac, which has the setuptools "install_requires" thing?

Yes. Same goes for pytz and a couple of other libraries (pytz in
EPEL/Fedora uses/will use the system-wide tz data so it should benefit
from platform updates).

fs

[Eli]

On Sunday 22 November 2009 11:24:38 am Felix Schwarz wrote:
> RHEL6 (based on Fedora 11/12) will ship Python 2.6 and will have egg
> info for all Python packages.

Is RHEL6 supposed to come out in Q1 2010, or has RedHat given any guidance on
their intentions?

[Christian Boos]

Thank you for these detailed informations.

Now I have another question. For building the packages, do you use the
`python setup.py bdist_rpm` method?
If not, are the extra changes needed a matter of changing the setup.cfg
or the setup.py files?
In this case, we should integrate those changes, if possible.

As I said to Eli yesterday, it seems that we never shipped .rpms
directly from t.e.o (or rather, that Jonas stopped doing so after Trac
0.7.1, see http://ftp.edgewall.com/pub/trac/old). Is there any interest
in doing that again, or was there a good reason against?

-- Christian

[Ed - 0x1b, Inc.]

don't forget...

if from a repo, it may be handy to add a PYTHONPATH entry so it can be
found - or make a .pth file in /usr/lib64/python2.5/site-packages/
(YDMV)
being there and being found makes it work.

thx All

[Felix Schwarz]

Am 23.11.2009 04:09, schrieb Eli:
> On Sunday 22 November 2009 11:24:38 am Felix Schwarz wrote:
>> RHEL6 (based on Fedora 11/12) will ship Python 2.6 and will have egg
>> info for all Python packages.
>
> Is RHEL6 supposed to come out in Q1 2010, or has RedHat given any guidance on
> their intentions?

Mostly rumors, but I'm confident that RHEL 6 will be released in Q2.

fs

[Felix Schwarz]

Am 23.11.2009 10:13, schrieb Christian Boos:
> Now I have another question. For building the packages, do you use the
> `python setup.py bdist_rpm` method?
> If not, are the extra changes needed a matter of changing the setup.cfg
> or the setup.py files?

In order to provide the best possible integration with the distro (and
to lower the maintenance burden), Fedora uses a custom spec file
template. However the packaging process still uses 'setup.py
build'/'setup.py install'.

This is the current trac.spec in Fedora:
http://cvs.fedoraproject.org/viewvc/rpms/trac/devel/trac.spec?revision=1.30&view=markup
(I will improve the file a bit, especially the require statements are
way to broad).

fs