Other Python DBs with TG

[Jimmie Houchin]

While exploring full text indexing options the other day I happened upon
a database which sounds very interesting to me.

Has anybody here every tried Axiom?

http://www.divmod.org/projects/axiom
"""
Axiom is an object database built on SQLite. In addition to
object-relational mapping it provides heterogenous object references and
schema upgrades.
"""

http://moxn.net/~mesozoic/axiom-howto/
"""
Axiom is a toolkit that lets you build object-oriented applications that
take advantage of queries. Axiom is not just a wrapper to be placed
around your existing database, which is what SQLObject does. Axiom is a
fully-featured object database, and the fact that it uses SQLite for its
backend is relatively tangential.
"""

There isn't a lot of info yet. I don't know how stable or ready it is.
However, I've long thought an object db with SQLite as backend would be
a great thing.

I am not trying to compare Axiom with SQLObject or that he represents
SQLObject correctly in his statement. But I am for choice in db and for
many a more possibly pure Python approach is wanted, desired or whatever.

I am not just talking about Axiom here, but also for ZODB, Durus, ...

I think SQLObject probably paves a nice Python way to RDBMS.
But what if you don't want an RDBMS?

Is this a part of TG which can be made pluggable or interface-able
without TG becoming too loose?

Just tossing out some random thoughts. :)

Jimmie

[Kevin Dangoor]

Hi Jimmie,

On 9/24/05, Jimmie Houchin <jlho...@gmail.com> wrote:
> While exploring full text indexing options the other day I happened upon
> a database which sounds very interesting to me.

I'd be interesting in hearing about any full text indexing options you
found, but that's not TurboGears-related...

> Has anybody here every tried Axiom?
>
> http://www.divmod.org/projects/axiom
> """
> Axiom is an object database built on SQLite. In addition to
> object-relational mapping it provides heterogenous object references and
> schema upgrades.
> """

SQLite is a great little database, but its database-level locks make
it suitable for webapps where you're writing very infrequently.

In addition, many people need to work with existing databases or other
(possibly non-Python) database tools. That's what an ORM getts you.

> I am not just talking about Axiom here, but also for ZODB, Durus, ...
>
> I think SQLObject probably paves a nice Python way to RDBMS.
> But what if you don't want an RDBMS?

While an object database that preserves some of the nice querying
capability you get with an RDBMS is appealing intellectually to me,
practically speaking I've found the tool support to be far superior
for RDBMSes... to the point where dealing with the "impedance
mismatch" is a worthwhile tradeoff.

But that's just me.

> Is this a part of TG which can be made pluggable or interface-able
> without TG becoming too loose?

At present, TurboGears has fairly minimal ties to SQLObject. I expect
that more will grow, but not to the point where someone is unable to
use whatever storage mechanism of choice they may have.

That said, the TurboGears project is out to help codify (in both docs
and code) easy-to-use solutions to the things that are universal or at
least very common among webapps. Storing and retrieving data is almost
as core to webapps as getting requests and sending responses!

Flexibility usually comes at the cost of added complexity. The code
becomes more complex, the docs become more complex.

Being able to focus on the application you're writing rather than the
infrastructure is the point. Being able to choose database engines is
a valid need for deployment... but choosing radically different
persistence schemes is just generally not required for a typical
application.

Another way to think about it is from a customer's perspective. When
we're writing webapps, we're writing them for someone to use. A
typical user cares a lot more that clicking on a widget on the screen
makes something slide or fade in neatly than they do about some nifty
database tech that's running under the hood.

> Just tossing out some random thoughts. :)

And it's much appreciated! This is the kind of stuff that helps to
define "what *is* TurboGears?".

Kevin

--
Kevin Dangoor
Author of the Zesty News RSS newsreader

email: k...@blazingthings.com
company: http://www.BlazingThings.com
blog: http://www.BlueSkyOnMars.com

[Jimmie Houchin]

Kevin Dangoor wrote:
> Hi Jimmie,
>
> On 9/24/05, Jimmie Houchin <jlho...@gmail.com> wrote:
>
>>While exploring full text indexing options the other day I happened upon
>>a database which sounds very interesting to me.
>
> I'd be interesting in hearing about any full text indexing options you
> found, but that's not TurboGears-related...

I've been thinking alot on how I want to handle my web app.
I've been playing in the Squeak world for a couple of years. I really
like Squeak but only recently decided to go with a Python solution
because I didn't (don't) think the Squeak solutions are as robust,
scalable or ready. Its a relatively small community of dedicated people.
But sometimes you just need more people on the problem.

Since you've explored Seaside you are somewhat familiar with the
situation. Seaside is great if you need all the magic. But if you don't
you still pay the price.

Okay back to my story. I've been considering data storage options. Since
much of my project contains a lot of static data. And because I can
easily segment my data into to simple files on the filesystem. And
because full text searching with databases isn't as transparent as
filesystem technologies. I explored Swish, htdig,...
(Yes I'm familiar with MySQL and PostgreSQL options)

Well while I was at the Gmane site that mentioned full text searching of
the archives. So I wanted to explore what they used.

They mentioned Xapian which I've never heard of so I searched.

http://www.xapian.org

which has Python bingings by DivMod with Xapwrap
http://www.divmod.org/projects/xapwrap

It claims you can index RDBMS via the PerlDBI module.
It would be interesting to see a Python solution for such.

But it looks quite interesting.

>>Has anybody here every tried Axiom?
>>
>>http://www.divmod.org/projects/axiom
>>"""
>> Axiom is an object database built on SQLite. In addition to
>>object-relational mapping it provides heterogenous object references and
>>schema upgrades.
>>"""
>
> SQLite is a great little database, but its database-level locks make
> it suitable for webapps where you're writing very infrequently.
>
> In addition, many people need to work with existing databases or other
> (possibly non-Python) database tools. That's what an ORM getts you.

Oh absolutely. If there is only one solution per se, it ought to be
SQLObject. It provides you with options for multiple backends. And I
would imagine some capacity to move from one backend to another.

>>I am not just talking about Axiom here, but also for ZODB, Durus, ...
>>
>>I think SQLObject probably paves a nice Python way to RDBMS.
>>But what if you don't want an RDBMS?
>
> While an object database that preserves some of the nice querying
> capability you get with an RDBMS is appealing intellectually to me,
> practically speaking I've found the tool support to be far superior
> for RDBMSes... to the point where dealing with the "impedance
> mismatch" is a worthwhile tradeoff.
>
> But that's just me.

Well you are most definitely right for you. :)
And you very well may be right overall.

As a non-RDBMS person, I only enter that arena if necessary. But
thankfully, SQLObject and such make it much more transparent. But there
is still the overall SQL/RDBMS mentallity and thinking.

Example from your http://www.turbogears.org/about/sqlobject.html

class Person(SQLObject):

firstName = StringCol(length=100)
middleInitial = StringCol(length=1, default=None)
lastName = StringCol(length=100)
lastContact = DateTimeCol(default=datetime.now)

Still a lot of SQL understanding, worldview, optimizing, conventions in
there.

Your still thinking in columns, sizes, constraints, keys, joins, etc.
(not all of that in that example but in other docs)

Which what prompted my thinking concerning this option.

>>Is this a part of TG which can be made pluggable or interface-able
>>without TG becoming too loose?
>
> At present, TurboGears has fairly minimal ties to SQLObject. I expect
> that more will grow, but not to the point where someone is unable to
> use whatever storage mechanism of choice they may have.
>
> That said, the TurboGears project is out to help codify (in both docs
> and code) easy-to-use solutions to the things that are universal or at
> least very common among webapps. Storing and retrieving data is almost
> as core to webapps as getting requests and sending responses!
>
> Flexibility usually comes at the cost of added complexity. The code
> becomes more complex, the docs become more complex.

Absolutely agree. Which was why my concern for such making it as I
expressed it "too loose".

> Being able to focus on the application you're writing rather than the
> infrastructure is the point. Being able to choose database engines is
> a valid need for deployment... but choosing radically different
> persistence schemes is just generally not required for a typical
> application.
>
> Another way to think about it is from a customer's perspective. When
> we're writing webapps, we're writing them for someone to use. A
> typical user cares a lot more that clicking on a widget on the screen
> makes something slide or fade in neatly than they do about some nifty
> database tech that's running under the hood.

Oh absolutely. But the developers care. Which is why we end up with the
different web app frameworks and persistence schemes.

I was not (maybe am not) seeing the necessarily radically different
nature in the persistence scheme from TGs view. Now I have not fully
examined TG, SQLObject, etc. nor am I a DB expert. So my thoughts maybe
in error naive.

But... again from your SQLObject page...

>>> p = Person(firstName="John", lastName="Doe")
>>> p
<Person 1 firstName='John' middleInitial=None
lastName='Doe' lastContact='datetime.datetime...)'>
>>> p.lastContact
datetime.datetime(2005, 9, 16, 9, 28, 7)
>>> p.firstName
'John'
>>> p.middleInitial = 'Q'
>>> p.middleInitial
'Q'

You speak on how Pythonic data access is. And it is. :)

And so without spending the time to research ZODB, Durus, or Axiom
fully. It naively seems that much of the differences are in the "class"
definition for the objects. From your example above it seems that
instance access is Pythonic. If that naive assessment is somewhat
accurate, it would seem we could make an interface which handles such.
Minimal appropriate datatypes which have optional parameters which can
be as minimal as the Python DBs allow, up to the additional stuff
available for RDBMSes through SQLObject.

Just more out there thinking. :)
I am totally alright with being tightly bound to SQLObject provided you,
TGBDFL, believe that it is the best decision for TG.

I believe it is better to have a consistent thought throughout your
framework, than to be inconsistent, but accomodating to all.

>>Just tossing out some random thoughts. :)
>
> And it's much appreciated! This is the kind of stuff that helps to
> define "what *is* TurboGears?".

Thanks. A good definition helps a lot. It is what enables proper, well
reasoned decisions. Or the lack of such clear definition disables proper
decision making.

Thanks for responding to these far out queries. :)

Jimmie

[Ian Bicking]

Jimmie Houchin wrote:
> They mentioned Xapian which I've never heard of so I searched.
>
> http://www.xapian.org
>
> which has Python bingings by DivMod with Xapwrap
> http://www.divmod.org/projects/xapwrap

I haven't used it seriously, but I have looked around and Xapian (with
xapwrap) was the only full-text indexer I was able to actually build and
try out from Python without much effort. So that's good. And the
divmod folks are using it more seriously, so I'm sure the bindings will
stay up to date.

Ian

[Oliver Andrich]

Hi Kevin,

2005/9/24, Kevin Dangoor <dan...@gmail.com>:

> On 9/24/05, Jimmie Houchin <jlho...@gmail.com> wrote:
> > While exploring full text indexing options the other day I happened upon
> > a database which sounds very interesting to me.
>
> I'd be interesting in hearing about any full text indexing options you
> found, but that's not TurboGears-related...

I did some research into full text indexing before I started with
TurboGears stuff, cause I am currently developing a news agency system
for print newspapers (now using TurboGears :). I looked into the usual
contenders in this Area Lucene (while I wanted to follow the company
guideline), Pylucene, Lupy, Joda (ioda.sf.net, which is developed by a
former boss of mine) and MySQL full text indexes.

Well, I will stick to MySQL as I heavily use SQLObject for the message
import, message searching and so on. It works just fine for me and is
way fast enough for a roughly around 800.000 text messages and the
IPTC headers for a little over 1.000.000 pictures. This is what we
keep inside the database for our editors.

Only Joda was faster and thanks to a friend of mine, it is also easily
accessible from Python. But I prefer MySQL, SQLObject and a single DB
abstraction layer.

Best regards,
Oliver

--
Oliver Andrich <oliver....@gmail.com> --- http://roughbook.de/

[bon...@gmail.com]

>firstName = StringCol(length=100)
>middleInitial = StringCol(length=1, default=None)
>lastName = StringCol(length=100)
>lastContact = DateTimeCol(default=datetime.now)

SQLObject can actually make this disappear. I just have the definition
in postgres then :

_fromdatabase = True

and you don't see the above in the python coding. However, it is not as
transparent as I thought it is(say plug different RDBMS). I am using
Postgres and have some constrains defined in the database(say UNIQUE,
NOT NULL). This cause psycopg to raise exceptions which means that I
need to import psycopg into my code to handle these exceptions, and
means that I cannot change to MySQL/SQLLite if I want to.

Also, for many application domains, SQL is still much better.

And in its current form, SQLObject is IMO just an optional part in
TurboGears, you can have an app that don't have SQLObject but some
other things you mentioned for data storage, or pure file system thing
using Pickle(?) or XML. And I am against having a "data access" layer
in TurboGears as how to access data is really application specific.

Kid and FormEncode on the other hand is an integral part of TurboGears.

[Ian Bicking]

bon...@gmail.com wrote:
> >firstName = StringCol(length=100)
> >middleInitial = StringCol(length=1, default=None)
> >lastName = StringCol(length=100)
> >lastContact = DateTimeCol(default=datetime.now)
>
> SQLObject can actually make this disappear. I just have the definition
> in postgres then :
>
> _fromdatabase = True
>
> and you don't see the above in the python coding. However, it is not as
> transparent as I thought it is(say plug different RDBMS). I am using
> Postgres and have some constrains defined in the database(say UNIQUE,
> NOT NULL). This cause psycopg to raise exceptions which means that I
> need to import psycopg into my code to handle these exceptions, and
> means that I cannot change to MySQL/SQLLite if I want to.

Right now the best way to do this (which is admittedly rather
long-winded) would be:

try:
something
except MyClass._connection.module.IntegrityError:
failure...

However, databases are not terribly consistent about when they throw
what error. E.g., Postgres will (with some drivers and database
versions) throw ProgrammingError for nearly anything. So it's only a
rough translation.

There's been some discussion on DB-SIG about the exception issue, and
the proposed fix is actually to define a separate exception hierarchy,
and then change the inheritance structure of any database driver modules
you are using, assigning to __bases__ so that, say,
psycopg.IntegrityError inherits from sqlobject.IntegrityError.