don't automatically stringify compiled statements - patch to base.py

[Monty Taylor]

Hi - I'd attach this as a patch file, but it's just too darned small...

I would _love_ it if we didn't automatically stringify compiled
statements here in base.py, because there is no way to override this
behavior in a dialect. I'm working on an NDBAPI dialect to support
direct access to MySQL Cluster storage, and so I never actually have a
string representation of the query. Most of the time this is fine, but
to make it work, I had to have pre_exec do the actual execution,
because by the time I got to do_execute, I didn't have my real object
anymore.

I know this would require some other code changes to actually get
applied - namely, I'm sure there are other places now where the
statement should be str()'d to make sense.

Alternately, we could add a method to Compiled. (I know there is
already get_str()) like "get_final_query()" that gets called in this
context instead. Or even, although it makes my personal code less
readable, just call compiled.get_str() here, which is the least
invasive, but requires non-string queries to override a method called
get_str() to achieve a purpose that is not a stringification.

Other than this, so far I've actually got the darned thing inserting
records, so it's going pretty well... other than a whole bunch of test
code I put in to find out why it wasn't inserting when the problem was
that I was checking the wrong table... *doh*

Thanks!
Monty

=== modified file 'lib/sqlalchemy/engine/base.py'
--- lib/sqlalchemy/engine/base.py 2007-02-13 22:53:05 +0000
+++ lib/sqlalchemy/engine/base.py 2007-03-14 23:17:40 +0000
@@ -312,7 +312,7 @@
return cursor
context = self.__engine.dialect.create_execution_context()
context.pre_exec(self.__engine, proxy, compiled, parameters)
- proxy(str(compiled), parameters)
+ proxy(compiled, parameters)
context.post_exec(self.__engine, proxy, compiled, parameters)
rpargs = self.__engine.dialect.create_result_proxy_args(self, cursor)
return ResultProxy(self.__engine, self, cursor, context,
typemap=compiled.typemap, columns=compiled.columns, **rpargs)
@@ -342,7 +342,7 @@
if cursor is None:
cursor = self.__engine.dialect.create_cursor(self.connection)
try:
- self.__engine.logger.info(statement)
+ self.__engine.logger.info(str(statement))
self.__engine.logger.info(repr(parameters))
if parameters is not None and isinstance(parameters, list)
and len(parameters) > 0 and (isinstance(parameters[0], list) or
isinstance(parameters[0], dict)):
self._executemany(cursor, statement, parameters,
context=context)

[Michael Bayer]

well at least make a full blown patch that doesnt break all the other
DB's. notice that an Engine doesnt just execute Compiled objects, it
can execute straight strings as well. thats why the dialect's
do_execute() and do_executemany() take strings - they are assumed to
go straight to a DBAPI representation. to take the "stringness" out
of Engine would be a large rework to not just Engine but all the
dialects.

im surprised the execute_compiled() method works for you at all, as
its creating a cursor, calling result set metadata off the cursor,
etc. all these DBAPI things which you arent supporting. it seems
like it would be cleaner for you if you werent even going through
that implementation of it.

the theme here is that the base Engine is assuming a DBAPI
underneath. if you want an Engine that does not assume string
statements and DBAPI's it might be easier for you to just provide a
subclass of Engine instead (or go even lower level, subclass
sqlalchemy.sql.Executor). either way you can change what
create_engine() returns by using a new "strategy" to create_engine(),
which is actually a pluggable API. e.g.

from sqlalchemy.engine.strategies import DefaultEngineStrategy

class NDBAPIEngineStrategy(DefaultEngineStrategy):
def __init__(self):
DefaultEngineStrategy.__init__(self, 'ndbapi')

def get_engine_cls(self):
return NDBAPIEngine

# register the strategy
NDBAPIEngineStrategy()

now you connect via:

create_engine(url, strategy='ndbapi')

if you want to go one level lower, which i think you do because you
dont really want pooling or any of that either, you dont even need to
have connection pooling or anything like that....you can totally
override what create_engine() does, have different connection
parameters, whatever. just subclass EngineStrategy directly:

class NDBAPIEngineStrategy(EngineStrategy):
def __init__(self):
EngineStrategy.__init__(self, 'ndbapi')
def create(self, *args, **kwargs):
# this is some arbitrary set of arguments
return NDAPIEngine(kwargs.get('connect_string'), kwargs.get
('some_other_argument'), new NDBAPIDialect(*args), etc etc)
# register
NBAPIEngineStrategy()

then you just say:

create_engine(connect_string='someconnectstring',
some_other_argument='somethingelse', strategy='ndbapi')

i.e. whatever you want. create_engine() just passes *args/**kwargs
through to create() after pulling out the "strategy" keyword.

if you dont like having to send over "strategy" i can add a hook in
there to look it up on the dialect, so it could be more like
create_engine('ndbapi://whatever'). but anyway this method would
mean we wouldnt have to rewrite half of Engine's internals.

[Monty Taylor]

On 3/15/07, Michael Bayer <mik...@zzzcomputing.com> wrote:
>
> well at least make a full blown patch that doesnt break all the other
> DB's. notice that an Engine doesnt just execute Compiled objects, it
> can execute straight strings as well. thats why the dialect's
> do_execute() and do_executemany() take strings - they are assumed to
> go straight to a DBAPI representation. to take the "stringness" out
> of Engine would be a large rework to not just Engine but all the
> dialects.
>
> im surprised the execute_compiled() method works for you at all, as
> its creating a cursor, calling result set metadata off the cursor,
> etc. all these DBAPI things which you arent supporting. it seems
> like it would be cleaner for you if you werent even going through
> that implementation of it.

Well, I was trying my best to be a good citizen, so I made some
classes that implement all of the methods that the pieces of
execute_compiled seemed to want, faking it for now when I didn't need
it. The semantics of most of the DBAPI map to the NDBAPI fairly well
(it's still all the same underlying db ideas - just no sql strings)

BUT...

> the theme here is that the base Engine is assuming a DBAPI
> underneath. if you want an Engine that does not assume string
> statements and DBAPI's it might be easier for you to just provide a
> subclass of Engine instead (or go even lower level, subclass
> sqlalchemy.sql.Executor). either way you can change what
> create_engine() returns by using a new "strategy" to create_engine(),
> which is actually a pluggable API. e.g.

This seems like what I really want to try, because you are right,
trying to get this to pretend to be totally DBAPI is going to be not
totally fun. I'll see what trouble I get myself into this way... or
I'll send a patch that changes all the internals. :)

Thanks!

[Monty Taylor]

YES. This is good stuff.

Thanks again.