allowing migrate commands to explicitly close engines

[Michael Bayer]

While trying to get the test suite running for Oracle XE, I was
getting errors related to connection resources. Ultimately the issue
is an Oracle configuration one which is solved using the recipe at
http://forums.oracle.com/forums/thread.jspa?messageID=1145120 .

However, I did observe that migrate's test suite opens a lot of
connections when it performs the "Shell" tests, and this is due to the
many calls to create_engine(). I worked up a patch which allows each
api call to explicitly dispose() the engine that it creates, using a
decorator to do the work which should preserve the full original
argument/doc signature for each function for docgen purposes. In the
usual migrate use case of one command per script invocation it's not
really needed, but in the use case where an application scripts many
api.py calls in one process it is probably worthwhile. Note that
the technique of "exec"ing a code string to create a decorator , while
a little ugly, is the most effective way to go and its what the
decorator module does too (as does SQLA in one or two places).

The patch also explicitly places the StaticPool into create_engine()
calls to ensure each engine is essentially a "single connection",
which seems to be how migrate is using create_engine().

Anyway, I will leave the adoption of this patch up to the migrate
devs. The StaticPool usage and the dispose decorator should be
considered separately as they are not dependent on each other.

http://paste.pocoo.org/show/127874/

[Domen Kožar]

Hey,

first I should warn you that migrate switched to Mercurial and your
code is not really up to date.

Your post gives another approach how to touch "too much connection in
unittests" question, and I like the solution. I will apply StaticPool
patch tomorrow. About construct_engine, I would like to wait a while.
Maybe we should just put try/finally in inner code of those api
functions that construct engine. It wouldn't be hard to maintain.
Somehow this approach seems to much "hackish" to me. Also note that
latest tip provides option to pass in already constructed engine and
users may not want it to be cleared.

[Michael Bayer]

On Jul 11, 8:49 pm, Domen Kožar <iElect...@gmail.com> wrote:
> Maybe we should just put try/finally in inner code of those api
> functions that construct engine. It wouldn't be hard to maintain.

well if you're going to repeat something many times, at least it
shouldn't be an implementation that's repeated. Here's another
method:

def db_version(url, repository, **opts):
schema = with_engine(url, opts,
ControlledSchema,
repository
)
return schema.version

where with_engine is:

def with_engine(url, opts, fn, *args, **kw):
engine = create_engine(url, **opts)
try:
return fn(engine, *args, **kw)
finally:
engine.dispose()

since I'm assuming the "hackish" part is the decorator generated from
a string. Take a look at the source to decorator.py, they're doing
the same thing.

> Also note that
> latest tip provides option to pass in already constructed engine and
> users may not want it to be cleared.

thats a good feature, and you should make sure it can take in a
connection as well as an engine, so I can do many operations within a
single transaction (PG has CREATE/ALTER statements participate in
transactions).

I've put up a new patch on the tracker that finishes off firebird
support and adds support for a name change of "postgresql" in 0.6.
Oracle tests also pass 100% with 0.6 + migrate.