Invoke commands with custom env.py from library

[Daniel Cardin]

I am attempting to write a library which invokes alembic commands, while referencing the migrations of a separate package which has installed said library.

The intent here, is for the library to invoke the alembic commands with an env.py defined in that package. This seems to work through

config.get("script_location", "package_name:foldername")

but then obviously expects the actual migrations to be colocated at the same location.

My guess would be, if this is possible at all, that there'd be something I could put in the env.py which would reconfigure it to execute the `context.run_migrations()` migration context (and therefore search path, back at the original call site.

I realize that this won't always work, given that env.py is often likely customized enough such that a generic one wouldn't be able to execute them, but per your cookbook suggestions about programmatic invocation of commands, this sort of thing requires the user to opt into changing their env.py to use

connectable = context.config.attributes.get("connection", None)

in order to make use of a connection handed to it.

[Mike Bayer]

I'm not following all the moving pieces here, like when you say "commands", i assume you mean the commands in alembic.command, , like alembic.command.upgrade().   when you say "an env.py defined in that package", I guess you mean in the separate package that is using your library.     

this doesn't seem different from what Alembic itself does, "that package" would have a directory where the env.py and its migration files are present?       So... I'm not really sure what you're trying to do beyond call an Alembic command that is against a particular directory.

if OTOH by "commands" you mean migration operations, like you want to call op.create_table() yourself, OK, but I don't really understand enough to formulate an answer for you.

--

You received this message because you are subscribed to the Google Groups "sqlalchemy-alembic" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy-alem...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-alembic/5612f1d1-9e93-46ed-9be7-0db362b815a8%40googlegroups.com.

[Daniel Cardin]

So, yes I mean "commands" as in `alembic.command.upgrade()`.

The idea would be that the library defines an env.py (e.g. the important portion of which would look something like:)

...

connectable = context.config.attributes.get("connection", None)

with self.connection.connect() as connection:
    alembic.context.configure(connection=connection)

    with alembic.context.begin_transaction():
        alembic.context.run_migrations()

which lives at `src/library/foo/env.py`. If the migrations were colocated with the env.py, then afaik setting "script_location" to "library:foo" would enable me to run `alembic.command.upgrade()` and it would just work. However in this case (if possible), the migrations/versions/* would live at an arbitrary other location (e.g. the code which has installed this library).

General workflow:

* person working on project "foo" invokes some cli/command/whatever which requests an upgrade

* library does whatever requisite setup

* library invokes `alembic.command.upgrade()`

* upgrade() ends up routing the code through the library's `env.py`

* the context of the migration command is targeting the project "foo"'s local migrations/versions folder

The specifics of the above are just based on my knowledge of alembic, so if there's another process i could be doing where env.py isn't "invoked" so much as the above code block is just called normally, then that's ideal.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy-alembic+unsub...@googlegroups.com.

[Mike Bayer]

On Thu, Mar 5, 2020, at 12:02 PM, Daniel Cardin wrote:

So, yes I mean "commands" as in `alembic.command.upgrade()`.

The idea would be that the library defines an env.py (e.g. the important portion of which would look something like:)

...

connectable = context.config.attributes.get("connection", None)
with self.connection.connect() as connection:
    alembic.context.configure(connection=connection)

    with alembic.context.begin_transaction():
        alembic.context.run_migrations()

which lives at `src/library/foo/env.py`. If the migrations were colocated with the env.py, then afaik setting "script_location" to "library:foo" would enable me to run `alembic.command.upgrade()` and it would just work. However in this case (if possible), the migrations/versions/* would live at an arbitrary other location (e.g. the code which has installed this library).

Ok two thoughts on that:

1. usually the way this goes is that the project that has the migrations does everything normally, it has its own env.py, and inside that env.py, that's where it imports *your* env.py.  that is, instaed of having the usual env.py it would only have:

# env.py

from daniel_cardins.library.env import run_migrations

run_migrations(<important arguments>)

2. the locations of the version files vs. the env.py script are actually separate in the config.    there is a script_location that it uses to find the home base where env.py is, however there is also a version_locations config that can have any number of other directories in it and these supersede script_location for the version files themselves.

I think in general, if the end-user has a bunch of version files set up, it's probably not a big deal that they have a stub "env.py" right there that just calls out to your library.  that's something very clear that people looking at a project that uses your library can understand quickly if they already know alembic.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy-alem...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-alembic/598f480c-e977-4ce2-a315-8563aa9944fc%40googlegroups.com.

[Daniel Cardin]

1. I always expected that, with any reasonably non-trivial app, I would inevitably need to fall back to using their env.py. My hope was that for simpler cases, since this is a testing tool, there would be a way to not require any changes to their env.py by default. Ultimately I can document what the recommended changes to a typical env.py would be, and that's not the end of the world.

2. I did try to change version_locations, but unless I'm mistaken what I'm asking for maybe seems like an unsupported usecase. Attempting to target the library env with "script_location" and the local versions/ with "version_locations" still seems to look at the wrong location. Given how ScriptDirectory is constructed and and then `run_env` directly looks for "env.py", it seems like what im asking for is just not an expected usecase.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-alembic/598f480c-e977-4ce2-a315-8563aa9944fc%40googlegroups.com.

[Mike Bayer]

On Thu, Mar 5, 2020, at 1:59 PM, Daniel Cardin wrote:

1. I always expected that, with any reasonably non-trivial app, I would inevitably need to fall back to using their env.py. My hope was that for simpler cases, since this is a testing tool, there would be a way to not require any changes to their env.py by default. Ultimately I can document what the recommended changes to a typical env.py would be, and that's not the end of the world.

2. I did try to change version_locations, but unless I'm mistaken what I'm asking for maybe seems like an unsupported usecase. Attempting to target the library env with "script_location" and the local versions/ with "version_locations" still seems to look at the wrong location. Given how ScriptDirectory is constructed and and then `run_env` directly looks for "env.py", it seems like what im asking for is just not an expected usecase.

it's true, it is an assumption that env.py is co-located with the versioning files, although looking at the source, I don't know that i see why it would look in the wrong location, ScriptDirectory pretty clearly looks in "dir" for env.py and "version_locations" for the version files.

I'm wondering why these users would not have an env.py set up?   or they would, and your tool is attempting to do some magic thing on an existing installation without them making any changes?  If there is truly something blocking ScriptDirectory from working this way, maybe I'd suggest monkeypatching, but script_directory and version_locations look separate to me and are handled separately all the way from ScriptDirectory.from_config().

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy-alem...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-alembic/99a77147-76da-490a-944d-7f2f40f42fd3%40googlegroups.com.