alembic: downgrade raises KeyError

[Dirk Makowski]

Hi all,

after upgrading alembic to 0.7.7 and SQLAlchemy to 1.0.8, when I try to downgrade one step from head, I get the below KeyError. The allegedly missing key is the revision ID of head. It is correctly stored in the alembic table, and the file and its predecessor are also present.

Other commands however do find that key:

{{{

$ alembic -c etc/Morrigan/development/alembic.ini heads --verbose

Rev: 32f69b44319 (head)

Parent: 1644537aa7

Path: /home/dm/myprojects/Foo/alembic/versions/32f69b44319_20150714_feat_mdv_mgmt.py

    20150714_feat_mdv_mgmt

    

    Revision ID: 32f69b44319

    Revises: 1644537aa7

    Create Date: 2015-07-20 14:52:40.227436

}}}

and

{{{

$ alembic -c etc/Morrigan/development/alembic.ini show  32f69b44319 

Rev: 32f69b44319 (head)

Parent: 1644537aa7

Path: /home/dm/myprojects/Foo/alembic/versions/32f69b44319_20150714_feat_mdv_mgmt.py

    20150714_feat_mdv_mgmt

    

    Revision ID: 32f69b44319

    Revises: 1644537aa7

    Create Date: 2015-07-20 14:52:40.227436

}}}

But downgrading gives:

{{{

$ alembic -c etc/Morrigan/development/alembic.ini  downgrade  -1

INFO  [alembic.migration] Context impl PostgresqlImpl.

INFO  [alembic.migration] Will assume transactional DDL.

Traceback (most recent call last):

  File "/home/dm/myprojects/atrial-py34-venv/bin/alembic", line 9, in <module>

    load_entry_point('alembic==0.7.7', 'console_scripts', 'alembic')()

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/config.py", line 439, in main

    CommandLine(prog=prog).main(argv=argv)

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/config.py", line 433, in main

    self.run_cmd(cfg, options)

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/config.py", line 416, in run_cmd

    **dict((k, getattr(options, k)) for k in kwarg)

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/command.py", line 193, in downgrade

    script.run_env()

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/script.py", line 390, in run_env

    util.load_python_file(self.dir, 'env.py')

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/util.py", line 244, in load_python_file

    module = load_module_py(module_id, path)

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/compat.py", line 68, in load_module_py

    module_id, path).load_module(module_id)

  File "<frozen importlib._bootstrap>", line 539, in _check_name_wrapper

  File "<frozen importlib._bootstrap>", line 1614, in load_module

  File "<frozen importlib._bootstrap>", line 596, in _load_module_shim

  File "<frozen importlib._bootstrap>", line 1220, in load

  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked

  File "<frozen importlib._bootstrap>", line 1129, in _exec

  File "<frozen importlib._bootstrap>", line 1471, in exec_module

  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed

  File "alembic/env.py", line 97, in <module>

    run_migrations_online()

  File "alembic/env.py", line 92, in run_migrations_online

    context.run_migrations()

  File "<string>", line 7, in run_migrations

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/environment.py", line 738, in run_migrations

    self.get_context().run_migrations(**kw)

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/migration.py", line 302, in run_migrations

    for step in self._migrations_fn(heads, self):

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/command.py", line 182, in downgrade

    return script._downgrade_revs(revision, rev)

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/script.py", line 315, in _downgrade_revs

    current_rev, destination)

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/revision.py", line 512, in iterate_revisions

    inclusive, assert_relative_length

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/revision.py", line 450, in _relative_iterate

    inclusive=inclusive, implicit_base=implicit_base))

  File "/home/dm/myprojects/atrial-py34-venv/lib/python3.4/site-packages/alembic/revision.py", line 672, in _iterate_revisions

    total_space.remove(rev.revision)

KeyError: '32f69b44319'

}}}

Thanks for your help,

Dirk

[Mike Bayer]

can you confirm no issue with Alembic 0.7.6?    Can you please send along all of your migration scripts; feel free to erase everything but the version headers and empty upgrade() / downgrade() functions, I just need to see the structure you're working with.  thanks.

--
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[Dirk Makowski]

I had do revert back to alembic 0.6.7 for the down-migration to work again (upgraded today from 0.6.6). Please find attached the bare scripts.

[Mike Bayer]

On 8/3/15 4:59 PM, Dirk Makowski wrote:

Hi all,

after upgrading alembic to 0.7.7 and SQLAlchemy to 1.0.8, when I try to downgrade one step from head, I get the below KeyError. The allegedly missing key is the revision ID of head. It is correctly stored in the alembic table, and the file and its predecessor are also present.

I'm not able to reproduce any issue.   I'm using Python 3 like you are and even turning on PYTHONHASHSEED=random, I can upgrade, downgrade, -1, +1, whatever, does everything perfectly.   This is of course with a plain vanilla env.py file as i don't have your special libraries available.

Are you able to reproduce your issue using a brand new Alembic environment and using just these version files ?  there is nothing unusual here and you have a perfectly linear ordering.   The error message indicates the internal organization of revisions as it places them into dependency order is failing but I can't see how that would happen here.     Also, the 0.7 series has been out for months and if the revision logic were capable of failing on a perfectly straight line of files without even any branch or mergepoints, it seems like that would have been noticed.  I can simulate an internal condition that produces this stack trace, but I don't see how this condition can occur.

Also, what is the *exact* version of Python 3 you're using ?  Are there any .pyc files from the older Alembic 0.6.7 lying around in the virtual environment or is this a fresh environment ?

[Mike Bayer]

On 8/3/15 6:51 PM, Mike Bayer wrote:

On 8/3/15 4:59 PM, Dirk Makowski wrote:

Hi all,

after upgrading alembic to 0.7.7 and SQLAlchemy to 1.0.8, when I try to downgrade one step from head, I get the below KeyError. The allegedly missing key is the revision ID of head. It is correctly stored in the alembic table, and the file and its predecessor are also present.

I'm not able to reproduce any issue.   I'm using Python 3 like you are and even turning on PYTHONHASHSEED=random, I can upgrade, downgrade, -1, +1, whatever, does everything perfectly.   This is of course with a plain vanilla env.py file as i don't have your special libraries available.

Are you able to reproduce your issue using a brand new Alembic environment and using just these version files ?  there is nothing unusual here and you have a perfectly linear ordering.   The error message indicates the internal organization of revisions as it places them into dependency order is failing but I can't see how that would happen here.     Also, the 0.7 series has been out for months and if the revision logic were capable of failing on a perfectly straight line of files without even any branch or mergepoints, it seems like that would have been noticed.  I can simulate an internal condition that produces this stack trace, but I don't see how this condition can occur.

Also, what is the *exact* version of Python 3 you're using ?  Are there any .pyc files from the older Alembic 0.6.7 lying around in the virtual environment or is this a fresh environment ?

OK.   Here's what will cause this exactly:

sqlite> select * from alembic_version;
32f69b44319
32f69b44319

duplicate revisions in alembic_version.  Do you have that going on?  If so, was that via manual intervention?  I'll add a protection for this that will prevent the issue but the alembic_version table should not have duplicate rows in it.

[Mike Bayer]

On 8/3/15 7:02 PM, Mike Bayer wrote:

On 8/3/15 6:51 PM, Mike Bayer wrote:

On 8/3/15 4:59 PM, Dirk Makowski wrote:

Hi all,

after upgrading alembic to 0.7.7 and SQLAlchemy to 1.0.8, when I try to downgrade one step from head, I get the below KeyError. The allegedly missing key is the revision ID of head. It is correctly stored in the alembic table, and the file and its predecessor are also present.

I'm not able to reproduce any issue.   I'm using Python 3 like you are and even turning on PYTHONHASHSEED=random, I can upgrade, downgrade, -1, +1, whatever, does everything perfectly.   This is of course with a plain vanilla env.py file as i don't have your special libraries available.

Are you able to reproduce your issue using a brand new Alembic environment and using just these version files ?  there is nothing unusual here and you have a perfectly linear ordering.   The error message indicates the internal organization of revisions as it places them into dependency order is failing but I can't see how that would happen here.     Also, the 0.7 series has been out for months and if the revision logic were capable of failing on a perfectly straight line of files without even any branch or mergepoints, it seems like that would have been noticed.  I can simulate an internal condition that produces this stack trace, but I don't see how this condition can occur.

Also, what is the *exact* version of Python 3 you're using ?  Are there any .pyc files from the older Alembic 0.6.7 lying around in the virtual environment or is this a fresh environment ?

OK.   Here's what will cause this exactly:

sqlite> select * from alembic_version;
32f69b44319
32f69b44319

duplicate revisions in alembic_version.  Do you have that going on?  If so, was that via manual intervention?  I'll add a protection for this that will prevent the issue but the alembic_version table should not have duplicate rows in it.

OK that is https://bitbucket.org/zzzeek/alembic/issues/314/dupe-entries-in-alembic_versions-can-break and that's fixed in the latest master, for 0.8.   I can backport to 0.7 but I'm not sure there's going to be another 0.7 release.   For now you just need to remove the duplicate entries from your alembic_version table.

[Dirk Makowski]

Cool, that's it. I noticed those duplicate entries some time ago, oblivious how they came into existence, but since they had not done any harm kept them. With only one entry, alembic works great again. Thanks a lot.