On the limitations of monotone version numbers
Yannick Gingras
Greetings Migrators,
I really like SQLAlchemy-Migrate but I have to mention that I foresee
some problems with the current versioning scheme. Maybe this was
discussed before but let me develop and propose and alternate
numbering strategy.
My comment is based on my experience working in a branch heavy
environment. Indeed, the rise of DVCS has allowed many groups and
organizations to switch from centralized development to something that
is more fork intensive, with feature branches that can stay on the
side of the main maintenance branch for several releases until it's
ready for production.
What we did for internal development was not unlike what you can see
in many projects now hosted on BitBucket and on GitHub. We would have
maintenance releases every few weeks and major feature that could not
be complete in one iteration would be developed in a forked code
branch on the side with regular merge from the maintenance branch into
the feature branch but not the other way around until the feature was
ready for delivery. The benefit of that is that we can release
anytime and that unexpected problems with the implementation of the
new feature can't delay the release of the maintenance improvements.
Now this is not very special as a development strategy but I think
it's worth mentioning because every developer in our group had his own
database and we had a custom schema migration system that did work
pretty well with our development model.
Before I develop on the proposed solution, let's see where monotone
(sequential) versions fail with code branches. A typical case is for
a new feature to mandate the addition of a new a column, which
requires a changes script, so a developer runs something like the
following in the feature branch:
./migrate.py script "new column for Frobnoz"
./migrate.py version
23
But in the mean time, another developer closes a a ticket in the
maintenance branch by increasing the size of a varchar column:
./migrate.py script "larger col for employee name"
./migrate.py version
23
There you see the problem, when the maintenance branch is merged into
the feature branch, you have two scripts with the same version number
in the repository:
023_new_column__for_Frobnoz.py
023_larger_col_for_employee_name.py
Migrate will detect this clash and raise an exception but the
resolution requires some tedious manual work.
One way to prevent that problem is to use dependencies instead of
version numbers. Using dependencies has its own problem because it
introduces more record keeping for the developer but it scales well
to branch based development.
A simple dependency scheme is to follow release numbers and to assume
that the order in which you apply change scripts does not matter
within a given version. Let's say that the latest released version is
1.5. Then the feature developer would recored that his change script
applies only to 1.5, so would the maintenance developer. To record
that the change script is applied or not in the database, you can use
the script name or the hash of the full script in the migrate_version
table. If the feature it not released when 1.6 rolls out, the feature
developers will get an error message when he tries his change script
on the latest prod snapshot. It will then be his job to make sure
that his script still applies clean on 1.6 then to change his
dependency meta-infos or to modify his script to make it compatible
with 1.6.
Should Migrate move do dependency based numbering? Maybe not, but it
should certainly move to a numbering scheme that plays better with
branches than the current monotone numbering scheme. Maybe there's a
better way to do it but my experience shows that dependencies work
well and that it provides enough meta information to track and debug
migration errors.
What do you guys think about it?
--
Yannick Gingras
http://ygingras.net
http://confoo.ca -- track coordinator
http://montrealpython.org -- lead organizer
Domen Kožar
I fully understand the issue and support your point. Linear versioning
is way beyond today's usecases.
This is one of many things that needs to be recatored in migrate (and
other parts like CLI and Python API, scheming, data population.. etc)
I would say there is no perfect solution for all usecases and migrate
could support more than one versioning strategy. Of course, once
repository is created there is not turning back.
I currently still do not have any time to work on migate, but patches
are welcome.
Cheers, Domen
On Dec 28 2009, 6:39 pm, Yannick Gingras <yging...@ygingras.net>
wrote:
> signature.asc
> < 1KViewDownload
Domen Kožar
unfortunately my financial status is not stable enough for visiting USA
for a week:)
We can still use some communication app and try doing some specs over
the web.
Greetings, Domen
On Wed, 2010-01-27 at 13:28 -0500, Yannick Gingras wrote:
> On January 27, 2010, Domen Kožar wrote:
> > I would say there is no perfect solution for all usecases and migrate
> > could support more than one versioning strategy. Of course, once
> > repository is created there is not turning back.
> >
> > I currently still do not have any time to work on migate, but patches
> > are welcome.
>
> Hello Domen,
> are you going to PyCon? It would be nice to brainstorm on that idea a
> bit. I might have time to work on a solution if we get some good specs.
>
Yannick Gingras
> could support more than one versioning strategy. Of course, once
> repository is created there is not turning back.
>
> I currently still do not have any time to work on migate, but patches
> are welcome.
Hello Domen,
are you going to PyCon? It would be nice to brainstorm on that idea a
bit. I might have time to work on a solution if we get some good specs.
--
Diez B. Roggisch
> I'm still studying Electrical Engineering in Slovenia, Europe and
> unfortunately my financial status is not stable enough for visiting USA
> for a week:)
>
> We can still use some communication app and try doing some specs over
> the web.
We currently implement our own migration system - sqlalchemy-migrate wasn't
suitable for reasons to lengthy to discuss here right now.
However, we also develop based on feature-branches, creating similar
situations as the ones you sketched.
What we do now is to use the SVN-branch-revision as numbering scheme. So,
essentially it's a global sequence counter we rely on.
So maybe if you make the revision naming pluggable, you could use this scheme
as well, amongst others. I don't know enough about mercurial (let alone other
dvcs') to really assess if using e.g. changeset-names (which are
un-orderable, but unique) works as well.
Diez
Domen Kožar
- revision based (can be genarated, like: current implementation with
numbering, SVN revision, datetime stamp, etc)
- strict (each migration has specified FROM state and TO state)
- dependency based
It would be great to write some basic specs and implement that in
separate branch for migrate. It would eliminate the need for
individuals to reinvent the wheel for themselfs and choose one of
supported backends or just write their own.
Cheers, Domen