automating deployments, recording syncdb
Carl Meyer
I've done some work on automating deployments for projects using South;
there's one sticky issue I've run into. When updating the deployment of
a project with a new app which has migrations, none of which have yet
been applied on that deployment, there are two possible cases:
1) The app is new to the project since this deployment was last updated,
and has therefore never even been synced to this database. All its
migrations should be run.
2) The app was previously synced, but has only recently been converted
to South. Its initial migration should be faked, and the rest run normally.
Currently, South's MigrationHistory table (and by extension the output
of migrate --list) provide no information to distinguish those cases, so
I have to record the list of synced apps elsewhere, adding an extra
state file to each deployment.
Since South overrides Django's syncdb command with its own, it seems
like "which apps have already been synced sans migrations?" would be a
reasonable question for South to be able to answer natively. This could
take the form of a new model to South's models.py, or could be done via
some "magic" value for the migration field in the existing
MigrationHistory table (would have to be something that is not a valid
migration name; I'm not sure what the options would be).
In terms of UI, I'd been thinking in terms of an extra marker in the
output of migrate --list, but am not set on any particulars.
Or more ambitious: having access to this data could allow South's
convert_to_south command to be intelligent, faking or not-faking the
initial migration as appropriate. Or one step more ambitious yet:
convert_to_south could be ditched entirely, and the migrate command
itself could behave intelligently if no migrations have been run for an app.
Am I missing something? Is this a direction you'd be open to, Andrew?
Carl
Andrew Godwin
solution that correctly distinguished between "this app has been
converted" and "this app is newly installed" that didn't use database
introspection.
The essence of the problem is that both these states are very similar;
in both, there are no records of the migrations having been run, and
South has just been installed (so there's nothing it could have tracked
before). Currently, South defaults to believing it's just been installed
but migrations need to be run.
If you can come up with a concrete proposal that correctly separates
these two cases, I'll happily include it. Last time, it was harder than
it looked...
Andrew
Carl Meyer
Andrew Godwin wrote:
> The essence of the problem is that both these states are very similar;
> in both, there are no records of the migrations having been run, and
> South has just been installed (so there's nothing it could have tracked
> before). Currently, South defaults to believing it's just been installed
> but migrations need to be run.
Yes, I see. In outlining my two cases I did not assume that "South has
just been installed (so there's nothing it could have tracked before);"
I assumed the opposite. Our projects always have South installed from
the beginning, but we don't bother creating migrations for a given app
until we need a schema change. (Perhaps this is unusual, and we ought to
just be creating at least a single initial migration for every app from
the beginning). So in our case, it would be very useful if South's
syncdb could track when it synced a migration-less app.
When South itself is first synced, I agree that the assumption it
currently makes about apps with migrations (that all migrations need to
be run) is the best default assumption, and faking must be done manually.
Although... what's actually so bad about database introspection?
Django's syncdb already uses it to figure out what needs to be synced,
and database backends provide API for it. Why can't South just use this
API and assume that if some tables already exist for an app, the initial
migration should be faked?
Anyway, I'll work on all this when I find some time and see what I come
up with.
Carl
Andrew Godwin
> Yes, I see. In outlining my two cases I did not assume that "South has
> just been installed (so there's nothing it could have tracked before);"
> I assumed the opposite. Our projects always have South installed from
> the beginning, but we don't bother creating migrations for a given app
> until we need a schema change. (Perhaps this is unusual, and we ought to
> just be creating at least a single initial migration for every app from
> the beginning). So in our case, it would be very useful if South's
> syncdb could track when it synced a migration-less app.
>
Right, I see what you mean; here, we always convert everything once we
install South, and use migrations from the beginning (that way,
migrations are always always used to create app tables, which I find
reassuring).
I don't really want to encourage this approach; convert_to_south is
intended to be used when you have to move your legacy code over, not for
every app you make from now on.
> When South itself is first synced, I agree that the assumption it
> currently makes about apps with migrations (that all migrations need to
> be run) is the best default assumption, and faking must be done manually.
>
Indeed; even if it's not the most common case (and I suspect it is),
it's certainly the one South _should_ be doing.
> Although... what's actually so bad about database introspection?
> Django's syncdb already uses it to figure out what needs to be synced,
> and database backends provide API for it. Why can't South just use this
> API and assume that if some tables already exist for an app, the initial
> migration should be faked?
>
There's just something about it I object to. Firstly, I've already
identified one bug in Django's table introspection code that causes
false negatives, and I'm not sure there's an easy fix.
Still, if South detects that only some of the tables are there, then it
knows that you're not only converting, you're converting wrongly (you
made changes before converting, not after). But that could provide false
reliance on that feature, since we can't do the same for any other type
of change.
Basically, it's more of a design question. Do I really want to ship a
whole extra lot of code, which will have to have different variants for
every database backend (or at the very least tie into parts of Django
core that aren't guaranteed to stay in the same place), in order to
promote a usage pattern that I don't want to encourage? I'm personally
still not sure of the answer.
Still, I can see there being a case for tracking syncdb's run while
South is installed (and then, potentially, when South itself is
installed, pre-populating that tracker with app names somehow). I'll
wait to see what you come up with.
Andrew
Carl Meyer
Andrew Godwin wrote:
> I don't really want to encourage this approach; convert_to_south is
> intended to be used when you have to move your legacy code over, not for
> every app you make from now on.
Understood. While we could start making migrations right away for all
our apps, I don't feel I can write automated deployment scripts that
assume this will always be the case. For just one case, think about
transitions from not-having-migrations to having-migrations in
third-party apps. ISTM that when South has been installed long enough to
have the info it needs to handle this transition correctly and reliably,
there's no reason it shouldn't do so.
> Basically, it's more of a design question. Do I really want to ship a
> whole extra lot of code, which will have to have different variants for
> every database backend (or at the very least tie into parts of Django
> core that aren't guaranteed to stay in the same place), in order to
> promote a usage pattern that I don't want to encourage? I'm personally
> still not sure of the answer.
Fair enough; I don't feel strongly about using introspection for this.
> Still, I can see there being a case for tracking syncdb's run while
> South is installed (and then, potentially, when South itself is
> installed, pre-populating that tracker with app names somehow). I'll
> wait to see what you come up with.
Good; that's the direction I'm leaning. AFAICS to maintain similar
behavior to what South currently does, a new installation of South would
prepopulate the "synced" marker to True for apps without migrations and
False for apps with migrations (so all migrations are run by default).
Carl
philz
or may not be an upgrade from a pre-south world. Right now I'm just
placing a try/except around the initial forwards() method, which seems
to get the job done. Does anyone have introspection code that they'd
like to share?
Cheers,
-- Philip
On Apr 1, 8:39 am, Carl Meyer <c...@oddbird.net> wrote:
> Andrew Godwin wrote:
> > I don't really want to encourage this approach; convert_to_south is
> > intended to be used when you have to move your legacy code over, not for
> > every app you make from now on.
>
> Understood. While we could start making migrations right away for all
> our apps, I don't feel I can write automated deployment scripts that
> assume this will always be the case. For just one case, think about
> transitions from not-having-migrations to having-migrations in
> third-party apps. ISTM that when South has been installed long enough to
> have the info it needs to handle this transition correctly and reliably,
> there's no reason it shouldn't do so.
>
> > Basically, it's more of a design question. Do I really want to ship a
> > whole extra lot of code, which will have to have different variants for
> > every database backend (or at the very least tie into parts of Django
> > core that aren't guaranteed to stay in the same place), in order to
> > promote a usage pattern that I don't want to encourage? I'm personally
> > still not sure of the answer.
>
> Fair enough; I don't feel strongly about usingintrospectionfor this.
Andrew Godwin
try selecting from a tell-tale table and see if it fails (some of the
South unit tests use this to see if tables aren't there).
I'd not thought the issue of possibly-Southed installs was a big one -
South was born out of an environment of co-ordinated, single-company
development teams - but it seems there's certainly some edge cases.
Andrew
You received this message because you are subscribed to the Google Groups "South Users" group.
To post to this group, send email to south...@googlegroups.com.
To unsubscribe from this group, send email to south-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/south-users?hl=en.
Philip Zeyliger
I've not seen any in particular, but you could at the very least just try selecting from a tell-tale table and see if it fails (some of the South unit tests use this to see if tables aren't there).
I'd not thought the issue of possibly-Southed installs was a big one - South was born out of an environment of co-ordinated, single-company development teams - but it seems there's certainly some edge cases.
Andrew Godwin
>
>
> On Thu, Apr 15, 2010 at 3:51 PM, Andrew Godwin <and...@aeracode.org
> <mailto:and...@aeracode.org>> wrote:
>
> I've not seen any in particular, but you could at the very least
> just try selecting from a tell-tale table and see if it fails
> (some of the South unit tests use this to see if tables aren't there).
>
>
> Yep. The worry I have is reliably figuring out whether it's failing
> because the database isn't configured properly or because the table's
> not there. I'm guessing that every backend will throw a different
> exception in this case.
every backend throws a different kind of exception, and they can vary
based on what the error is. It's not a fun game.
>
>
> I'd not thought the issue of possibly-Southed installs was a big
> one - South was born out of an environment of co-ordinated,
> single-company development teams - but it seems there's certainly
> some edge cases.
>
>
> I think it comes about because, in a typical application, you might
> not have thought (or researched Django's support thereof) about schema
> migration until you had to do your first one... And then, boom!
migration at conferences. However, I know I'll never convert everyone.
Still, as long as you convert all of your codebase's installations at
once, there shouldn't be too much hassle - the key is to do it all at
once, as that way you can run "./manage.py --fake appname initial" on
all the installations and thus know they've all got South enabled.
It becomes more of a problem if you're distributing software, but my
recommended approach there is to switch over entirely in one release and
make sure it's obvious in the release notes.
Andrew