integrate into or reset existing project
Aljosa Mohorovic
something, like drop table or column, it has no option (afaik) to
reset/re-sync/update evolution data with current database.
anybody has any tips on how to handle this situation?
Aljosa Mohorovic
Russell Keith-Magee
You are correct - there is not currently a way to reset the evolution
data. This issue is currently logged as ticket 65.
If you're interested in contributing, a patch to implement this
feature would be gratefully accepted, and shouldn't be too hard to
implement. It would need to be a new management command; the internals
for that command could be largely derived from the behaviour that is
implemented for the initial syncdb when the signature is initially
established. This code lives in
django_evolution/management/__init__.py.
Yours,
Russ Magee %-)
Derek Anderson
maintain an additional third schema representation:
http://code.google.com/p/deseb/
derek
Russell Keith-Magee
>
> aljosa, deseb does exactly this already, by not requiring you to
> maintain an additional third schema representation:
>
> http://code.google.com/p/deseb/
For those that are reading this thread and don't know the history -
Derek was the Google Summer of Code 2006 student that worked on Schema
Evolution. When I reviewed his code, I rejected it because of some
fundamental flaws in the way it represented the evolution process. A
long discussion followed, and no progress was made. The endpoint of
the discussion was that Ben and I developed and released Django
Evolution, and Derek started deseb. If you want the full details on
the problems I found with deseb, I suggest you read the
django-developers archives.
I realize that this puts me in an awkward situation (as a Django core
developer and a founder of Django Evolution),
but my advice is the following: If you are looking for an alternative
to Django Evolution, I heartily encourage you to look at South
(written by Andrew Godwin) or dbmigrations (written by Simon
Willison). Each of these implementations have their own strengths and
weaknesses, but they are not affected by the same fundamental problems
that plague deseb. In particular, I would strongly contest the
assertion that deseb will fix Aljosa's problem.
However, I am just one man, and this is just my opinion. If anyone
wants to look into deseb, feel free. I won't be offended.
Yours,
Russ Magee %-)
Derek Anderson
evolution should be handled. but i backed down when you pulled rank,
and changed my work into a stand-alone project. why do you keep
referring to it as "fundamentally flawed" and "plagued by" and such,
when it's just a base architectural idea different from yours? i don't
understand your continued hostility towards my tiny little project when
i've not pushed for its integration into django for over 2 years.
aljosa, your question about how to mix manual db changes with evolution
changes was use-case #1 of deseb, and why i designed my project around
introspection. (the cornerstone of russ's dissatisfaction) the big
selling point is that deseb looks at your models and your existing
schema (no matter how you got them there), and offers SQL that will
migrate your schema to match your models. so there is no internal
evolution data to have to resync. which sounds to me like it would
solve your problem, and why i replied to your email.
but of course, whether it does or not is up to you to decide. :)
ttyl,
derek
Russell Keith-Magee
>
> russ, yes, we have some (well documented) differences on how schema
> evolution should be handled. but i backed down when you pulled rank,
> and changed my work into a stand-alone project. why do you keep
> referring to it as "fundamentally flawed" and "plagued by" and such,
> when it's just a base architectural idea different from yours?
Derek - when a "base architectural idea" is incapable of representing
a class of problems that are fundamental to the underlying problem
that needs to be solved, I call the "base architectural idea"
fundamentally flawed.
To state my position - yet again: the fundamental architectural flaw
of deseb, and the flaw that you have repeatedly ignored or denied in
favour of the introspection straw-man, is that A->B->C != A->C. That
is, if you migrate from model definition A, to definition B, to
definition C, it is *not* sufficient, in the general case, to just
migrate from A directly to C. You can't eliminate the intermediate
state as part of the migration process. The trivial case that
demonstrates this is a migration from A->B->A. In your original
proposals, this is "no change to the database"; in reality, it could
represent significant data migration requirements.
The A->B->C problems are then exacerbated by the fact that deseb
doesn't maintain a full audit history. If a developer has an old
checkout of the code at point A, and does an update to point C, their
checkout will not indicate that state B ever existed, because the way
you represent database changes (as annotations in the Meta block) are
destructive to old database states. Sure, they are contained in SVN
history, but nothing forces a developer to check out the intermediate
state.
> i don't
> understand your continued hostility towards my tiny little project when
> i've not pushed for its integration into django for over 2 years.
I'd be interested to know what you consider hostility. If I say "I
started Django Evolution because IMHO deseb was broken by design",
that isn't hostility - it's the truth. I have been interviewed a few
times now about Django Evolution, and when asked, I have given the
history of the Django Evolution project. Each time, I have mentioned
deseb, and said that in my opinion I considered it flawed, people
should read the mailing list discussion if they want to know why, and
Django Evolution was my way of stopping a mailing list discussion that
was going nowhere.
It isn't hostile to point out flaws, either. If someone asks me why I
don't use the approach proposed by deseb, I can and do enumerate the
specific technical problems I have with your approach.
I have also I have been extremely careful to point out that my dislike
for the approach proposed by deseb is my own opinion.
Which part exactly is being hostile?
> aljosa, your question about how to mix manual db changes with evolution
> changes was use-case #1 of deseb, and why i designed my project around
> introspection. (the cornerstone of russ's dissatisfaction)
Really? Introspection is the cornerstone of my dissatisfaction? Hrm....
http://groups.google.com/group/django-developers/browse_thread/thread/72fe2d0061c73fab#cb8e4d0d7c0176e5
http://groups.google.com/group/django-developers/browse_thread/thread/72fe2d0061c73fab#9a39f11959c6d3b1
http://groups.google.com/group/django-developers/browse_thread/thread/72fe2d0061c73fab#f4e4860943ee74e1
All of these messages come from a _single thread_ on
django-developers. If you widen your search, I'm sure you'll find more
posts that make exactly the same point. In these messages I state
quite explicitly that introspection is not the cause of my problem
with your proposal, and that introspection has a place in any
comprehensive evolution scheme. Introspection even has a place in
Django Evolution - if you don't believe me, check out:
However, I do think introspection is incredibly hard to get right
(exhibit A: manage.py inspectdb), and there is a class of data
migrations that introspection *cannot* identify completely and
correctly (e.g., migrate all text in the 'name' column to upper case;
change the 'area' field from acres to hectares).
As a result of these complications, Ben and I chose not to make
introspection the core of the Django Evolution approach, and we
deferred actually implementing introspection to a later version. Yes,
this imposes (for the moment) the restriction that you can't manually
modify the database. As a value judgement, I don't see this as a huge
limitation. On a large project, you don't want people randomly
tweaking their databases. You want them to document everything they
do, and a formal evolution history is as good a place as any to do
this. If, as in Aljosa's case, there is a legitimate reason to perform
manual database updates, the limitations of having a database-based
signature can be overcome with a relatively simple management command
that nudges the stored signatures. This is the subject of Django
Evolution ticket #65: a ticket that I will openly admit is
outstanding, but this is due to my schedule being busy rather than any
technical limitation.
The problem I have with deseb is entirely derived from the A->B->C
problem, coupled with the problems inherent in the way changes are
represented.
Now - just so I don't get accused of hostility: I have no personal
animosity towards Derek or his project - I merely maintain the
personal opinion that his project is based on some flawed design
decisions. I heartily encourage anyone interested to go and look for
themselves and make up their own mind.
Yours,
Russ Magee %-)
Derek Anderson
stored evolutions 2 years ago. which *are* an audit history, stored and
used automatically, by default. plus they give you free cryptographic
hashing of your schemas to verify that you deploy the exact database the
primary developer wishes you to have. (again, automatically) if this is
truly what your concern is "entirely derived" from, rest easy.
btw, for those who haven't read the threads, the A->B->C problem, even
unattended to, could only ever lead to a database retaining data; never
losing it. (and only when your schema evolution was cyclic) i still
maintain this is a problem only in the broadest sense of the word.
about all the other stuff you wrote, sorry if i misinterpreted the tone
of your email.
derek
Russell Keith-Magee wrote:
> blah
Russell Keith-Magee
>
> russ, your "A->B->C problem" was solved when i finished implementing
> stored evolutions 2 years ago. which *are* an audit history, stored and
> used automatically, by default. plus they give you free cryptographic
> hashing of your schemas to verify that you deploy the exact database the
> primary developer wishes you to have. (again, automatically) if this is
> truly what your concern is "entirely derived" from, rest easy.
Well, that's great to hear. I'll have to look into your code again and
see what has changed.
By the way, you might want to check your calendar. The thread I was
referencing yesterday was dated August 2007. That's only slightly over
1 year ago, not 2 years ago. The last version of your code that I
looked at was from just prior to the time of that thread; at that
time, you hadn't implemented stored evolutions, and deseb still had
all the problems that I have documented here.
> btw, for those who haven't read the threads, the A->B->C problem, even
> unattended to, could only ever lead to a database retaining data; never
> losing it. (and only when your schema evolution was cyclic) i still
> maintain this is a problem only in the broadest sense of the word.
Again, this is incorrect. Go back and read the thread. It can also
affect A->B->C transitions without cycles. More importantly, it
doesn't just lead to mistakenly retained data - it can lead to
incorrect and misleading data in the database. The conditions under
which this can happen are not edge cases - they are very easy to
reproduce.
Yours,
Russ Magee %-)
Derek Anderson
the hooks into django. the first thread you linked to includes our
discussion of fingerprints, versioning and controlled migrations. (i
argued that we should keep them optional to make it easier for
less-db-savvy devs, remember?) and you argued my insistence on
supporting multi-developer, non-linear evolutionary paths (through
directed, acyclic graphs) was too complex. and i argued that that
complexity could be automated...(which i hope now at least you'll
concede to, since it *is* automated) how can you act like "i didn't
know it did that" when we both argued about it till we were blue in the
face *so many times*?
anyway... as i said before, you won. my project is tiny, infrequently
maintained, and with only a handful of regular users. i've barely even
done any web development since i've returned to academia. i don't
believe i'll ever understand why my project drew so much ire from you
(esp. back when it was the "aka" syntax you were so focused on being the
root design flaw), but i've learned to live with it. let's just go back
to our previous, more quiet worldly co-existence.
derek
Russell Keith-Magee
> let's just go back
> to our previous, more quiet worldly co-existence.
Sounds marvelous to me.
Yours,
Russ Magee %-)