"Crazy" migrations are exactly that

[psada...@gmail.com]

I've read a couple different places now that there are plans in the
works to use DM's property definitions to automagically ALTER the
table as needed to match the existing schema. I cannot see any
benefits to this whatsoever.

Dropping and recreating the tables is the correct solution in
development and test envs. Obviously this isn't the right choice for a
production db, but I would say that guessing the right ALTER TABLE
isn't, either.

From my personal experience with several "mature" Rails/AR projects,
most of the migrations we write this point also involve migrating
data. For example, adding 'firstname' & 'lastname' columns, copying
the data from the 'name' column, then dropping the 'name' column.
There is absolutely no way to guess this requirement automagically,
and no clean way to specify it in the #property method.

DataMapper does need to have some way to migrate schemas, but I'm not
aware of any standalone migration project we could co-opt. I don't
know that should we write our own, if it should belong in data_mapper,
or a separate project that would be closely coupled to DM, use DO, and
follow the same coding guidelines.

I've hacked on the current rails migration system enough to know what
works and what's broken. I'm willing to start on a migration system
that we could use. I have several ideas on how it should work that I'd
be happy to post and discuss, if there's enough interest. If we insist
on using the automagic "crazy" migrations, then I'll probably write my
own migration system anyway, as DM's way would be completely unusable
to me in a live production setting.

If there's enough interest in this, please post here, and we can
discuss any implementation ideas in this thread as well.

Paul

[psada...@gmail.com]

Well, it looks like Sam already started on this (
http://pastie.textmate.org/private/wpxp3zz9d5vtj03fulxo4g ). It looks
like its still incomplete and missing a few required features
(revisions/dependencies, adding/removing indexes, executing raw SQL).
Maybe I need to start hanging out in IRC more :)

Paul

On Jan 6, 12:16 pm, "psadaus...@gmail.com" <psadaus...@gmail.com>
wrote:

[Matthew B Gardner]

I haven't been able to try DataMapper out yet due to a MySQL bug
(ticket pending), but I've been using Og and its evolution is my
favorite thing about it. To help with concerns about control over the
migrations, why not add a migration-level field into setup? For
example, Og's evolution can be none, add-only, or full, which is drop/
add. I think the default is add-only. Og also uses a destroy field,
which will destroy and recreate the database each time, for
development purposes.

-Matt

[psada...@gmail.com]

I dont' have any experience with Og. How does it handle migrating
data?

[Matthew B Gardner]

Pretty much the same way being proposed here, I think. It looks at a
class's property definitions, then checks if there's an existing
table, and then compares the class schema to that table (or creates
the table). It will add/drop columns, depending on the evolution
schema (add-only, full, etc). It won't make any further changes to a
column that already exists though...you'd need to drop the column
through omission or manually in the db if your initial column
attributes change. I'd actually like to see an ORM that allowed
something that dynamic, though I'm not sure how hard it would be -- my
time pretty much limits me to being strictly a user. I don't know if
this answered your questions...let me know if it didn't. :)

-Matt

[Steve V]

On Jan 6, 1:16 pm, "psadaus...@gmail.com" <psadaus...@gmail.com>
wrote:

I'm in full agreement on this. While the crazy migrations thing sounds
kind of neat, it also sounds like a great setup for completely munging
your data. I understand the desire for things to be DRY, but I think
data has some special rules. Migrations are great for this. One
complete little set of atomic changes. All for exactly the reasons you
stated. Sometimes you have to alter the existing data when you change
the schema. It's not just a simple table add, or column size change.
Heck some migrations I have have nothing to do wish changing the
schema, and just deal with changing some pre-existing data values.

Steve

[Daniel N]

For example.

I generate a model and it's basic migration

A few migrations later I add a migration to add data to that model

A few migrations later I add a column, and set that column to have a validates_presence_of

Now I can't replay the migrations forward, since I'm adding data to an invalid model, that doesn't have a hope of being valid during the replay.

Am I the only one that's had this issue or is it a bit more widespread?  I know this isn't really related that much to crazy migrations, but since we're talking about migration systems in general too, I thought it was relevant.

I'd love to hear some thoughts on how to overcome this issue.

Cheers
Daniel

[Adam French]

I've been burned by migrations which add in data. It's better (IMHO)
to create a rake task that creates default data in your database,
rather than to use migrations for this.

===
~Adam
irc: afrench

[jmoses]

I've been burnt by this behavior a few times. It's my main gripe
about AR migrations, and one of the scale tipping future features of
data mapper.

-joj

On Jan 6, 8:09 pm, "Daniel N" <has....@gmail.com> wrote:

> One of the real issues I've found with classic migrations is that models
> change, but when a migration is run forward from 0, the migrations assume
> that it's all good.

[Sam Smoot]

I'd like to support both types of migrations.

The AR-type migrations are now in DM as of last week. They should be
fairly complete as far as I know. They support an AR-like syntax,
along with an alternative tweaked syntax. You can check out the spec
here: http://datamapper.rubyforge.org/svn/trunk/spec/migration_spec.rb

Thanks, -Sam

[psada...@gmail.com]

On Jan 6, 6:09 pm, "Daniel N" <has....@gmail.com> wrote:
> One of the real issues I've found with classic migrations is that models
> change, but when a migration is run forward from 0, the migrations assume
> that it's all good.

The solution to this is to stop using your models in your migrations.
If I have a migration that needs to create, alter or remove a record,
I write the raw SQL INSERT or UPDATE statement to do that. There's no
guarantee that the model will work, or even be around and called the
same thing when the migration is run. If, for some reason, I really
want a model, I write a one-off class, and have it in-line with the
specific migration.

On Jan 6, 6:15 pm, Adam French <afcoo...@gmail.com> wrote:
> I've been burned by migrations which add in data. It's better (IMHO)
> to create a rake task that creates default data in your database,
> rather than to use migrations for this.

We have several migrations that need to alter data as part of a schema
change. A rake task can't be versioned the same way as a migration,
and there's know way to know if the rake task has already been run.
Inserting default data is a different beast altogether.

On Jan 6, 10:15 pm, Sam Smoot <ssm...@gmail.com> wrote:
> I'd like to support both types of migrations.
>
> The AR-type migrations are now in DM as of last week.

Sam, I saw these and started playing with them. They're still missing
a few critical features, like versioning and creating indexes. I
started to work on it yesterday, but the current core DM code doesn't
seem to facilitate the adding of additional indexes after the table
has already been created. The other features also seem difficult to
implement when relying on DM::Mappings::Table & ::Column. I was
thinking of pulling all the sql CREATE TABLE and ALTER TABLE parts
into a migrations module, and having DM::Base#automigrate! use it
instead. What are your thoughts on this?

Paul

[Bernerd Schaefer]

On Jan 6, 2008 1:41 PM, psada...@gmail.com <psada...@gmail.com> wrote:

Well, it looks like Sam already started on this (
http://pastie.textmate.org/private/wpxp3zz9d5vtj03fulxo4g ). It looks
like its still incomplete and missing a few required features
(revisions/dependencies, adding/removing indexes, executing raw SQL).

Executing raw SQL, of course, can be done just as usual: database.query/execute("SQL"). I suppose for compatibility with rails we can add just a simple "execute" method, though it seems almost silly since the DM way is so easy already.

[Tim Lucas]

On Jan 7, 12:09 pm, "Daniel N" <has....@gmail.com> wrote:
>
> One of the real issues I've found with classic migrations is that models
> change, but when a migration is run forward from 0, the migrations assume
> that it's all good.
>

> For example.
>
> I generate a model and it's basic migration
>
> A few migrations later I add a migration to add data to that model
>
> A few migrations later I add a column, and set that column to have a
> validates_presence_of
>
> Now I can't replay the migrations forward, since I'm adding data to an
> invalid model, that doesn't have a hope of being valid during the replay.
>
> Am I the only one that's had this issue or is it a bit more widespread?  I
> know this isn't really related that much to crazy migrations, but since
> we're talking about migration systems in general too, I thought it was
> relevant.
>
> I'd love to hear some thoughts on how to overcome this issue.

Yeah this is something Scott first noticed in the typo project in Nov
05:
http://scottstuff.net/blog/articles/2005/10/31/migrating-in-two-dimensions

and me in Feb 06:
http://toolmantim.com/article/2006/2/23/migrating_with_models

What I've been doing ever since is defining the AR class within the
migration:

class AddTasmania < ActiveRecord::Migration
class State < ActiveRecord::Base; end
def self.up
State.create(:name => "Tasmania", :code => "TAS")
end
def self.down
end
end

-- tim

[psada...@gmail.com]

I finally got around to implementing a clean way of doing migrations.
Right now, its just a standalone project, but can easily be merged
into DM, and replace the existing migrations. Here's a sample of what
a migration script looks like: http://pastie.caboo.se/142573 . In the
case of something like a merb project, the database would already be
set up, and a rake task would snarf migrations/*.rb and load them all.

Advantages of this over other migration implementations:

* No reliance on filename for name or versioning. Its all in the
method signature #migration( version, name ).
* No requirement for unique versions. Migrations with the same
version # are assumed to be independent of each other. Names must be
unique, however, because:
* Keeps track of which migrations have been run by name. Will run any
that have not been run, just not ones greater than the latest version.
This makes it easy for multiple developers.
* Syntax is clean, and extensible.

Not done yet:

* Any kind of #create_table, #add_column helpers. If this gets
acceptance, I can copy the ones in DM when this gets merged.
* A rake task to do this, and integration with the merb_datamapper
plugin.

Feel free to check out the project here: http://theamazingrando.com/dm_migrations.tar.bz2
. If nobody hates it, and noone is too attached to migrations as the
exist in DM now, I'll start working on a patch that would integrate
this into DM itself, rather than this external project. I'm also eager
to hear suggestions for improvement.

Paul

[Luke Sutton]

The fact that it avoids collisions in version numbers is exciting
enough! I've run into that issue when using AR more than once.
Anything that avoids that gets my thumbs up :)

On Jan 24, 7:57 am, "psadaus...@gmail.com" <psadaus...@gmail.com>
wrote:

> I finally got around to implementing a clean way of doing migrations.
> Right now, its just a standalone project, but can easily be merged
> into DM, and replace the existing migrations. Here's a sample of what

> a migration script looks like:http://pastie.caboo.se/142573. In the

[Sam Smoot]

I like it.

Honestly, I like Bernerd's migrations just as much, but you've gone
the extra mile and done the database work as well, so the one-stop
nature is nice. Perhaps you guys could collaborate?

I'm not against multiple ways of handling migrations. It's one area
where I think a little choice and flexibility may make a certain
amount of sense depending on the project. As long as they're well
documented, then I'd be happy to see them added to DM-core.

Of course, if there's no real need for divergent efforts if there's
some common ground to be found, then that's an obvious win as well.

So I leave it up to you guys to sort out. I'll ask Bernerd to get in
touch if you don't spot him first.

But my main concern is that it's well documented. DataMapper in
general suffers from a lack of documentation, but I want to make sure
that migrations in DM are reasonably documented and easy to get
started with.

Thanks, -Sam

On Jan 23, 3:57 pm, "psadaus...@gmail.com" <psadaus...@gmail.com>
wrote:

> I finally got around to implementing a clean way of doing migrations.
> Right now, its just a standalone project, but can easily be merged
> into DM, and replace the existing migrations. Here's a sample of what

> a migration script looks like:http://pastie.caboo.se/142573. In the

[psada...@gmail.com]

I actually wrote up some more specs & documentation for it last night,
and was panning on further documentation today. I set it up in a
public git repo ( git://theamazingrando.com/git/dm_migrations ) if
anyone wants to check it out.

Paul

[MArcin Raczkowski]

Good to know that someone is already working on it.

Anyway my company started working on our own migration system, and i
wanted to announce it when i found this tread, anyway i'll to check
out your code, and add my code if it's not duplicating what you
already done.

greets

[Matthew B Gardner]

Just wondering -- what's the status of "Crazy Migrations"? Still
planned?

-Matt

[psada...@gmail.com]

Yup, I'm working on getting them into dm-more this week.

[sambo99]

Hi there,

I think this is a fascinating discussion and I also think datamapper
rules !

Crazy migrations you say :)

I think there is a way both the crazy migration system can co-exist
with a clean migration system. In my projects this kind of system
would have saved me lots of work and time. How about having a switch
that enables auto migrations (which you enable during dev), then the
system tracks the changes it makes in a migration file, which you
finally merge into your main migration file.

I think this pastie kind of captures the spirit: http://pastie.caboo.se/167687

There are lots of ideas in this pastie, ill try and list them

1. Have a flag somewhere, that enables / disables non-destructive auto
migrations

2. flip the property on its head so instead of :

property :name, :string
its:
string :name

3. More smarts in migrations, the opposite of create_table is
drop_table, so no need to specify up or down for trivial operations
(create_table, rename_column), you still need to be explicit for
drop_column and perhaps some other stuff

4. A new rename_column primitive in migrations

5 A new change_column_type primitive in migrations


What do you people think about this? Does any of this look appealing?

I also think there needs to be a separate system in place for seed
data.. but have not thought it through completely

Cheers
Sam

[sambo99]

also, I meant to explain that auto migrate flag i am proposing a bit
better

when the automigration flag is on, the system will automatically
migrate if there is a discrepancy between the model and the database.
so you dont need to call automigrate! it just happens automatically.
This flag is only meant to be used in dev and is just a shortcut for
creating your migration file, so you dont have to type as much. The
real nice thing about this is

1. I go edit a model and add a column
2. I edit my view to show the column
3. I hit refresh on the the web page and its there

no need for rake db:migrate, creating migration files etc... its a way
cleaner way to work during dev imho, then when I check in I can decide
how to integrate the change.