Auto-migrations not working as expected in 1.0.0.rc3
11 views
Skip to first unread message
Nate Wiger
Jun 1, 2010, 12:31:10 AM6/1/10
to DataMapper
I've been experimenting with DM 1.0.0.rc3 and auto-migrations don't
appear to be working as expected. Specifically, it appears that
modifications to columns are not reflected in the DB if the columns
already exist.
In this example the DB is mysql.
I created a model:
class User
include DataMapper::Resource
property :id, Integer, :min => 0, :max => 2**32-1, :key => true
property :money, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :points, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :num_logins, Integer, :default => 1
end
And ran DataMapper.auto_upgrade! The users table was created as
expected:
mysql> desc users;
+------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+-------+
| id | bigint(10) unsigned | NO | PRI | NULL | |
| money | bigint(10) unsigned | YES | | 0 | |
| points | bigint(10) unsigned | YES | | 0 | |
| num_logins | int(11) | YES | | 1 | |
+------------+---------------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
But then, I tried to increase the size of the ID column, and also
change num_logins to unsigned:
class User
include DataMapper::Resource
property :id, Integer, :min => 0, :max => 2**64-1, :key => true
property :money, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :points, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :num_logins, Integer, :default => 1, :min => 0
end
When I re-ran the auto_upgrade! method though, nothing changed.
However, if I dropped the table and ran auto_upgrade!, I got the
expected table:
mysql> desc users;
+------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+-------+
| id | bigint(20) unsigned | NO | PRI | NULL | |
| money | int(10) unsigned | YES | | 0 | |
| points | int(10) unsigned | YES | | 0 | |
| num_logins | int(10) unsigned | YES | | 1 | |
+------------+---------------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
Notice how id is bigint(20) and num_logins is unsigned.
Thoughts?
Thanks,
Nate
appear to be working as expected. Specifically, it appears that
modifications to columns are not reflected in the DB if the columns
already exist.
In this example the DB is mysql.
I created a model:
class User
include DataMapper::Resource
property :id, Integer, :min => 0, :max => 2**32-1, :key => true
property :money, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :points, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :num_logins, Integer, :default => 1
end
And ran DataMapper.auto_upgrade! The users table was created as
expected:
mysql> desc users;
+------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+-------+
| id | bigint(10) unsigned | NO | PRI | NULL | |
| money | bigint(10) unsigned | YES | | 0 | |
| points | bigint(10) unsigned | YES | | 0 | |
| num_logins | int(11) | YES | | 1 | |
+------------+---------------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
But then, I tried to increase the size of the ID column, and also
change num_logins to unsigned:
class User
include DataMapper::Resource
property :id, Integer, :min => 0, :max => 2**64-1, :key => true
property :money, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :points, Integer, :min => 0, :max => 2**32-1, :default =>
0
property :num_logins, Integer, :default => 1, :min => 0
end
When I re-ran the auto_upgrade! method though, nothing changed.
However, if I dropped the table and ran auto_upgrade!, I got the
expected table:
mysql> desc users;
+------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+-------+
| id | bigint(20) unsigned | NO | PRI | NULL | |
| money | int(10) unsigned | YES | | 0 | |
| points | int(10) unsigned | YES | | 0 | |
| num_logins | int(10) unsigned | YES | | 1 | |
+------------+---------------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
Notice how id is bigint(20) and num_logins is unsigned.
Thoughts?
Thanks,
Nate
Dan Kubb (dkubb)
Jun 1, 2010, 3:10:24 AM6/1/10
to DataMapper
Nate,
> I've been experimenting with DM 1.0.0.rc3 and auto-migrations don't
> appear to be working as expected. Specifically, it appears that
> modifications to columns are not reflected in the DB if the columns
> already exist.
This is actually by design. The auto-upgrade feature is only
*additive*, in that it will add things that don't exist, but it will
not change anything that is present. So it adds new tables and
columns, but does not change any columns that are present.
There will be some work after 1.0 to make auto-upgrading a bit
smarter, but up until now we've been conservative in what it does.
Anytime to loosen a constraint it should be fairly easy to make the
change. In some cases it might even be possible to tighten constraints
if we inspect the data beforehand. (for example, changing CHAR(20) to
CHAR(10) might be possible as long as SELECT MAX(LENGTH(col)) is <=
10) The reason we've avoided this up until now is that it can get
complex quickly, and even a tiny mistake on our end may cause a data
loss. If we do approach this in the future it will have to be
extremely well specced and tested before we'll release it to the
public.
At the moment though I personally don't see auto-upgrading as being
very useful, since there will always be things it can't handle
automatically like renames and deletions. For me it's more a question
between auto-migrating (which is destructive) and classic migrations.
--
Dan
> I've been experimenting with DM 1.0.0.rc3 and auto-migrations don't
> appear to be working as expected. Specifically, it appears that
> modifications to columns are not reflected in the DB if the columns
> already exist.
*additive*, in that it will add things that don't exist, but it will
not change anything that is present. So it adds new tables and
columns, but does not change any columns that are present.
There will be some work after 1.0 to make auto-upgrading a bit
smarter, but up until now we've been conservative in what it does.
Anytime to loosen a constraint it should be fairly easy to make the
change. In some cases it might even be possible to tighten constraints
if we inspect the data beforehand. (for example, changing CHAR(20) to
CHAR(10) might be possible as long as SELECT MAX(LENGTH(col)) is <=
10) The reason we've avoided this up until now is that it can get
complex quickly, and even a tiny mistake on our end may cause a data
loss. If we do approach this in the future it will have to be
extremely well specced and tested before we'll release it to the
public.
At the moment though I personally don't see auto-upgrading as being
very useful, since there will always be things it can't handle
automatically like renames and deletions. For me it's more a question
between auto-migrating (which is destructive) and classic migrations.
--
Dan
Nate Wiger
Jun 1, 2010, 12:41:56 PM6/1/10
to DataMapper
Thanks for the clarifications. Sounds like it was a misunderstanding
on my part.
FWIW, we actually use the ActiveRecord auto_migrations plugin by
pjhyett in Production for all migrations and it works awesome:
http://github.com/pjhyett/auto_migrations That plugin handles
modifications/deletes of existing columns, such as changing sizes,
removing null contraints, etc.
As you said, auto-upgrading can't handle renaming columns, but you
shouldn't be changing column names after the fact IMO. If you chose a
bad name, add an alias in the model, or just get over it. If you live
with this minor restriction, then your life gets infinitely easier
across the board (eg at the controller/view level too).
But I realize my team is in the minority, which is unfortunate for the
community as a whole because auto-migrations are really amazing.
There's no way we could survive without them given how fast we have to
develop. Thanks again for the clarifications on DM's expected
behavior; sounds like our existing toolset is a better fit.
Cheers,
Nate Wiger
PlayStation
on my part.
FWIW, we actually use the ActiveRecord auto_migrations plugin by
pjhyett in Production for all migrations and it works awesome:
http://github.com/pjhyett/auto_migrations That plugin handles
modifications/deletes of existing columns, such as changing sizes,
removing null contraints, etc.
As you said, auto-upgrading can't handle renaming columns, but you
shouldn't be changing column names after the fact IMO. If you chose a
bad name, add an alias in the model, or just get over it. If you live
with this minor restriction, then your life gets infinitely easier
across the board (eg at the controller/view level too).
But I realize my team is in the minority, which is unfortunate for the
community as a whole because auto-migrations are really amazing.
There's no way we could survive without them given how fast we have to
develop. Thanks again for the clarifications on DM's expected
behavior; sounds like our existing toolset is a better fit.
Cheers,
Nate Wiger
PlayStation
Dan Kubb (dkubb)
Jun 2, 2010, 2:32:00 AM6/2/10
to DataMapper
Nate,
>
> FWIW, we actually use the ActiveRecord auto_migrations plugin by
> pjhyett in Production for all migrations and it works awesome:http://github.com/pjhyett/auto_migrations That plugin handles
> modifications/deletes of existing columns, such as changing sizes,
> removing null contraints, etc.
Ahh yes. This is similar to what I had planned for auto-migrations in
DM. At the moment the auto-migration and with the "classic migration"
code is not very DRY. We'll be rebuilding the classic migrations API,
and then making auto-migrations and auto-upgrading use the same
interfaces rather than implementing their own logic. Part of the
reason for the duplication is that up until recently the logic for
both was in different packages; but we unified them to help make
further improvements.
One of the plans is to have it so that we can use the model state to
compare against the DB state, and either apply the migration
immediately or create a classic migration that you can tweak if you
want. It should be possible to remove the need to ever write classic
migrations by hand again, we should be able to infer everything from
the models which IMHO is even better than what this plugin does.
> As you said, auto-upgrading can't handle renaming columns, but you
> shouldn't be changing column names after the fact IMO. If you chose a
> bad name, add an alias in the model, or just get over it. If you live
> with this minor restriction, then your life gets infinitely easier
> across the board (eg at the controller/view level too).
I can understand your point of view. I think it depends on what you
can live with. In some cases, if you understand the scope of the
problem you can design a schema that doesn't change very frequently.
In other cases you have very little information up-front and the
schema can change drastically as your understanding grows. In the
latter case I think at some point if you had alot of aliases built up
you might want to deprecate the aliases and remove the extra mapping
layer from your design.
I know in the case of some classes I write with little up-front
knowledge I am continuously refactoring with "Rename Method", and I
know that I would only want to maintain aliases for as little time as
possible. If they were part of my private API, I would want to change
those names immediately without aliasing anything. For public methods
I would only want to keep aliases, with deprecations, around until the
next major release.
> But I realize my team is in the minority, which is unfortunate for the
> community as a whole because auto-migrations are really amazing.
> There's no way we could survive without them given how fast we have to
> develop.
I think a smarter auto-upgrade is very important too. I just don't
think our current implementation is anything I would recommend at the
moment. If it was able to make changes to the schema based on the
current model structure it would be something I would recommend alot
more often to people.
> Thanks again for the clarifications on DM's expected
> behavior; sounds like our existing toolset is a better fit.
No problem. If you have a good toolset that you're comfortable with, I
think that's awesome. I'm sure that when we get auto-upgrade
"upgraded" Matt will let you know so you can take another look at
it ;)
--
Dan
(dkubb)
P.S. Thanks for SQL::Abstract. Back in the day I wrote alot of perl/
DBI code with your lib.
>
> FWIW, we actually use the ActiveRecord auto_migrations plugin by
> pjhyett in Production for all migrations and it works awesome:http://github.com/pjhyett/auto_migrations That plugin handles
> modifications/deletes of existing columns, such as changing sizes,
> removing null contraints, etc.
DM. At the moment the auto-migration and with the "classic migration"
code is not very DRY. We'll be rebuilding the classic migrations API,
and then making auto-migrations and auto-upgrading use the same
interfaces rather than implementing their own logic. Part of the
reason for the duplication is that up until recently the logic for
both was in different packages; but we unified them to help make
further improvements.
One of the plans is to have it so that we can use the model state to
compare against the DB state, and either apply the migration
immediately or create a classic migration that you can tweak if you
want. It should be possible to remove the need to ever write classic
migrations by hand again, we should be able to infer everything from
the models which IMHO is even better than what this plugin does.
> As you said, auto-upgrading can't handle renaming columns, but you
> shouldn't be changing column names after the fact IMO. If you chose a
> bad name, add an alias in the model, or just get over it. If you live
> with this minor restriction, then your life gets infinitely easier
> across the board (eg at the controller/view level too).
can live with. In some cases, if you understand the scope of the
problem you can design a schema that doesn't change very frequently.
In other cases you have very little information up-front and the
schema can change drastically as your understanding grows. In the
latter case I think at some point if you had alot of aliases built up
you might want to deprecate the aliases and remove the extra mapping
layer from your design.
I know in the case of some classes I write with little up-front
knowledge I am continuously refactoring with "Rename Method", and I
know that I would only want to maintain aliases for as little time as
possible. If they were part of my private API, I would want to change
those names immediately without aliasing anything. For public methods
I would only want to keep aliases, with deprecations, around until the
next major release.
> But I realize my team is in the minority, which is unfortunate for the
> community as a whole because auto-migrations are really amazing.
> There's no way we could survive without them given how fast we have to
> develop.
think our current implementation is anything I would recommend at the
moment. If it was able to make changes to the schema based on the
current model structure it would be something I would recommend alot
more often to people.
> Thanks again for the clarifications on DM's expected
> behavior; sounds like our existing toolset is a better fit.
think that's awesome. I'm sure that when we get auto-upgrade
"upgraded" Matt will let you know so you can take another look at
it ;)
--
Dan
(dkubb)
P.S. Thanks for SQL::Abstract. Back in the day I wrote alot of perl/
DBI code with your lib.
Tony Mann
Jun 2, 2010, 10:46:04 AM6/2/10
to datam...@googlegroups.com
We use autoupgrade all the time on our medium-size project, since almost all of our db modifications involve adding fields. It is tremendously useful -- we have not yet written a single migration! Sometimes we have to tweak the db by hand, but this is so rare that it is not a problem. Since autoupgrade has never caused data loss, I would in fact recommend using it, as long as you understand its limitations. It is one of my favorite parts of DM.
..tony..
--
You received this message because you are subscribed to the Google Groups "DataMapper" group.
To post to this group, send email to datam...@googlegroups.com.
To unsubscribe from this group, send email to datamapper+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/datamapper?hl=en.
Nate Wiger
Jun 2, 2010, 11:17:15 AM6/2/10
to DataMapper
Cheers Dan - and thanks for the SQL::Abstract nod! Seems alot of us
old-school Perl hackers have made the move to Ruby. :-)
It took a look at the README for SQLA to remember you sent me a few
patches back when. Once the DM standard migration code gets ironed
out, maybe I can help you/Matt with some auto-upgrade tweaks to return
the favor.
-Nate
On Jun 2, 7:46 am, Tony Mann <thephatm...@gmail.com> wrote:
> We use autoupgrade all the time on our medium-size project, since almost all
> of our db modifications involve adding fields. It is tremendously useful --
> we have not yet written a single migration! Sometimes we have to tweak the
> db by hand, but this is so rare that it is not a problem. Since autoupgrade
> has never caused data loss, I would in fact recommend using it, as long as
> you understand its limitations. It is one of my favorite parts of DM.
>
> ..tony..
>
> > datamapper+...@googlegroups.com<datamapper%2Bunsubscribe@googlegrou ps.com>
> > .
old-school Perl hackers have made the move to Ruby. :-)
It took a look at the README for SQLA to remember you sent me a few
patches back when. Once the DM standard migration code gets ironed
out, maybe I can help you/Matt with some auto-upgrade tweaks to return
the favor.
-Nate
On Jun 2, 7:46 am, Tony Mann <thephatm...@gmail.com> wrote:
> We use autoupgrade all the time on our medium-size project, since almost all
> of our db modifications involve adding fields. It is tremendously useful --
> we have not yet written a single migration! Sometimes we have to tweak the
> db by hand, but this is so rare that it is not a problem. Since autoupgrade
> has never caused data loss, I would in fact recommend using it, as long as
> you understand its limitations. It is one of my favorite parts of DM.
>
> ..tony..
>
> On Tue, Jun 1, 2010 at 11:32 PM, Dan Kubb (dkubb) <dan.k...@gmail.com>wrote:
>
>
>
> > Nate,
>
> > > FWIW, we actually use the ActiveRecord auto_migrations plugin by
> > > pjhyett in Production for all migrations and it works awesome:
> >http://github.com/pjhyett/auto_migrationsThat plugin handles
>
>
>
> > Nate,
>
> > > FWIW, we actually use the ActiveRecord auto_migrations plugin by
> > > pjhyett in Production for all migrations and it works awesome:
> > .
Reply all
Reply to author
Forward
0 new messages