Maglev schema evolution

[Patru]

I am still experimenting with Maglev, but it looks really promising
for now.

However, one very nice thing about rails is, that it takes care of
database evolution with migrations. As long as we just redefine method
semantics an add new methods there is no real problem, a simple

ruby -Mcommit model.rb

will do the trick and all persisted objects learn the new tricks.
However, once we start adding new instance variables things become a
little more tricky, as we might want to initialize them. Migrations
take care of this too, but I do not like the thought of scattering my
class definitions across a sequence of "class migrations". How does
Smalltalk deal with schema/class evolution in Gemstone/S?

Persistent code also introduces additional problems if you start
removing methods. As far as I can see you will have to call

remove_method :method_to_remove

exactly once in a persistent context. Of course I could guard this
inside the class definition using

remove_method :method_to_remove if method_defined? :method_to_remove

but this looks terrible in a piece of code that should stick around to
define the behavior of a class.

On the one hand I still think there would have to be something like a
migration to be able to go back and forth with the schema/class
definition, but on the other hand I would like to keep the definition
of the class in one file that I can track with a revision control
system. This does not seem to mix well with the persistence semantics
of Maglev and the dynamic behavior of classes in Ruby.

If someone has already come up with a solution I would like a pointer.

[Conrad Taylor]

On Thu, Nov 10, 2011 at 5:09 PM, Patru <pat...@gmail.com> wrote:

I am still experimenting with Maglev, but it looks really promising
for now.

However, one very nice thing about rails is, that it takes care of
database evolution with migrations. As long as we just redefine method
semantics an add new methods there is no real problem, a simple

ruby -Mcommit model.rb

will do the trick and all persisted objects learn the new tricks.
However, once we start adding new instance variables things become a
little more tricky, as we might want to initialize them. Migrations
take care of this too, but I do not like the thought of scattering my
class definitions across a sequence of "class migrations". How does
Smalltalk deal with schema/class evolution in Gemstone/S?

Patru, I would recommend taking a look at the migrations example which

you can find here:

https://github.com/MagLev/maglev/blob/master/examples/persistence/migrations/example/example.rdoc

https://github.com/MagLev/maglev/blob/master/examples/persistence/migrations/example2/README.rdoc

 

Persistent code also introduces additional problems if you start
removing methods. As far as I can see you will have to call

remove_method :method_to_remove

exactly once in a persistent context. Of course I could guard this
inside the class definition using

remove_method :method_to_remove if method_defined? :method_to_remove

but this looks terrible in a piece of code that should stick around to
define the behavior of a class.

The above problems exists within non-persistent code by manually removing a method(s).  Also, this can happen as easily with Rail by adding a migration without updating the dependent code.  Thus, if you would like to refactor code, you should have a test suite which tests the public API of your application.  Well, I think Maglev has very good solutions that addresses your concerns in the examples that appear above.

Good luck,

-Conrad

 

On the one hand I still think there would have to be something like a
migration to be able to go back and forth with the schema/class
definition, but on the other hand I would like to keep the definition
of the class in one file that I can track with a revision control
system. This does not seem to mix well with the persistence semantics
of Maglev and the dynamic behavior of classes in Ruby.

If someone has already come up with a solution I would like a pointer.

--
You received this message because you are subscribed to the Google Groups "MagLev Discussion" group.
To post to this group, send email to maglev-d...@googlegroups.com.
To unsubscribe from this group, send email to maglev-discuss...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maglev-discussion?hl=en.

[Peter McLain]

On Nov 10, 2011, at 5:09 PM, Patru wrote:

> I am still experimenting with Maglev, but it looks really promising
> for now.

Nice to hear that!

> How does Smalltalk deal with schema/class evolution in Gemstone/S?

For the official scoop, see chapter 9 of the GS64 Programming Guide
http://community.gemstone.com/download/attachments/6816350/GS64-ProgGuide-3.0.pdf?version=1

I also briefly discuss and compare with ruby in examples/persistence/migrations/migrations.org (see Appendix A).
NOTE: migrations.org are just my notes and half-baked writing as I was trying to thing through the issues; it is an unfinished document, i.e., there is a lot more room for discussion here, I may have things wrong, etc.

> Persistent code also introduces additional problems if you start
> removing methods. As far as I can see you will have to call
>
> remove_method :method_to_remove
>
> exactly once in a persistent context. Of course I could guard this
> inside the class definition using
>
> remove_method :method_to_remove if method_defined? :method_to_remove
>
> but this looks terrible in a piece of code that should stick around to
> define the behavior of a class.

I think that there is a good opportunity to create a migration DSL. We were reluctant to create to much of an official migration DSL, since, as implementors of the language rather than hard-core users of ruby, we lacked the real-world application level experience and we didn't want to start the whole DSL off on the wrong foot. Most of the experience the MagLev team has with migrations, are from the Smalltalk side, but the tools, issues and mostly the usage patterns (metaprogramming) between Smalltalk and Ruby are quite different (some of this is mentioned in migrations.org).

My personal view is that a migration DSL that is similar to the Rails migrations is probably the way to go (and by "similar", I mean that you deal with persistent classes and objects mostly or exclusively through the dsl (rails generate model ...; edit ; rake db:migrate). This forces you to think about and consider schema evolution. The flip side is that you may not want to persist rails itself (although you could).

> On the one hand I still think there would have to be something like a
> migration to be able to go back and forth with the schema/class
> definition, but on the other hand I would like to keep the definition
> of the class in one file that I can track with a revision control
> system. This does not seem to mix well with the persistence semantics
> of Maglev and the dynamic behavior of classes in Ruby.

I don't think there is a silver bullet here. Take a look at examples/persistence/migration (keeping in mind it is just a sketch) and I'd love to hear some good discussion about this.

--
Peter McLain
pmc...@vmware.com

[Patru]

On 11 Nov., 02:48, Peter McLain <pmcl...@vmware.com> wrote:
>
> > How does Smalltalk deal with schema/class evolution in Gemstone/S?
>
>   For the official scoop, see chapter 9 of the GS64 Programming Guide

>  http://community.gemstone.com/download/attachments/6816350/GS64-ProgG...

>
>   I also briefly discuss and compare with ruby in examples/persistence/migrations/migrations.org (see Appendix A).
>   NOTE: migrations.org are just my notes and half-baked writing as I was trying to thing through the issues; it is an unfinished document, i.e., there is a lot more room for discussion here, I may have things wrong, etc.
>
> > Persistent code also introduces additional problems if you start
> > removing methods. As far as I can see you will have to call
>
> > remove_method :method_to_remove
>
> > exactly once in a persistent context. Of course I could guard this
> > inside the class definition using
>
> > remove_method :method_to_remove if method_defined? :method_to_remove
>
> > but this looks terrible in a piece of code that should stick around to
> > define the behavior of a class.
>
>   I think that there is a good opportunity to create a migration DSL.

As this is right at the point which defines the difference which
separates
Maglev from your regular ruby interpreter I think we should spend a
moment
to consider wether we *really* want to solve it in the same way rails
solves
these problems. The differences between the rails-model and the maglev-
model
are not vast, but they do exits as data and logic become a little
blurred.
Let me try to sum this up in a table (even if this does not sound
appropriate
for maglev :-)

| Rails | Maglev | Smalltalk
------------+---------------------+---------------------
+-----------------------
data | database | image / stone | image /
stone
------------+---------------------+---------------------
+-----------------------
logic/code | files/VCS | ?files/VCS?/image | ?files/VCS?/
image
------------+---------------------+---------------------
+-----------------------
classes are | open, transient | open, persistent | invariant,
persistent
| | single definition | version
history
------------+---------------------+---------------------
+-----------------------
structure | DB-structure | class definitions, | class
definitions,
definition | | persistent objects | all objects
persist
------------+---------------------+---------------------
+-----------------------
metadata | schema.rb automatic | ?? | ?not
necessary?
------------+---------------------+---------------------
+-----------------------
migrate | database structure | data structure | data
structure
| | some code changes | all code
changes
------------+---------------------+---------------------
+-----------------------
deployment | deploy VCS delta | ?deploy VCS delta? | ?run all
data and
| run all migrations | run all data and | code
migrations?
| | some code migrations|

Being no smalltalker I can only speculate about the role of a VCS in a
smalltalk
repository, but it would most probably be possible, to also serve the
code
history entirely from the image, even on a method basis as it used to
be in
VisualAge for Java almost 10 years ago. Given todays development-
environments
I am pretty sure we want to stick with the VCS integration we have,
but in
that case we would still have to manage some code migrations manually.
In the beginning there will not be a way determine automatically which
changes
have to be run to get the system in a consistent state, maybe that
could change
later, but it will require significant work.

A big question mark for me is the area of metadata. Rails determines
the
schema.rb from the database structure automatically, this will not be
possible
in maglev, but there is not immediate need as we do not have to cast
data.
However, there are a lot of nice gems using this metadata nowadays
(take formtastic for example) to at least consider having some
metadata
as well. For me there would not have to be a central schema.rb as it
would probably turn out to be a bottleneck in the process of larger
teams,
but a metadata section in a class that centralizes data types and
validations
sounds like an attractive alternative.

Maybe some Smalltalkers can improve the table above and I would like
to
read about other takes on metadata and deployment.

[Patru]

sorry, my ASCII-art broke, will try to improve on it if there is
interest.

[Conrad Taylor]

Sent from my iPad

On Nov 11, 2011, at 7:21 AM, Patru <pat...@gmail.com> wrote:

> sorry, my ASCII-art broke, will try to improve on it if there is
> interest.
>

Patru, I am very interested in the discussion.

-Conrad

[Patru]

Hi Conrad

I put it in the tag-wiki on stackoverflow, appears to be a more
appropriate place at it will be easier to update and to keep it as a
working document (and google will index it real quickly :-). You can
find it on

http://stackoverflow.com/tags/maglev/info

At some point it should probably migrate to a page on github.

Unfortunately there seems to have been little discussion on the point,
maybe we can spice it up a little.

I am pretty sure we will need some type-Metadata to work with Rails
and the web. Of course Maglev will be able to store anything we care
to throw into an instance variable, but it will *not* know how to
demarshall a string coming in from a web-form. In order to enable a
correct conversion we will have to provide rails with some type
information (as we will use plain objects, not ActiveRecords). I never
dug far enough into rails to tell what exactly this might have to be,
but I guess we should be able to do this in a simple hash:

META = {
:name => :String,
:first_name => :String,
:date_of_birth => :Date,
:zip_ccde => :Fixnum,
:city => :String
}

you get the idea. Rails usually relies on the information in schema.rb
which is generated by reading the metadata from the database. As long
as we are not even providing Maglev with this information we can
hardly expect to guess it correctly. And as classes are the only
schema information I need to provide to Maglev Address::META looks
like a natural place to put it.

For the structural migrations I am still not quite sure where I really
want to keep them. There is of course the possibility to keep it in
migration files as rails usually does it, but that will scatter the
definition of our class which is a thought I do not really like (at
least until there is a development system that would be able to stitch
it together again).

On the other hand we might introduce a class constant with the version
of the class. It ought to be possible to determine the current version
number of the persistent class while loading the file and write plain
old ruby code (there might not even be a need of a special DSL to do
it) which migrates the class (or at least its disruptive changes) to
the current version. Might look something like

...
if VERSION < "0.9.0"
remove_method :postal_code
VERSION = "0.9.0"
end
if VERSION < "0.9.5"
attr_accessor :street
VERSION = "0.9.0"
end
...

again you get the idea. I will have to try if this really works, but I
think it should do so for now.

This would have the advantage to keep the whole class definition in
one file which could be written in the development environment and
then just loaded into your Maglev image in test, staging or production
environments. It would be possible to load a class multiple times and
it could break in a controlled way if it needs to do so. It would
however rely on migrations which are isolated inside one class. This
might be too much of a restriction, but I think we will all have to
experiment with some projects which use Maglev objects inside rails to
be able to know better.

I could also imagine that it does not really play nice for "large"
migrations, whatever that might turn out to be, but you probably do
not want to initialize new attributes in millions of existing objects
during class load time.

:: Patru ""

[Patru]

On 20 Nov., 04:21, Patru <pat...@gmail.com> wrote:
> ...
>   if VERSION < "0.9.0"
>     remove_method :postal_code
>     VERSION = "0.9.0"
>   end
>   if VERSION < "0.9.5"
>     attr_accessor :street
>     VERSION = "0.9.0"
>   end
> ...

I still seem to be the only one trying to develop anything with Maglev
which is a pitty as it really is a very nice feeling to have your Ruby
objects act as your data store. However, you have to get used to do
some things different. I decided to persist my model objects and to
keep my tests and some PDF generating code transient. Inevitably at
some point I ended up refactoring a little and I wanted to extract a
few methods into a separate module. Usually this is a simple thing,
but even after re-persisting my model class (it is a simple rake task
to write) and dropping my instance variable manually the "new" methods
still looked like the "old" ones and failed on lines where they
certainly were not anymore. Thinking a little about it I realized,
that I had run in the exact problem I described above. The "old"
versions of my method would not be deleted and the "new" one in the
newly included module would not be found as it was "higher up" the
call chain. I ended up putting the following piece of code *in front
of* my class:

if Company::VERSION < "1.0.1"
# puts "I am an old version #{Company::VERSION}"
Company.remove_method :rate_for
Company.remove_method :add_rate
end

and then setting

VERSION = "1.0.1"

inside my class. Like this I kept the "migration" code outside the
class, but still in the same file. This is not necessary when you just
add to your class, but it is required to be done when you remove (or
as in my case refactor) some code. As soon as old objects need to be
migrated (probably by initializing new instance variables and the
like) as will be the case in most realistic examples, you will need
some more code to do the migration.

By the way: in the process of figuring this out I noticed a slight
semantic difference between MRI Ruby and Maglev. If you query

Object::VERSION

in MRI you get the ruby version (at least up to 1.8.7, 1.9 does not
seem to define this Constant anymore). Maglev will return a version
number for the class, for Object that currently is 1.0.0, other
classes have their own VERSION constant <class>::VERSION, currently
all classes (even your own ones) get a VERSION String 1.0.0 b< default
which might be an indication of the version of Maglev I am using. As
shown above you can redefine this CONSTANT in your own classes,
although it is probably not a good idea to set it lower than it
initially is.
You can use the VERSION constant to migrate your persistent classes,
but it is still just a one-way trip, there does not seem to be an easy
way to "migrate back". However, you may abort the transaction if your
code does not compile, but if the code compiles ok but introduces
other problems (which you will agree should never happen :-) then
there is no obvious way to get "back". This is still not quite what we
are used to in Rails.

:: Patru ""

[Patru]

Of course there is one more caveat (which should have been obvious to
me in the first place). As all classes descend from Object
Company::VERSION will always be "visible" even if it is not defined in
Company. This also explains why all classes in Maglev do seem to be of
version "1.0.0".

As VERSION is called an obsolete constant in Ruby 1.8.7, because of
this it is probably a bad idea to use the (obvious) name as a
versioning construct. While it would not interfere with Ruby 1.9 and
above its use in 1.8 (although I have not been able to determine the
exact semantics) makes it a suspicious candidate for a versioning
construct. I guess I will resort to using CLASS_VERSION for now.

On 21 Dez., 23:56, Patru <pat...@gmail.com> wrote:

> Object::VERSION
>
> in MRI you get the ruby version (at least up to 1.8.7, 1.9 does not
> seem to define this Constant anymore). Maglev will return a version
> number for the class, for Object that currently is 1.0.0, other
> classes have their own VERSION constant <class>::VERSION, currently

> all classes (even your own ones) get a VERSION String 1.0.0 by default

[Conrad Taylor]

On Thu, Dec 22, 2011 at 9:18 AM, Patru <pat...@gmail.com> wrote:

Of course there is one more caveat (which should have been obvious to
me in the first place). As all classes descend from Object
Company::VERSION will always be "visible" even if it is not defined in
Company. This also explains why all classes in Maglev do seem to be of
version "1.0.0".

As VERSION is called an obsolete constant in Ruby 1.8.7, because of
this it is probably a bad idea to use the (obvious) name as a
versioning construct. While it would not interfere with Ruby 1.9 and
above its use in 1.8 (although I have not been able to determine the
exact semantics) makes it a suspicious candidate for a versioning
construct. I guess I will resort to using CLASS_VERSION for now.

I would continue to use the following scheme for Maglev:

$ rvm

maglev 1.0.0 (ruby 1.8.7) (2011-10-31 rev 1.0.0-27184)[Darwin x86_64]

Why?  Ruby 1.8.6 is slightly different from Ruby 1.8.7.  The goal of 

Ruby 1.8.7 was to bridge the gap between Ruby 1.8.7 and Ruby 1.9.x.

Furthermore, Rubinius does something very similar. For example,

$ ruby -v

rubinius 2.0.0dev (1.8.7 ac282c63 yyyy-mm-dd JI) [x86_64-apple-darwin11.2.0]

One thing to note, Rubinius provides the ability to switch from 1.8.7 to 1.9.x

implementation of its Ruby interpreter.  Thus, my recommendation would be

to stick with the current versioning scheme because it makes it clear to the

user.

Think different and code well,

-Conrad

 

On 21 Dez., 23:56, Patru <pat...@gmail.com> wrote:

> Object::VERSION
>
> in MRI you get the ruby version (at least up to 1.8.7, 1.9 does not
> seem to define this Constant anymore). Maglev will return a version
> number for the class, for Object that currently is 1.0.0, other
> classes have their own VERSION constant <class>::VERSION, currently

> all classes (even your own ones) get a VERSION String 1.0.0 by default

> which might be an indication of the version of Maglev I am using. As
> shown above you can redefine this CONSTANT in your own classes,
> although it is probably not a good idea to set it lower than it
> initially is.

[Conrad Taylor]

Yes, I think this may be a bug because Object::VERSION should define the

underlying target interpreter version.  If you're targeting the Ruby 1.8.7, then

Object::VERSION == RUBY_VERSION == "1.8.7".  

In Maglev:

VERSION == "1.0.0" which is the version of the Maglev Ruby VM but

RUBY_VERSION == "1.8.7".

In CRuby (MRI):

VERSION == RUBY_VERSION == "1.8.7" which is the version of the MRI.

In Rubinius (1.8.7 mode):

VERSION == RUBY_VERSION == "1.8.7" which is the version of Rubinius.

In Ruby 1.9.2/1.9.3:

VERSION, the global constant, has been removed from Ruby 1.9.x.  However,

RUBY_VERSION == "1.9.2" or RUBY_VERSION == "1.9.3" depending on the

interpreter that you're using.  Thus, if you're wanting the version of the Ruby

interpreter and you're wanting to make your code portable between Ruby 1.8.7

and Ruby 1.9.x, I would recommend using the RUBY_VERSION.  I'm guessing

that the Ruby core team removed this constant in Ruby 1.9.x because it added 

a level of confusion (i.e. are you talking about the version of the object or the 

interpreter).  Just my 2 cents.

You can use the VERSION constant to migrate your persistent classes,
but it is still just a one-way trip, there does not seem to be an easy
way to "migrate back". However, you may abort the transaction if your
code does not compile, but if the code compiles ok but introduces
other problems (which you will agree should never happen :-) then
there is no obvious way to get "back". This is still not quite what we
are used to in Rails.

Next, I would like to propose that the Maglev make the following modifications:

1)  change VERSION to be consistent RUBY_VERSION

2)  introduce a hook method which can be overridden by classes or

     an appropriate constant like OBJECT_VERSION that can defined on a 

     class by class basis.

Last but not least, I just wanted to let you know that I'm working on some services

and other components using Maglev.  Also, I'm hoping to create more posts on Maglev

which cover more of its features as well as its overall performance.

Think different and code well,

-Conrad

 

:: Patru ""

[Patru]

Hi Conrad, nice to know there are others :-)

I do not particularly cars about the definition of Object::VERSION as
it has gone away in 1.9 which seems to be a good thing as I was not
able to find a consistent definition of its semantics (at least on
rdoc that is).

I will now use CLASS_VERSION to track class changes that "removes
things" which I think is necessary to keep maglevs reasonably
consistent, even if that just provides "oneway migrations".

So I end up with code like

if defined?(Invoice::CLASS_VERSION)
if Invoice::CLASS_VERSION < "0.0.2" then
class << Invoice
remove_method :next
end
end
# more for later class versions
end

*in front* of my class. Like this I should be able to load a class
definition to any version of Maglev I care to update, even if the
class has not yet been defined. This introduces an additional layer of
complexity, but that seems a reasonable thing to expect when you start
treating your objects as the database (which is a Good Thing (TM?-) as
far as I can tell).

On 23 Dez., 04:47, Conrad Taylor <conra...@gmail.com> wrote:

> Yes, I think this may be a bug because Object::VERSION should define the
> underlying target interpreter version.  If you're targeting the Ruby 1.8.7,
> then
> Object::VERSION == RUBY_VERSION == "1.8.7".
>

> *In Maglev:*

>
> VERSION == "1.0.0" which is the version of the Maglev Ruby VM but
> RUBY_VERSION == "1.8.7".
>

> *In CRuby (MRI):*

>
> VERSION == RUBY_VERSION == "1.8.7" which is the version of the MRI.
>

> *In Rubinius (1.8.7 mode):*

>
> VERSION == RUBY_VERSION == "1.8.7" which is the version of Rubinius.
>

> *In Ruby 1.9.2/1.9.3:*
> *
> *