Changing State / State Migrations
Chris Eidhof
Is there currently any functionality for migrating between two
different state versions? Suppose I deploy my app, then change the
person datatype to add a field "email :: String". I don't want to lose
all my existing data, so this means I have to migrate my data from the
old version to the new version. Ideally, I should only need to write a
function that "calculates" the email field, or fills it with some
predefined value.
-chris
Anton van Straaten
Take a look at the AllIn example:
http://happs.org/repos/HAppS-HTTP/Examples/AllIn.hs
It demonstrates migration from an OldSession type to a Session type, via
the Migrate type class.
Anton
Chris Eidhof
I tried around a bit, and it is actually quite easy. I wrote it down
so others might benefit as well.
You start of with a State datatype (I already named it OldState) and
make it an instance of Version:
\begin{code}
data Page = Page { title :: String
, body :: String
} deriving (Read, Show, Eq, Typeable, Data)
data OldState = OldState { oldPages :: [(String, Page)]
} deriving (Read, Show, Typeable, Data)
instance Version OldState
instance Version Page
$(deriveSerialize ''Page)
$(deriveSerialize ''OldState)
\end{code}
Now you can run your application, do some testing, and confirm that it
works. Next, we are going to write a migration (we are going to use a
Map instead of a [(key,value)]:
\begin{code}
type Pages = Map.Map String Page
data State = State { pages :: Pages
} deriving (Read, Show, Typeable, Data)
instance Migrate OldState State where
migrate (OldState p) = State (Map.fromList p)
instance Version State where
mode = extension 1 (Proxy :: Proxy OldState)
\end{code}
We wrote a migration form OldState to NewState, and the migrate method
simply has type OldState -> NewState. We also gave an instance of
version where we explicitly say that State depends on OldState.
This is all you have to do! Super-easy.
Thanks,
-chris
Chris Eidhof
To make sure my migrations are maintable, I put them in seperate
modules. My code looks like this:
module Migration.State (module Migration.State1) where
import Migration.State1
I've got all my methods in Migration.State1:
$(mkMethods ''State ['askPages, 'lookupPage, 'insertPage])
Then, when I want to create a new migration, I do the following:
module Migration.State (module Migration.State2) where
import Migration.State2
In State2 I do a qualified import of State1, and I write my migrations
functions. Everything compiles correctly, but I get a run-time
exception when initializing the state:
*** Exception: Couldn't find handler for event of type:
Migration.State1.InsertPage
I didn't change anything in Migration.State1, but it still gives this
error. Does anybody have a clue what might be going on here? Or is
there a better way to do migrations?
Thanks!
-chris
Lemmih
Make sure you do a full checkpoint before you rename your state. HAppS
gets confused if it has to replay actions which refer to a state isn't
in use anymore.
--
Cheers,
Lemmih
Chris Eidhof
createCheckpoint and shutdownSystem). However, now the migration isn't
being done at all, but the initialValue for State2 is used. I did a
trace in the migrate function, but it isn't called at all. I have an
instance for both Migrate and Version that depend on State1.State. Do
you have a clue about what is happening here?
-chris
Chris Eidhof
> Thanks, that helped removing the error message (I used both
> createCheckpoint and shutdownSystem). However, now the migration isn't
> being done at all, but the initialValue for State2 is used. I did a
> trace in the migrate function, but it isn't called at all. I have an
> instance for both Migrate and Version that depend on State1.State. Do
> you have a clue about what is happening here?
Hm, Eelco Lempsink and I dug into the source, and we came up with this
hack, which solves the problem:
hunk ./src/HAppS/State/Checkpoint.hs 97
- forM_ (ctlAllComponents ctl) $ \stateType
- -> case M.lookup stateType checkpoints of
- Just state -> setNewState stateType state
- Nothing -> return ()
+ forM_ (ctlAllComponents ctl) $ \stateType ->
+ forM_ (M.toList checkpoints) (\(_,v) -> setNewState
stateType v)
This is being done because setNewState otherwise isn't called, i.e.
when there's a checkpoint of type State1.State but you want a
State2.State. Instead of just a lookup, it should check whether a
*previous* version is equal to a key in the checkpoint map. However,
this information is only known in the Data-library, so it's probably
not trivial.
-chris
Lemmih
This becomes problematic when HAppS is used across multiple machines.
Updating a running application is done by restarting each node one at
a time. This of course requires that the software is compatible with
the previous version and, hence, renaming of components is not
allowed. You're also not allowed to drop support for methods or use
new methods immedialy.
The solution lies in two-step upgrades. First you add the new
component and secondly you move date from one to the other. The same
goes for adding/removing methods.
We are aware that this is very error-prone and that the user has no
way of knowing if/when something has gone wrong.
--
Cheers,
Lemmih
Chris Eidhof
-chris
I'm not sure if I really understand this. We didn't drop support, we
just created a new module with extended functionality. The old module
is still there. Could you maybe give a simple code example of what you
mean? That would help a lot =)
-chris
Lemmih
Let's say version one has a method named ComponentX.doSomething. When
the namespace is changed in version two, HAppS has no idea that
ComponentX.doSomething is now ComponentY.doSomething. To HAppS, it
looks exactly like ComponentX has been removed and a completely
unrelated component has been added.
This is a general problem (it applies to all changes, no just the
renaming of components) so a specialized solution isn't all that
desirable. Our apologies for making this more complicated than it
needs to be in most cases.
--
Cheers,
Lemmih
Eelco Lempsink
I disagree, because HAppS _should_ know it's related, since the old
type is wrapped in a 'Previous' data type:
< instance Version State where
< mode = extension 1 (Proxy :: Proxy Old.State)
> This is a general problem (it applies to all changes, no just the
> renaming of components) so a specialized solution isn't all that
> desirable. Our apologies for making this more complicated than it
> needs to be in most cases.
That's actually an interesting point. This is not meant as an attack,
but there don't seem to be many HAppS deployments 'in the wild', so
I'm wondering what the specific use-case for the way migrations
currently work is. It would really help to see something more
complicated than a toy example.
--
Regards,
Eelco Lempsink
Lemmih
> On 1 jul 2008, at 10:51, Lemmih wrote:
>>
>> On Mon, Jun 30, 2008 at 7:12 PM, Chris Eidhof <ch...@eidhof.nl> wrote:
>>>
>>> We didn't drop support, we
>>> just created a new module with extended functionality. The old module
>>> is still there. Could you maybe give a simple code example of what you
>>> mean? That would help a lot =)
>>
>> Let's say version one has a method named ComponentX.doSomething. When
>> the namespace is changed in version two, HAppS has no idea that
>> ComponentX.doSomething is now ComponentY.doSomething. To HAppS, it
>> looks exactly like ComponentX has been removed and a completely
>> unrelated component has been added.
>
> I disagree, because HAppS _should_ know it's related, since the old type is
> wrapped in a 'Previous' data type:
>
> < instance Version State where
> < mode = extension 1 (Proxy :: Proxy Old.State)
Granted, it is very possible to solve this problem. However, this
doesn't solve the more general problem upgrading/modifying your
application and, hence, will be treated as a convenience feature. Our
developer time is quite tight at the moment and we cannot afford to
deviate from our present priorities.
Patches are more than welcome, of course.
>> This is a general problem (it applies to all changes, no just the
>> renaming of components) so a specialized solution isn't all that
>> desirable. Our apologies for making this more complicated than it
>> needs to be in most cases.
>
>
> That's actually an interesting point. This is not meant as an attack, but
> there don't seem to be many HAppS deployments 'in the wild', so I'm
> wondering what the specific use-case for the way migrations currently work
> is. It would really help to see something more complicated than a toy
> example.
Migrations are only meant to facilitate fairly simple changes. Things
just become too complex when multimaster and partitioning protocols
enter the picture.
Our immediate goals include relatively easy scaling, guaranteed 100%
uptime (even when upgrading) and guaranteed data persistence.
Unfortunately, this means that we sometimes sacrifice simplicity for
features that aren't commonly used. Are we trying our best to make
everything as simple as possible but, as said before, developer time
is limited.
--
Cheers,
Lemmih
Eelco Lempsink
Could you elaborate on that? I've spent some time digging around in
the source code and I got the idea that since the Component class
directly depends on the Version class, it's just a matter of making
sure versioning is kept into account at all times. In what kind of
situations would this not solve the problem?
> Our developer time is quite tight at the moment and we cannot afford
> to
> deviate from our present priorities.
> Patches are more than welcome, of course.
I understand. I'm willing to put some time into this, but I want to
make sure I understand all the concepts first.
> Migrations are only meant to facilitate fairly simple changes. Things
> just become too complex when multimaster and partitioning protocols
> enter the picture.
I'd say that's an opportunity ;) I don't think migrations is a
feature you can just skip over. While multimaster may be important,
migrations of data is a feature that's probably high on the list of
every developer wanting to write a maintainable HAppS application
(i.e. use it for big/paid projects), so it might bite you later.
As I don't have any real experience with multimaster or partitioning
protocols, could you give an example of problems with migrations? I
can imagine it's important for all the masters to have the same
(updated) code, or else they can't read migrated data, but that's a
problem for almost every modification.
> Our immediate goals include relatively easy scaling, guaranteed 100%
> uptime (even when upgrading) and guaranteed data persistence.
> Unfortunately, this means that we sometimes sacrifice simplicity for
> features that aren't commonly used. Are we trying our best to make
> everything as simple as possible but, as said before, developer time
> is limited.
I respect that and I must say that HAppS has come a long way. It's a
really nice idea, well executed! I'm wondering: what are the long-
term goals?
--
Regards,
Eelco Lempsink
Eelco Lempsink
Here you go. The patches are against the latest version from the
repository. The zipfile also contains example code, illustrating the
use-case. If you accept the patches they might make a nice addition
to the examples.