Best way to persist desktop application data

[Yaron Naveh]

I build a desktop application. Basically user creates a "project",
which I internally represent as a set of model classes. The classes
know each other so they combine an object graph. When user saves or
opens a project I need to save or load the model data from disk.

Which way is best to persist the data (model classes)?

1. (Net) Data contract serialization.

Pros: fast to develop, has good performance (comparing with other
options).

Cons: Versioning. When my model classes change in the next version I
will need to migrate the data. Data contract backward compatibility
features only cover small changes like adding or removing fields, but
I want to consider cases where I will do more series changes. The
resulting xml is daunting and I would not like to deal with it
directly. Shipping the old model each time would complicate
deployment.


2. In memory database (SQLite)

Pros: Good versioning story (SQL scripts are standard).
Cons: runtime performance and development overhead.


3. Custom xml serialization (I manually serialize to xml, but in my
format). I'm still not sure if it gets the best or the worse of the
two worlds...

What do you think?

[Ayende Rahien]

Serialize to json, write a simple converter from v1->v2 for each release.

--
You received this message because you are subscribed to the Google Groups "altnetisrael" group.
To post to this group, send email to altnet...@googlegroups.com.
To unsubscribe from this group, send email to altnetisrael...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/altnetisrael?hl=en.

[Ken Egozi]

Use any loose format (ServiceStack, JSON, even xml) with customized mappers only when needed.

On Tue, Aug 31, 2010 at 3:49 PM, Yaron Naveh <yaro...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "altnetisrael" group.
To post to this group, send email to altnet...@googlegroups.com.
To unsubscribe from this group, send email to altnetisrael...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/altnetisrael?hl=en.

--
Ken Egozi.
http://www.kenegozi.com/blog
http://www.delver.com
http://www.musicglue.com
http://www.castleproject.org
http://www.idcc.co.il - הכנס הקהילתי הראשון למפתחי דוטנט - בואו בהמוניכם

[Yaron Naveh]

Thanks.

Do you mean to serialize the whole object graph at once using some
library, or to do my own own custom serializtion?

Would json handle correctly these cases:

- a single object is referenced by a few objects.
- model classes use inheritance. defining the derived types over the
base is not an option since the system supports addins so a concrete
model class may be an addin which I do not know of yet.

Also, how would the converter look like, do you mean an external
application that contains all my models so far (v1..vn-1) and converts
one to another?

> > altnetisrael...@googlegroups.com<altnetisrael%2Bunsubscribe@google groups.com>
> > .

[Ayende Rahien]

Assuming that you have an object model of where anything of interest can be reached from the root Project instance.

Just serialize the whole thing to JSON. You can do that with Newsoft.Json.dll as a single line of code.

And yes, it can handle polymorphism just fine.

And your migration would be a linq query, basically.

To unsubscribe from this group, send email to altnetisrael...@googlegroups.com.

[Yaron Naveh]

Just to clarify on the migration:

Would I need to bring all previous models with the converter and
convert model<-->model?
If you mean to run LINQ on the json itself - the json is expected to
be huge and I'm afraid other developers will have very hard time to
find the correct path to a value. Also this seems fragile - one
developer found a path, but another one pushed another object before
it and the "path" is broken. Isn't DB more robust for such data /
schema migrations?

[Ayende Rahien]

TINSTAFL

Migrations are hard

To unsubscribe from this group, send email to altnetisrael...@googlegroups.com.

[Avishay Lavie]

It depends heavily on what exactly you want to achieve, but for flexible data like 'a generic object graph' I think a loose data format like those suggested above would be much easier than a RDBMS. It'd be easier to plug in custom logic saying something like 'if you only find data from v1, here's how to convert it to v2 during deserialization' that way. So basically, I'm with Ken :)

Avish

To unsubscribe from this group, send email to altnetisrael...@googlegroups.com.

[Yaron Naveh]

Thanks everyone. I'll check the json option.

[Ariel Raunstien]

Still don't understand why not use Data Contract serialization. Will it not provide better versioning possibilities? 

I mean - I use it for simple object graphs (where the graph structure doesn't change over versions), and it provides very simple ways to manage change, mostly OOTB.

What changes are better handled with JSON serialization?

Arielr

http://www.QuickUnit.com/blogs/ariel

[Ayende Rahien]

Do a rename refactoring on data contract, for example.

[Ariel Raunstien]

How would you handle that with the json variant?

With data contracts I just replace the old xml property name with the new, and then deserialize, and I have a bunch of those rules to support all the version changes.

BTW, once when I had that problem (wanted to rename a field) I just created the "new"  property to access the "old" one, marked the old as [DebuggerHidden] and did not mark the new as [DataMember]. Worked great (until it drove me mad).

Arielr

http://www.QuickUnit.com/blogs/ariel

[Dotan N.]

sorry to barge in, but in this case isn't it preferable to explicitly use Name?
[DataMember(Name="Foo" ..)]
... Goo ...

you can refactor away now.

[Ayende Rahien]

I meant, when you have old data files lying around that you need to convert.

[Yaron Naveh]

Consider when you refactor property "Name" to "FirstName" and "LastName". You need to migrate the data as well. Sure, you can add accessors or do the IExtension magic. But for the long run (4 versions from now) the migration code is scattered all over. I prefer to have my implementation clean (=aligned with the latest design) and migrate the data externally. However the xml generated by the data contract is overwhelming for large object graphs and hard to migrate. This is why I was seeking alternatives (DB, json etc.)

[Dotan N.]

I'd deserialize old data, convert (map), serialize, done. i'm wondering what difference does it make if you use json or xml, or, why is the xml being overwhelming an issue?

[Yaron Naveh]

This requires me to ship every time with all the old model versions. In my case it would make deployment complex (although it is an option).

You are right that xml and json are variants of one alternative. The other alternative is to convert the business model to dto and save it in SQLite.

[Dotan N.]

so *abstractly*, you want to devise a set of *commands* that will move you from data model A to data model B. you do not want to keep an old model around.
in the case of SQLite, you model these commands as SQL statements.
in the case of XML, you'd XSLT (good luck :)
in the case of json (?).
so pick your option.

in any case, if this is a data model, and its anemic (if its not anemic I'd make the effort to generate one just for the migration process), i _personally_ wouldn't mind having a v-previous model laying around in order to save the overhead and assist with the migration.

[Ariel Raunstien]

I wouldn't like having all the previous version of the old domain model with every version either, but I'd still go with the string conversion before deserailizing to "current" object.

And the conversion logic isn't scattered around - it's, as Ken and Oren said, a series of converters from Vn to Vn+1, running serially.

Regarding [DataMember(Name="")] - true, but insufficient when renaming DataContracts (since the type name is embedded in the XML).

When doing something "deeper" than simple renamings - I wouldn't DARE try anything but object to object conversion, and would definitely keep the two version of domain models in code.

Arielr

http://www.QuickUnit.com/blogs/ariel

[Yaron Naveh]

I'm not even sure how a model-2-model conversion look like in code. I would need to load to memory 2 assemblies with the same name (ok) but how would I write a code that converts between two types with the same name. Or I would need to add a numerical suffix to class names - but I would need to do it in every drop to qa...

As for not doing anything but object-2-object - if the data is in DB and not xml then it does make sense to do migration without the old objects (there is always a migration script with db).

[Dotan N.]

On Tue, Aug 31, 2010 at 11:32 PM, Ariel Raunstien <ari...@gmail.com> wrote:

I wouldn't like having all the previous version of the old domain model with every version either, but I'd still go with the string conversion before deserailizing to "current" object.

And the conversion logic isn't scattered around - it's, as Ken and Oren said, a series of converters from Vn to Vn+1, running serially.

Regarding [DataMember(Name="")] - true, but insufficient when renaming DataContracts (since the type name is embedded in the XML).

IIRC, if you DataContract(Name="..") it should not embed type name but your explicit name.

[Ken Egozi]

you can also keep the latest version as typed dto code, and have old-style instances migrate with a dynamic/loose code style.

[Ken Egozi]

train of thoughts here:

say you read the items into a loosely typed thing (map from string to object or something, or you use IronRuby, whatever)

you then can run something like the following recursively:

#warning - pseudo code with language mix in tha house. do not try to copy-and-paste into any known compiler.

EnsureLatestVersion(item):

  var  type = item["type"];

  var migrators = GetMigratorsFor(type);

  while item["version"] < migrators.CurrentVersion:

    item = migrators.From(item["version"]);

  return item;

now you have graph containing latest-versions-only of the data.

you then will give it to AutoMapper or whatever in order to get it as typed objects. you might even serialize to a temporary JSON and deserialize using the typed option of JSON.NET now that you ensured the correct structure of the data.

btw, imo a version should be on aggregate root level, so scrap the word "recursively" I wrote earlier.

[Ariel Raunstien]

Dotan -

IIRC I was hoping that would be the exact behavior, but had a rude awakening. Was I wrong?

Yaron - 

If you're talking about substituting a path of the object graph between versions (changing member W of type X to member Y of type Z) - you're right - I don't know how to handle that. I was referring to complete graph changes. I understand how my solution isn't good enough if other parts of the code still want to reference objects from the graph (and I wouldn't go puritan and say you should have DTO for persistence only.)

I'd still *try* to avoid ETLs and keep the versioning "internal".

Arielr

http://www.QuickUnit.com/blogs/ariel

[Dotan N.]

On Tue, Aug 31, 2010 at 11:50 PM, Yaron Naveh <yaro...@gmail.com> wrote:

I'm not even sure how a model-2-model conversion look like in code. I would need to load to memory 2 assemblies with the same name (ok) but how would I write a code that converts between two types with the same name. Or I would need to add a numerical suffix to class names - but I would need to do it in every drop to qa...

there must be some assumption i'm missing from your part. you can do that in several ways (did you consider namespacing?), i think its not a problem in any case.

As for not doing anything but object-2-object - if the data is in DB and not xml then it does make sense to do migration without the old objects (there is always a migration script with db).

it does make sense yea, and its already done somewhere for what you regard as "desktop applications" and if you want to go there you should

both ways are possible and valid, i think its a trade off based on time constraints and complexity of both your project and data (ymmv).

[Dotan N.]

On Wed, Sep 1, 2010 at 12:09 AM, Ariel Raunstien <ari...@gmail.com> wrote:

Dotan -

IIRC I was hoping that would be the exact behavior, but had a rude awakening. Was I wrong?

I'm afraid so; the generated XML will match the explicit Name and Namespace properties that you provide. it won't expose the type name in that case.
 

[Ariel Raunstien]

Yeah, I'm sorry, I always use the NetDataContractSerializer, and it stores all required type data in the XML (full type name included).

Regular (type specific) DataContractSerializer doesn't do that.

My bad.

Arielr

http://www.QuickUnit.com/blogs/ariel