Schema version evolution

[erdody]

We are using json value serialization, with a simple schema that is a map with a few attributes. If we decide to add a few more attributes, and create a new schema version, old values will be deserialized correctly BUT how do we deploy it?

a) Voldemort first: we add the new schema version, it will let the clients know about the change and the serialization will fail because it always uses the latest version and there are missing attributes

b) Code first: the serialization will complain because we have "extra" attributes

Shouldn't we have a way to tell the StoreClientFactory what schema version to use instead of it always assuming we want to use the latest one? In this way we can add a new schema version in voldemort without the client serializations failing.

Am I missing something?

Thanks,

Diego

[erdody]

Anyone? Nobody using json and tried to add fields to the schema?

Diego

[Greg Moulliet]

I ran into the same issue, and didn't see a clear path around it using
Voldemort's json serialization.
I now only use Voldemort's json serialization for keys, knowing that
they will not change.

For values I use either Thrift or custom json serialization. Thrift
handles versioning correctly, as does Protocol Buffers, and both are
supported by Voldemort.
Lately, I've found it simpler to just convert objects into json.
Then, anything can read the json strings, and I handle versioning
internally, which is simple to test.

> --
> You received this message because you are subscribed to the Google Groups
> "project-voldemort" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/project-voldemort/-/DiFgzc9fF4EJ.
>
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to
> project-voldem...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/project-voldemort?hl=en.

[Chinmay Soman]

erdody:

You can use multiple versions to do this. For eg: version 0 can be the
one with the old map and version 1 with the new map (including the
extra attributes).
This version number (only used with Voldemort's JSON) is attached as a
header to each value. The clients will continue to read the old values
(based on this version number).

However, puts are done only based on the latest version. So here's
what you should do:
1) Update the stores.xml (by adding another version schema to that
store)
2) Bounce the clients (to register this change) : At this point you're
free to use the old code / new code.

[Jeff Clites]

Diego:

I ran into the same issue when I first started using the JSON
serializer, and patched JSON serializer the code so that it tolerates
missing keys during serialization (so it serializes as though they
were there but null; basically I just altered a check that was in
place). That allows me to deploy using your scenario (a). (Still, your
"old" client code has to be tolerant of getting maps with extra keys
that it doesn't know about. But that's easy to code for, and sometimes
you get it for free, depending on how your code is written.)

JEff