[erlang-questions] Key-value stores and value part

[Vinayak Pawar]

Hi,

I've a key-value store. The value part is a tuple stored using term_to_binary().

   

    -record(some_rec, {e1, e2}).

    Rec = #some_rec{e1 = E1, e2 = E2}

    kv_store:put(Key, term_to_binary(Rec)).

I access the values in this key-value store as below:

    case kv_store:get(Key) of

        {ok, Bin} ->

            #some_rec{} = Rec = binary_to_term(Bin),

            do_some_work(Rec);

        false ->

            do_something_else(Key)

    end.

Now there is change in my tuple structure. A new element/field is been added to it:

    -record(some_rec, {e1, e2, e3}).

Henceforth tuples with this new structure will get stored in the KV store.

This means, everytime I change the tuple structure, I need to run through each key-value pair that has been stored using older tuple structure and update the value part to be in sync with new tuple structure?

This gives a hint that I'm doing something wrong in the way I store the values in schema-less KV store. Can you please help me figure out what is wrong?

Question: is it possible to store values in KV store so that I don't have to update the KV store whenever there is change in tuple structure? store values in JSON form?

Regards,

Vinayak

[Chandru]

On 25 February 2012 02:37, Vinayak Pawar <vinaya...@gmail.com> wrote:

Hi,

I've a key-value store. The value part is a tuple stored using term_to_binary().

   

    -record(some_rec, {e1, e2}).

    Rec = #some_rec{e1 = E1, e2 = E2}

    kv_store:put(Key, term_to_binary(Rec)).

I access the values in this key-value store as below:

    case kv_store:get(Key) of

        {ok, Bin} ->

            #some_rec{} = Rec = binary_to_term(Bin),

            do_some_work(Rec);

        false ->

            do_something_else(Key)

    end.

Now there is change in my tuple structure. A new element/field is been added to it:

    -record(some_rec, {e1, e2, e3}).

Henceforth tuples with this new structure will get stored in the KV store.

This means, everytime I change the tuple structure, I need to run through each key-value pair that has been stored using older tuple structure and update the value part to be in sync with new tuple structure?

Yes, if you want to use a record for the value part, this is unavoidable.
 

This gives a hint that I'm doing something wrong in the way I store the values in schema-less KV store. Can you please help me figure out what is wrong?

Nothing is wrong really.
 

Question: is it possible to store values in KV store so that I don't have to update the KV store whenever there is change in tuple structure?

No.
 

store values in JSON form?

Yes, or a simple list of {tag, value} tuples.

cheers
Chandru

[Max Bourinov]

You can try something like this:

Key = term(),

Value =[{Key :: atom(), Value :: term()]

With this approach you can dynamically manage fields in your Value. Of course some extra business logic and helper functions are needed.

Sent from my iPhone

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Jesper Louis Andersen]

On 2/25/12 3:37 AM, Vinayak Pawar wrote:
>
> Question: is it possible to store values in KV store so that I don't
> have to update the KV store whenever there is change in tuple
> structure? store values in JSON form?
>

The "Trick" is to version your values. That is, given data

{E1, E2}

you build a tagged variant:

{'version1', {E1, E2}}

Now when you need to alter the structure, you change the version tag:

{'version2', {E1, E2, E3}}

and henceforth, when you load the data from the underlying store, you
define a function that is akin to a code_change/3 function in a gen_server:

data_change({'version1', {E1, E2}}) ->
produce_version2_structure_here;
data_change({'version2', {E1, E2, E3}}) ->
{E1, E2, E3].

This means you now factorize every load of your data through
data_change/1 and you always store data in the newest version you know
about. This means you can lazily update your data as you go along
processing them. You can also force a persist in the new version right
away so subsequent reads will get the new version.

This is not a new idea. To a certain extent this is what Facebook does
with their data as they have too much in their store to be able to
convert all of it. It also means you can do database changes easily:
First make your erlang code cope with the new format. Then gradually
change the data store to the new format.

It takes some more work, but this is one way to do it.

--
Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen, DK

[Anthony Molinaro]

Also, if it's not clear, version1 and version2 could be records which you can match in multiple clauses.

-Anthony