[erlang-questions] Records / Proplists / JSON / BSON

[Steve Strong]

Hi,

We have built a number of projects recently that have a mongodb backend, an HTML / javascript frontend and Erlang in the middle - pretty standard stuff.  One of the things the we end up repeating over and over is mapping data from BSON (the mongo format) to Records (our preferred in-memory format for Erlang) and JSON (to send / receive from the browser).  To add to the mix, we also like using proplists in configuration files, so have mappings from those to records as well.

On the last project I finally got sick of doing it by hand, so knocked up a fairly simple parse transform to take the records (with their type specifications) and generate the mapping code, which has resulted in being able to do things like (note - pseudocode only!):

Foo = build_my_record(),

mongo:insert(collection, mapper:record_to_bson(Foo)),

web_socket:send(Client, mapper:record_to_json(Foo)),

receive

{client, Response} ->

do_stuff_with(mapper:json_to_record(foo, Response)

end

This has worked very well, and handles about 80% of the types we throw at it.  The sorts of types that it doesn't deal with are unions and tuples, e.g.

-record(bla, {

metadata :: x() | y(),

ratio :: {integer(), integer()}

}). 

 

Due to the value the simple version has had to us, I'm about to embark on a re-work that is going to aim to handle (pretty much) any type you can throw at it, and give full two-directional fidelity on the conversions (e.g., you can assert that Data == xxx_to_record(record_to_xxx(Data)) ).  

So, a couple of questions:

1. Does such a thing already exist?  I'm no fan of re-inventing the wheel :)

2. If not, would anyone be interested in it being open-sourced?

3. If 2., then does anyone have opinions on the functionality / API etc?

Interested in any feedback,

Cheers,

Steve

-- 

Steve Strong

Sent with Sparrow

[Steve Strong]

My plan for the actual JSON encode / decode is for the mapper to produce a structure that's compatible with jsx (https://github.com/talentdeficit/jsx) - that's the encoder / decoder that we currently use.   If we go the open-source route, I'd probably aim to make that pluggable so that folk can use their preferred JSON encoder.

Cheers,

Steve

-- 

Steve Strong

Sent with Sparrow

On Friday, 14 June 2013 at 12:31, Slava Yurin wrote:

 See https://github.com/iskra/jsonx. Maybe it decoder/encoder will help you.

 

14.06.2013, 17:13, "Steve Strong" <st...@srstrong.com>:

,

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Steve Strong]

Hi Thomas,

The first view projects where we did the mapping "by hand" were exactly using record_info, and for very simple records it works just fine.

The problems that we had with that approach is that we don't have any type information, so it makes it hard to get some types (atoms, timestamps) to roundtrip to JSON and back with full type fidelity. For example, you have to convert atoms to strings to pass to the browser but on the way back it's impossible to know whether the string you're receiving from the browser should stay a string to be turned back into an atom.  From a parse transform with type information, all these transformations can be done automatically, and nested structures can be handled easily.

If type information was available at runtime, that would be great but unfortunately the only way to get that is via the parse_transform mechanisms.

Cheers,

Steve

-- 

Steve Strong

Sent with Sparrow

On Friday, 14 June 2013 at 14:06, Thomas Allen wrote:

Hi Steve,

I've done some similar things, except rather than using a parse

transform to do this (I do not like parse transforms except as a last

resort, I find macros a more explicit way to express compile-time

things) I simply store my record info in application env when my

application starts, and define functions that access these values:

db:fields(RecName), etc. Then, I have higher level functions that use

this metadata. It's easy to test, and I reason it's pretty fast, with

application env being backed by ETS (it's never been a bottleneck).

%% in .hrl

-define(DB_FIELDS(Model), {Model, record_info(fields, Model)}).

%% in app:start/0:

application:set_env(db, model_fields,

[?DB_FIELDS(blog_post),

?DB_FIELDS(contact_message)]),

%% in record utils

fields() ->

{ok, Fields} = application:get_env(db, model_fields),

Fields.

fields(Model) ->

proplists:get_value(Model, fields()).

%% etc.

Maybe others here with more experience can correct me, but I think some

Erlang developers are too eager to introduce parse transforms to solve

simple problems.

Thomas

On Fri, Jun 14, 2013, at 03:34 AM, Steve Strong wrote:

My plan for the actual JSON encode / decode is for the mapper to produce

a structure that's compatible with jsx

(https://github.com/talentdeficit/jsx) - that's the encoder / decoder

that we currently use. If we go the open-source route, I'd probably aim

to make that pluggable so that folk can use their preferred JSON encoder.

Cheers,

Steve

--

Steve Strong

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Friday, 14 June 2013 at 12:31, Slava Yurin wrote:

See https://github.com/iskra/jsonx. Maybe it decoder/encoder will help you.

14.06.2013, 17:13, "Steve Strong" <st...@srstrong.com (mailto:st...@srstrong.com)>:

Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

,

_______________________________________________

erlang-questions mailing list

erlang-q...@erlang.org (mailto:erlang-q...@erlang.org)

http://erlang.org/mailman/listinfo/erlang-questions

[Thomas Allen]

> My plan for the actual JSON encode / decode is for the mapper to produce
> a structure that's compatible with jsx
> (https://github.com/talentdeficit/jsx) - that's the encoder / decoder
> that we currently use. If we go the open-source route, I'd probably aim
> to make that pluggable so that folk can use their preferred JSON encoder.
>
> Cheers,
>
> Steve
>
> --
> Steve Strong

> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

>
>
> On Friday, 14 June 2013 at 12:31, Slava Yurin wrote:
>
> > See https://github.com/iskra/jsonx. Maybe it decoder/encoder will help you.
> >

> > 14.06.2013, 17:13, "Steve Strong" <st...@srstrong.com (mailto:st...@srstrong.com)>:

> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

> > >
> > >
> > >
> > > ,
> > > _______________________________________________
> > > erlang-questions mailing list

> > > erlang-q...@erlang.org (mailto:erlang-q...@erlang.org)
> > > http://erlang.org/mailman/listinfo/erlang-questions

[Thomas Allen]

That makes sense. My only hesitation there is that I might eventually
need the ability to embed more information than type specifications
allow me to: Leaning on type specs feels like a dead end.

Thomas

On Fri, Jun 14, 2013, at 05:16 AM, Steve Strong wrote:
> Hi Thomas,
>
> The first view projects where we did the mapping "by hand" were exactly
> using record_info, and for very simple records it works just fine.
>
> The problems that we had with that approach is that we don't have any
> type information, so it makes it hard to get some types (atoms,
> timestamps) to roundtrip to JSON and back with full type fidelity. For
> example, you have to convert atoms to strings to pass to the browser but
> on the way back it's impossible to know whether the string you're
> receiving from the browser should stay a string to be turned back into an
> atom. From a parse transform with type information, all these
> transformations can be done automatically, and nested structures can be
> handled easily.
>
> If type information was available at runtime, that would be great but
> unfortunately the only way to get that is via the parse_transform
> mechanisms.
>
> Cheers,
>
> Steve
>
> --
> Steve Strong

[Richard A. O'Keefe]

On 15/06/2013, at 12:34 AM, Thomas Allen wrote:

> That makes sense. My only hesitation there is that I might eventually
> need the ability to embed more information than type specifications
> allow me to: Leaning on type specs feels like a dead end.
>

There's another approach.

Define your own "record" language,
and generate the Erlang code (*including* -record declarations) from that.

Your "language" could just a data structure like
{record, foo, [
{Field,Type,...},
...
}.
which could be enriched with whatever hints you want for JSON, BSON,
or whatever else takes your fancy.

[Vincent Siliakus]

Op zondag 16 juni 2013 23:48:21 UTC+2 schreef Richard A. O'Keefe het volgende:

There's another approach.

Define your own "record" language,
and generate the Erlang code (*including* -record declarations) from that.

 

And if you go down that route, be sure to first checkout the Piqi project at :

http://piqi.org

-vincent

 

 

 

 

 

 

 

[Steve Strong]

Thanks for the pointers to piqi, it looks close to what I was looking to build.  However, I've been playing a little and can't seem to express some of the constructs that we regularly use, for example we use tuples quite a lot for simple things such as ratios, so I may have a record such as:

-record(foo, {

ratio :: {integer(), integer()},

%% other stuff here

}).

piqi certainly doesn't have support for these out of the box, but I'm also struggling to see how I can define one as an extension.  I'm trying to provide a mapping from an Erlang tuple to a piqi list; it wouldn't be perfect, since piqi doesn't know the list should be precisely 2 long, but it would be good enough.  I can get piqi to make calls to custom code where I can do the tuple <-> list mapping, but the generated .hrl file ends up with the primitive piqi type (so list(integer()) ) rather than my erlang type ({integer(), integer()}).   This would likely upset Dialyzer quite a lot ;) .  Any piqi gurus out there have any hints?

Cheers,

Steve

-- 

Steve Strong

Sent with Sparrow

[Mahesh Paolini-Subramanya]

Your specific requirement is quite probably not immediately solvable via piqi.  

For similar situations, we use nested records, viz.

-record(foo, {

ratio :: #bar{},

%% other stuff here

}).

-record(bar, {

baz :: integer(),

bel :: integer()

}).

And the piqi config would end up w/ nested records, viz.

.record [

    .name foo

    .field [

        .name ratio

        .type bar

    ]

]

.record [

    .name bar

    .field [

        .name baz

        .type int64

    ]

    .field [

        .name bel

        .type int64

    ]

]

I know, this really isn't a direct match to what you want, but its pretty much what you'd end up w/ piqi.

FWIW, a good way to reason about piqi is to ask yourself "How would I do this with a protobuf?"…

Cheers

p.s. If you really  need it your way, you are probably best off w/ ROK's suggestion, viz., defining your own record language.

A fairly easy/quick-start way of doing that is to build it out using DTL, and then use escript/erlydtl as a 'preprocessor' to generate your records, accessors, etc.

It sounds goofy, but actually works quite well - we do something similar to generate pretty much all the scaffolding around our piqi <--> protobuf <--> json <--> records, including for CT, propEr generators, validators, etc.

Mahesh Paolini-Subramanya

That Tall Bald Indian Guy...

Google+  | Blog   | Twitter  | LinkedIn