[erlang-questions] Protocol Mapping

[Steve Davis]

Hi,

I'm frequently facing situations with binary protocol translations/transforms where I need to map binary codes to e.g. atoms and back. 

This is of course "easy" in Erlang, but suffers some inconveniences. I've attached a code snippet distilled down to the simplest case to explain my issue.

I'm sure the first codec pattern below is familiar, and for sure is efficient. 

However, there are multiple places where updating is required to add new message types.

The second pattern is much less usual, but handy as one line achieves the addition of a new message type.

It helps particularly when there is more than one pairing involved (e.g. {atom, code, status_message}). 

For sure this cannot be something nobody has seen/thought about. I'm wondering if anyone has comment on this, and maybe suggestions for approaches that I haven't thought of.

/s

-------

-module(mapping).

-compile(export_all).

%% traditional

-define(HELLO, 100).

-define(HELP, 101).

-define(DONE, 102).

encode(hello) -> ?HELLO;

encode(help) -> ?HELP;

encode(done) -> ?DONE.

decode(?HELLO) -> hello;

decode(?HELP) -> help;

decode(?DONE) -> done.

%% alternative

-define(MAP, [

{hello, 100},

{help, 101},

{done, 102}

]).

encode0(X) ->

{X, Y} = lists:keyfind(X, 1, ?MAP),

Y.

decode0(X) ->

{Y, X} = lists:keyfind(X, 2, ?MAP),

Y.

[Attila Rajmund Nohl]

Hello!

You could try to generate the code that implements the first pattern.

2012/12/19 Steve Davis <steven.cha...@gmail.com>:

> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Jachym Holecek]

# Steve Davis 2012-12-19:

Option 3:

You already have macros in option one, can just use them everywhere. Ugly
but the fastest you can get (no translation = no cost). Decoded messages
will be very non-user-friendly though.

Option 4:

-record(foo_map, {key, val}).

%% Installation procedure, only run once.

install() ->
%% XXX create the table with appropriate options if nonexistent.
[install_one(K, C) || {K, C} <- fields()],
ok.

install_one(Key, Code) ->
mnesia:dirty_write(#foo_map{key = {enc, Key}, val = Code}),
mnesia:dirty_write(#foo_map{key = {dec, Code}, val = Key}).

fields() ->
[{hello, 1},
{help, 2},
{done, 3}].

%% Runtime interface.

encode0(Key) ->
ets:lookup_element(foo_map, {enc, Key}, #foo_map.val).

decode0(Code) ->
ets:lookup_element(foo_map, {dec, Code}, #foo_map.val).

Option 5:

Like option (1), only you write fun code that writes the boring code for you
when needed (these tables tend to change infrequently). All definitions would
be in your codegen module in a function similar to fields() above, there
would be a few functions converting that to AST and using erl_pp to write
out resulting source file. You than keep this under source control and
remember to run codegen when you change master table (or teach it to some
Makefile).

HTH,
-- Jachym

[Daniel Goertzen]

This worked for me:

-define(TRANSCODE(Atom,Int), transcode(Atom) -> Int; transcode(Int) -> Atom).

?TRANSCODE(hello, 100);

?TRANSCODE(help, 101);

?TRANSCODE(done, 102).

Inspection of the expanded output with compile:file(File, ['P']). shows...

transcode(hello) ->

    100;

transcode(100) ->

    hello;

transcode(help) ->

    101;

transcode(101) ->

    help;

transcode(done) ->

    102;

transcode(102) ->

    done.

Dan.

[Max Lapshin]

it is a very bad practice to have one function for  atom() -> int() and int() -> atom() that will detect whether it is encoding or decoding.

You will meet some unknown commands in protocol and decide to bypass them as an integer. Everything will break.

[Daniel Goertzen]

That does weaken things, but you can fix it with a wrapper:

encode(Val) when is_atom(Val) -> transcode(Val).
decode(Val) when is_integer(Val) -> transcode(Val).

Or add a direction parameter to transcode():

-define(TRANSCODE(Atom,Int), transcode(decode,Atom) -> Int; transcode(encode,Int) -> Atom).

?TRANSCODE(hello, 100);

yields...

transcode(encode, hello) ->

    100;

transcode(decode, 100) ->

    hello;

Dan.

[Max Lapshin]

I've just described why you will fail with idea that atom() is an internal and int() is an external representation.

[Ulf Wiger]

For fun, I made a version with parse_trans which generates a module:

-module(mapping).

-compile(export_all).
-include_lib("parse_trans/include/codegen.hrl").

-define(MAP, [
{hello, 100},
{help, 101},
{done, 102}
]).

codegen(Mod) ->
codegen:gen_module(
{'$var',Mod}, [{encode,1},
{decode,1}],
[{encode, [fun({'$var',X}) ->
{'$var', Y}
end || {X, Y} <- ?MAP]},
{decode, [fun({'$var', Y}) ->
{'$var', X}
end || {X, Y} <- ?MAP]}]).


Eshell V5.9.2 (abort with ^G)
1> c(mapping).
{ok,mapping}
2> mapping:codegen(x).
[{attribute,1,module,x},
{attribute,37,export,[{encode,1},{decode,1}]},
{function,39,encode,1,
[{clause,39,[{atom,39,hello}],[],[{integer,40,100}]},
{clause,39,[{atom,39,help}],[],[{integer,40,101}]},
{clause,39,[{atom,39,done}],[],[{integer,40,102}]}]},
{function,42,decode,1,
[{clause,42,[{integer,42,100}],[],[{atom,43,hello}]},
{clause,42,[{integer,42,101}],[],[{atom,43,help}]},
{clause,42,[{integer,42,102}],[],[{atom,43,done}]}]}]
3> compile:forms(v(2),[]).
{ok,x,
<<70,79,82,49,0,0,2,48,66,69,65,77,65,116,111,109,0,0,0,
71,0,0,0,9,1,120,...>>}
4> code:load_binary(x,"/tmp/x.beam",element(3,v(3))).
{module,x}
5> x:encode(hello).
100
6> x:decode(100).
hello

The codegen:gen_module/3 pseudo-function is documented here:

https://github.com/esl/parse_trans/blob/master/doc/parse_trans_codegen.md#gen_module3

BR,
Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com

[Joe Armstrong]

On Wed, Dec 19, 2012 at 4:50 PM, Max Lapshin <max.l...@gmail.com> wrote:

it is a very bad practice to have one function for  atom() -> int() and int() -> atom() that will detect whether it is encoding or decoding.

You will meet some unknown commands in protocol and decide to bypass them as an integer. Everything will break.

I've been doing this for years :-)

You shouldn't have unknown commands in the protocol, and if you do

the only thing you can do is crash with a decent error message so you can debug your program later. Having crashed some supervision layer should try

and repair the damage.

But if you do only have atoms and integers and you stick to your

conventions it will be fine. You should also add a final exit clause

...

?transcode(done, 102);

transcode(Last) -> exit({transcode,bug,Last}).

The important thing is to crash if you get an unknown code.

Produce two errors - a polite error for the user, and a long and informative error in the log for the developer

Cheers

/Joe

[Steve Davis]

Wow - an embarrassment of response riches.

I love the macro approach simplicity; I hear Max; and I wonder if the extra step involved with the transform may be worth it for the flexibility I need for n-tuple mappings.

Will take me a bit of thinking time/prototyping to digest all this properly.

Thank you all, as ever, for your help, expertise, time and responses.

regs,
/s