Hi Anton, others

I'm starting work on a messaging system between servers, using binary
messages that contain piqi records.

It seems that it's possible to add a 'Code' to the binary message
(piqirun:gen_record/2) that (probably) describes the type of message using
some kind of integer?. Am I correct in this?

If so, how would this work from the sender and the receiver side using some
record where the receiver doesn't know which one is coming in. Do I need to
manually (pattern match, case statement, ...) decode this Code, or can piqi
already do this for me.

I can of course build something with some record type identifier in the tcp
stream, but it looks to me that piqi might provide it out of the box.

Greetings

Bart

This is a typical problem. There's a way to solve it using Piqi, but
this method has a limitation. There are two variations of it:

1) Define a variant with all your messages as opitons. For example:

.variant [
   .name protocol-frame
   .option [ .type message-1 ]
   .option [ .type message-2 ]
   .option [ .type message-3 ]
]

There's a catch: you need to know all messages up-front, and you can't
extend that format for Piqi users without introducing backward
incompatibility between _running_ clients and servers if you needed to
upgrade them independently. I'm going to add support for an implicitly
generated "default" option in the next Piqi version.

Protobuf users, on the other hand, will be fine, but for them, such
format comes with an overhead. In order to figure out which message
they are dealing with, they would need to do this:

if (frame.has_message_1()) { ... }
else if (frame.has_message_2()) { ... }
else if (frame.has_message3()) { ... }

This is non-idiomatic and less efficient than the second alternative:

2)

.enum [
     % NOTE: this is also subject to the same limitation as for
variant, but it will be fixed soon

      .name message-type
     .option [ .name message-1 .code 1 ]
     .option [ .name message-2 .code 2 ]
     .option [ .name message-3 .code 3 ]
]

.record [
    .name protocol-frame
    .field [ .type message-type ]
    .field [ .type message-1 .optional ]
    .field [ .type message-2 .optional ]
    .field [ .type message-3 .optional ]
]

One could argue this is less elegant and less formal as there's a room
for an error when message-type doesn't correspond to the actual
message field. But I find this method to be more flexible and at least
for low-level and rarely changing protocols, the practical risk of
introducing an error is small.

Conceptually, the above methods are equivalent, and both of them have
the same limitation. If at some level of your protocol stack you
needed to make decisions based on message type, you would be forced to
decode the entire protocol-frame in order to figure out which message
it contains. Think of it as making HTTP request routing decisions
based on URL path. If you 100% sure you won't need that, then you are
good, just use one of the above methods. But sometimes it is really
hard to predict.

A proper solution to this problem is to add another protocol header in
front of each message payload. It can be an integer message code or
something more elaborate depending on your protocol needs. For
example, in HTTP, URL path is followed by a set of optional headers
and you can make some important decisions by looking at URL path,
headers or both without even touching payload.

I made all sorts of protocol design mistakes myself here:
https://github.com/alavrik/piqi/blob/dev/piqi.org/piqi-rpc.piqi
Don't worry -- nobody is affected, because it is purely internal so
far. But yes, if you are wondering, there's a plan to get Piqi-rpc
working on top of arbitrary transports (e.g. TCP). The idea is to
represent parts of HTTP meaningful for Piqi-RPC as a binary protocol
header and put it in front of each request / response payload.

There is going to be another relevant change. Piqi and Piqi-RPC will
support the concept of messages.

Anton

On Thu, Jul 12, 2012 at 5:58 AM, bart van deenen

Hi Anton

I decide to use an escript to generate some code (hardcoded for the
'wan.piqi' module). The generated code looks like this (for my wan.piqi
module).

serialize( #wan_fetch{} = Record ) ->
    wan_piqi:gen_fetch( 1, Record) ;
....
serialize( #wan_configuration{} = Record ) ->
    wan_piqi:gen_configuration( 9, Record) ;
serialize( Record ) ->
    ?ERROR("No type id defined for ~p",[Record]),
    {error, {no_code, Record}}.

deserialize( {1, {block, Block}} ) ->
    wan:handle_message(wan_piqi:parse_fetch( Block)) ;
....
deserialize( {9, {block, Block}} ) ->
    wan:handle_message(wan_piqi:parse_configuration( Block)) ;
deserialize( Args ) ->
    ?ERROR("Don't understand code for ~p",[Args]),
    {error, {wrong_code, Args}}.

And the script that generates the code is like this.

#!/usr/bin/env escriptmain(_A) ->
        Types =  [
                {1, fetch},
                {2, bucket_data},
                ....
                {9, configuration}
        ],
        ...
        writer(Types),
        reader(Types).

writer( [] ) ->
        io:format("serialize( Record ) -> ~n"
                "    ?ERROR(\"No type id defined for ~~p\",[Record]), ~n"
                "    {error, {no_code, Record}}.~n~n~n", []);

writer( [Type|Types] ) ->
        write_one(Type),
        writer(Types).

write_one( { Code, Name} ) ->
        io:format("serialize( #wan_~p{} = Record ) -> ~n"
                "    wan_piqi:gen_~p( ~p, Record) ;~n", [Name, Name, Code]).

reader( [] ) ->
        io:format("deserialize( Args ) -> ~n"
                "    ?ERROR(\"Don't understand code for ~~p\",[Args]), ~n"
                "    {error, {wrong_code, Args}}.~n~n~n", []);

reader( [Type|Types] ) ->
        read_one(Type),
        reader(Types).

read_one( { Code, Name} ) ->
        io:format("deserialize( {~p, {block, Block}} ) -> ~n"
                "    wan:handle_message(wan_piqi:parse_~p( Block)) ;~n", [Code, Name]).
                It works well.

Best regards

Bart van Deenen