piqirun:parse_record(Binary) and order of resulted blocks

[bart van deenen]

Hi Anton, all

I'm using the generated piqi code with the integer Code field to identify my records in a tcp stream:

Id = #wantransport_msg_id{cmd_nr = 123,reply_cmd_nr = 321, origin = 1,destination = 2}
erlang:list_to_binary(wantransport_piqi:gen_msg_id(99, Id)).
    <<154,6,10,8,246,1,16,130,5,24,2,32,4>>

And decoding:

piqirun:parse_record( <<154,6,10,8,246,1,16,130,5,24,2,32,4>> ).
    [{99,{block,<<8,246,1,16,130,5,24,2,32,4>>}}]
Id = wantransport_piqi:parse_msg_id( <<8,246,1,16,130,5,24,2,32,4>> ).

The problem I have is with the order of items in the list returned by piqirun:parse_record( Binary). Looking in the code I see that you order the list by ascending Code. Why? Am I abusing the integer Code?
In my case, when I concatenate some binaries from multiple piqi calls and send them over tcp, they come in in an order that I don't want.

Should I modify piqirun, or is there something I'm missing?

I really like the idea of the integer prefix, that way I can abstract my transport information from the payload, and don't have to hard code recordtypes in the protocol.

Thanks

Bart

[bart van deenen]

Hi Anton

maybe multiple calls to piqirun:parse_field/1. ?

Bart

[Anton Lavrik]

Hi Bart,

The integer Code you are talking about is a Protocol Buffers field tag
as specified by the Protocol Buffers documentation.

The specification also imposes certain requirements on how fields
should be decoded. For instance, if there are several required or
optional fields with the same Code, the last one should be taken as
the value of the field. The easiest way to implement this in Erlang is
to sort fields by codes as a first step. A good Protocol Buffers
implementation should order fields during serialization anyway.

But to be honest, this is an internal implementation detail. The
parse_record() function is meant to be internal and it is exposed only
for convenience. So, yes, by using it directly you are kind of abusing
the library.

You can still achieve the desired effect by assigning codes according
to the order you expect to have them in your input.

On the other hand, if you are worried about the abuse and portability,
you can easily implement a simple custom framing scheme. Essentially,
you want to serialize sections of {Code, Length, Data} that you can
stuck together. It shouldn't take more than a few dozens lines of code
in Erlang.

Anton