FBSP - Basic format for the variable part of the message

[Pavel Cisar]

Hi all,

The Firebird Butler Service Protocol (FBSP) concept defines a fixed
(common) and a variable part. Each part is represented by a separate
message frame. For the variable part, it is assumed to use a uniform
format or one of the acceptable formats from the standard defined
collection (see RAW notes for "type-data" field of HELLO/WELCOME message).

However, to solve the problem of selecting a variable part format, the
following questions need to be answered:

1. Does the FBSP standard have to define only one permissible format, or
should it allow one of several permissible formats to be used?

The admissibility of multiple formats presents a risk of reduced
interoperability of services (where client and service may not support
the same format), or increased cost of creating services and clients
(the need to support multiple formats).

I personally tend to use only one format.

2. Which format (or formats) to choose?

I think that the most suitable format for our needs (fast, compact
format with wide platform and language support) is the Google Protocol
Buffer (protobuf) format. The declarative approach used by protobuf also
significantly simplifies its definition (self-documentation) and
implementation (generated code).

NOTE: It's expected that we would use some from "well known" message
types (see
https://developers.google.com/protocol-buffers/docs/reference/google.protobuf).
Using protobuf without any of them (i.e. at very basic language support
level) is not good enough, as we would be forced to work around this
limitation somehow (for example to provide something like "struct" or
"timestamp").

Python protobuf support fully suits our needs and our intentions for use
within the Saturnin SDK. The question remains whether support for your
programming language is sufficient for your purposes? From what I saw
the Java should be ok, but there would be probably issues with
Delphi/Pascal support.

In the event that we fail to agree to the use of protobuf format, the
second format in the order that would suit our needs is the MsgPack
format (or suggest your alternative of choice except XML and JSON).

3. If the protobuf format is selected, which version (proto2 or proto3)
to use?

Here is a quick rundown of differences:

proto2:

a. Required (and optional) message fields with check of message
completeness.
b. Ability to set custom default values for a field.
c. Ability to determine if a missing field was not included, or was
assigned the default value.
d. Support for nested groups.

proto3:

a. Removal of required fields (all are optional).
b. Removal of default values. Primitive fields set to a default value
are not serialized.
c. Built-in JSON serializers/deserializers (but it could be solved
externally).
d. Extensions have been replaced by the Any type.
e. Addition of well-known type protos. These include any.proto,
timestamp.proto, duration.proto, etc. However, these should be available
in proto2 now as well, at least in officially supported languages
(verified it works in Python).
f. Strict UTF-8 checking is enforced.

Personally, I tend to prefer proto2 over proto3 due to the required
fields and the ability to detect empty (not serialized) fields.

You can use the attached example to test protobuf usability in your
programming language.

best regards
Pavel Cisar
IBPhoenix

[Paul Reeves]

Hmm, I can't see at the moment how this can be implemented cheaply in
Free Pascal.

A record type would be ideal, but AFAICT, you cannot nest records
recursively. Although you can nest a record within a record. So it
might be possible to do something ugly at that level.

On the other hand, I'm fairly sure this could be done with a TList and
TListItems. Whether the overhead is expensive or not is another
matter.

Paul

[Pavel Cisar]

Paul,

Dne 04. 02. 19 v 16:12 Paul Reeves napsal(a):

> Hmm, I can't see at the moment how this can be implemented cheaply in
> Free Pascal.

Did you checked out the protobuf part of Fundamentals library?
http://fundementals.sourceforge.net/dl.html

The second one I know about appears less evolved:
https://sourceforge.net/projects/protobuf-delphi/

In case that protobuf Pascal support would be not good enough for your
purposes, I see three options (in order of preference) how to deal with
that:

1. Improve protobuf code in Fundamentals to fit Butler needs.

2. Generate C code from Butler protobufs and create external library
that would wrap it for use in Pascal/Delphi.

3. Use other format that has better support in Pascal/Delphi (for
example MsgPack).

regards
Pavel

[Paul Reeves]

I must confess I have never studied this subject, but I came across
this, which looks like a useful summary :-)

https://scottlocklin.wordpress.com/2017/04/02/please-stop-writing-new-serialization-protocols/

Looking at the links to msgpack and the protobuf it seems to me that
msgpack has much wider support for different languages. I did find a
protoc for free pascal here:

https://code.google.com/archive/p/protoc-gen-pascal/source/default/source

Would that do the job?

I also tried to find comparisons between the msgpack and protobuf but
didn't come up with anything solid. Unless a site that claims ION is
faster than any of the competition. But adding more protocols to the
discussion is probably not helpful.

Anyway, I think you need to choose the one that has the best overall
fit for the project. It is too early for me to tell which one would be
best for FreePascal.


Paul

> --
> You received this message because you are subscribed to the Google Groups "Firebird Butler" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to firebird-butl...@googlegroups.com.
> To post to this group, send email to firebir...@googlegroups.com.
> Visit this group at https://groups.google.com/group/firebird-butler.
> To view this discussion on the web visit https://groups.google.com/d/msgid/firebird-butler/31712475-b8bb-9edd-fa83-714d0246a077%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.



--
Paul Reeves
http://www.ibphoenix.com

[Pavel Cisar]

Paul,

> Looking at the links to msgpack and the protobuf it seems to me that
> msgpack has much wider support for different languages.

Direct support (from Google) is only for C++, Java, Python, Objective-C,
C#, JavaScript, Ruby, Go, PHP and Dart. However, the language support
does not end there, see:

https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md

> I did find a protoc for free pascal here:
>
> https://code.google.com/archive/p/protoc-gen-pascal/source/default/source
>
> Would that do the job?

Hell, how I am supposed to know? I am not the Pascal SDK developer. I
thought that you would tell me.

AFAIK, the project you pointed out is not the sole option, see:
https://github.com/lalexs75/protobuf-fpc

So I know about at least 4 options for Pascal/Delphi, but I really don't
have time nor Pascal expertise to evaluate and judge them.

> I also tried to find comparisons between the msgpack and protobuf
> but didn't come up with anything solid. Unless a site that claims ION
> is faster than any of the competition. But adding more protocols to
> the discussion is probably not helpful.

Well, MessagePack is more close to JSON than protobuf, just more compact
over wire. The main difference between these two is that protobuf uses
static schema to describe the content and code to serialize/deserialize
them is generated while msgpack is just a data serialization format, and
it's up to the developer to handle all semantics.

> Anyway, I think you need to choose the one that has the best overall
> fit for the project. It is too early for me to tell which one would
> be best for FreePascal.

Well, that is the problem. I can evaluate the protocol only from Python
perspective. So I can easily pick up a protocol that works best in
Python but would give a headache to developers using other languages.

Here are conditions where such system would be used:

1. High traffic.

2. Quick data exchange between peers.

3. The schema of the transmitted data is predominantly stable. However,
a large number of individual schemes is expected. One message could use
multiple ones, even nested.

4. Each schema could change over time, so proper handling of versioning
is essential.

5. Extensible.

These conditions could be translated into next characteristics we need:

1. Fast serialization/deserialization in all languages. It does not need
to be equally fast in all languages, but it should be rated among 3
fastest options for each particular language.

2. Efficient, compact "wire" format.

3. Use of static schema is preferred, otherwise the schema has to be
handled in upper layer (like wrapper classes for example).

4. Should support all essential (used by us) data types, or allow easy
and efficient handling of those that are not directly supported.

Essential types beyond standard numeric, string, bytearray and boolean
types are:

- timestamp, duration
- list
- map (all basic types should be available for key values, all essential
types should be available for values, single map should handle values of
more than single value type)
- raw - encapsulation of unrecognized payload

5. Should allow detection of unserialized/unused parts of particular schema.

AFAIK both protobuf and msgpack more or less fit these requirements.

protobuf fits it better for all directly supported languages, but we
don't know how well it fits in Pascal. The general problem with protobuf
is that generated code could be shity and cumbersome. In this aspect,
the quality of generated Python code is acceptable for me, can't judge
Java, C#, Pascal and others.

Using MsgPack would require us to handle schemes in manually written
code for each language, and schema definition would be also more harder
than writing a self-describing proto file with comments. With volume of
schemes and versioning issues it could easily escalate to developer's
nightmare over time.

Well, and there are other options like FlatBuffers etc. that could be
also considered and discussed, but from those I checked the protobuf &
msgpack are IMHO first hot candidates to consider, but its just my
personal opinion.

So, where we'll go from here? MsgPack would be easier to start with, but
we will pay greatly for that by hard work. protobuf could be harder to
start with, but could spare us from a lot of tedious work. What poison
we will choose? Personally, I incline to second option, but can live
with either one.

regards
Pavel

[Paul Reeves]

If the package protoc-gen-pascal compiler actually works then it would
generate the correct code from a .proto definition. That is certainly
the kind of tool we need. I guess I need to find out more.
Unfortunately, it is stored in the google code archive so it may never
have worked, and is certainly unmaintained.

I guess for now we should go with protobuf. If it doesn't work out
then we will at least have a better idea of what we need. At the
moment I think we are just flying blind - at least I am. And I won't
have much time before the third week of february to look into this in
much detail.

Paul

> --
> You received this message because you are subscribed to the Google Groups "Firebird Butler" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to firebird-butl...@googlegroups.com.
> To post to this group, send email to firebir...@googlegroups.com.
> Visit this group at https://groups.google.com/group/firebird-butler.

> To view this discussion on the web visit https://groups.google.com/d/msgid/firebird-butler/0254c784-4fac-5662-2955-624721bb6e77%40gmail.com.

[Paul Reeves]

This looks as if it will do the job for FreePascal:

https://github.com/kami-soft/ProtoBufGenerator

I've cloned the repo and will try compiling it sometime in the near future.


Paul

[Pavel Cisar]

Hi all,

Dne 05. 02. 19 v 12:29 Paul Reeves napsal(a):

>
> I guess for now we should go with protobuf. If it doesn't work out
> then we will at least have a better idea of what we need. At the
> moment I think we are just flying blind - at least I am. And I won't
> have much time before the third week of february to look into this in
> much detail.

FBSP specification was updated with protobuf definitions and I also
created the spec file for protobuf compiler, see Butler/rfc/4/fbsp.proto

I also added initial set of error codes, added some new message exchange
patterns in Appendix A and did some additional adjustments:

- to CANCEL and STATE messages
- to 2.5 Handling of client requests
- ALL_REQUEST request code is gone and replaced by UNKNOWN request code
- Minor adjustments

for details see git repository.

Guess I have enough boilerplate to start some Python coding as we don't
need service state, config and control specifications for base plumbing
that can do some useful stuff. Although the Saturnin SDK should be based
primarily around asyncio and with use of async Python 3.5+ features,
I'll first write classic one that could be more easily replicated in
other languages. However, it's expected that this "classic" framework
could be somewhat limited in features and with worse performance in
comparison to async one. Which should not be a problem as the primary
goal for this stage is to create some very simple test services you
could use as examples and initial sparring partners for services you
will write in your language of choice.