Sockets, Streams, and Element_Arrays: Much confusion

[Mark Gardner]

Hello, I've been having a bit of difficulty doing some UDP socket
programming in Ada. As outlined in my stackoverflow question here
(https://stackoverflow.com/q/74953052/7105391), I'm trying to reply to
messages I am getting over UDP.

GNAT.Sockets gives me a Stream_Element_Array, which I can't find any
documentation on how to make use of other than "You should also be able
to get a Stream, which you should use instead" (About ten years ago, on
this very newsgroup, somebody said not to use streams with UDP, or at
least not GNAT.Sockets.Stream).

Adasockets gives me a String, which I can work with, except it throws
away the from data recvfrom gives it, apparently making it impossible to
reply to the querying address.

At this point, I'm half-tempted to make my own binding, but as I've
never done that sort of thing before, I thought I'd ask the wisdom of
the Usenet if there is a way to convert a Stream_Element_Array into the
exotic types of Unsigned_16 and String.

[Dmitry A. Kazakov]

On 2022-12-31 13:11, Mark Gardner wrote:

> GNAT.Sockets gives me a Stream_Element_Array, which I can't find any
> documentation on how to make use of other than "You should also be able
> to get a Stream, which you should use instead" (About ten years ago, on
> this very newsgroup, somebody said not to use streams with UDP, or at
> least not GNAT.Sockets.Stream).

Stream_Element_Array is declared in Ada.Streams as

type Stream_Element_Array is
array(Stream_Element_Offset range <>) of
aliased Stream_Element;

For communication purpose it is an array of octets. Your datagram is
represented as a Stream_Element_Array or a slice of.

As for streams, yes, it does not make sense to use them for networking,
unless you override all stream primitives. The reasons for that are

- non-portability of predefined primitives
- low efficiency for complex data types
- encoding inefficiency as well

You will need to handle some application protocol artifacts, checksums,
counters, strange encodings, sequence number etc. It is easier to this
directly on the Stream_Element_Array elements.

And, well, do not use UDP, expect for broadcasting. There is no reason
to use it. For multicast consider delivery-safe protocols like PGM. For
single cast use TCP/IP. (If you need low latency see the socket NO_DELAY
option)

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

[Mark Gardner]

On 31/12/2022 15:11, Dmitry A. Kazakov wrote:
> On 2022-12-31 13:11, Mark Gardner wrote:
>

>> ...

>
> Stream_Element_Array is declared in Ada.Streams as
>
>    type Stream_Element_Array is
>       array(Stream_Element_Offset range <>) of
>          aliased Stream_Element;
>
> For communication purpose it is an array of octets. Your datagram is
> represented as a Stream_Element_Array or a slice of.
>

According to RM 13.13.1, "Stream_Element is mod implementation-defined"
which to me says there is no guarantee that they will be octets, unless
this is specified elsewhere?

> As for streams, yes, it does not make sense to use them for networking,
> unless you override all stream primitives. The reasons for that are
>
> - non-portability of predefined primitives
> - low efficiency for complex data types
> - encoding inefficiency as well
>
> You will need to handle some application protocol artifacts, checksums,
> counters, strange encodings, sequence number etc. It is easier to this
> directly on the Stream_Element_Array elements.

So, how would I do this directly on the elements? I mean, if it is an
octet-array to a string, I expect an element-to-element copy, or type
conversion to work, but what about integers? Do I need to do something like
My_Int:=Unsigned_8(octet(1))+2**8*Unsigned_8(octet(2));
or whatever endianness demands? Or is this the time to learn how to use
Unchecked_Conversion?

> And, well, do not use UDP, expect for broadcasting. There is no reason
> to use it. For multicast consider delivery-safe protocols like PGM. For
> single cast use TCP/IP. (If you need low latency see the socket NO_DELAY
> option)
>

Well, my use case just so happens to be broadcasting, and
re-broadcasting data across a binary-tree-like p2p network.

[Dmitry A. Kazakov]

On 2022-12-31 14:50, Mark Gardner wrote:
> On 31/12/2022 15:11, Dmitry A. Kazakov wrote:
>> On 2022-12-31 13:11, Mark Gardner wrote:
>>
>>> ...
>>
>> Stream_Element_Array is declared in Ada.Streams as
>>
>>     type Stream_Element_Array is
>>        array(Stream_Element_Offset range <>) of
>>           aliased Stream_Element;
>>
>> For communication purpose it is an array of octets. Your datagram is
>> represented as a Stream_Element_Array or a slice of.
>
> According to RM 13.13.1, "Stream_Element is mod implementation-defined"
> which to me says there is no guarantee that they will be octets, unless
> this is specified elsewhere?

GNAT.Sockets is GNAT-specific. All GNAT compilers have Stream_Element 8
bits. I can imagine some DSP implementation with Stream_Element of 32
bits. But realistically add

pragma Assert (Stream_Element'Size >= 8);

and be done with that.

> So, how would I do this directly on the elements? I mean, if it is an
> octet-array to a string, I expect an element-to-element copy, or type
> conversion to work, but what about integers?

Hmm, it cannot be string. It is a string encoded in some specific way
(usually most peculiar (:-)). Then you will have to decode it into your
machine type e.g. Wide_String or String.

An UTF-8 string you could put into String ignoring Ada's Latin-1 stuff,
as most people would do:

function To_String (S : Stream_Element_Array) return String is
begin
return Result : String (1..S'Length) do
for I in S'Range loop
Result (Positive (I - S'First + 1) := Character'Val (S (I));
end loop;
end return;
end To_String;

> Do I need to do something like
> My_Int:=Unsigned_8(octet(1))+2**8*Unsigned_8(octet(2));
> or whatever endianness demands? Or is this the time to learn how to use
> Unchecked_Conversion?

There are many ways to convert Stream_Element_Array "in situ" to string.
However, in network protocols you rarely have any strings at all.
Usually it is some binary data you need to decode into some machine
type. (So is String or Wide_String actually)

[Jeffrey R.Carter]

On 2022-12-31 14:50, Mark Gardner wrote:
>

> According to RM 13.13.1, "Stream_Element is mod implementation-defined" which to
> me says there is no guarantee that they will be octets, unless this is specified
> elsewhere?

The ARM has always tried to ensure that the language could be implemented on any
kind of processor. Thus you have implementation-defined separate definitions of
Storage_Element and Stream_Element, which need not be the same, and no guarantee
that Interfaces contains declarations of Integer_8 or Unsigned_8.

But these days almost everything is byte oriented, so unless you need what
you're writing to work on some unusual H/W, you can presume that both of these
are bytes, and that Interfaces contains those declarations.

--
Jeff Carter
"My legs are gray, my ears are gnarled, my eyes are old and bent."
Monty Python's Life of Brian
81

[Simon Wright]

Mark Gardner <magard...@gmail.com> writes:

> GNAT.Sockets gives me a Stream_Element_Array, which I can't find any
> documentation on how to make use of other than "You should also be
> able to get a Stream, which you should use instead" (About ten years
> ago, on this very newsgroup, somebody said not to use streams with
> UDP, or at least not GNAT.Sockets.Stream).

The reasoning behind the recommendation not to use streams with UDP was
as follows (there's a faint possibility that it no longer applies!)

If the data type you want to send is e.g.

type Message is record
Id : Integer;
Val : Boolean;
end record;

and you create a datagram socket and from that a stream, then use
Message'Write to the stream, GNAT will transmit each component of
Message separately in canonical order (the order they're written in the
type declaration). This results in two datagrams being sent, one of 4
bytes and one of 1 byte.

If you take the same approach at the destination, Message'Read reads one
datagram of 4 bytes, and one of 1 byte, and it all looks perfect from the
outside. If the destination is expecting a 5 byte record, of course,
things won't work so well.

The approach we adopted was to create a 'memory stream', which is a
chunk of memory that you can treat as a stream (see for example
ColdFrame.Memory_Streams at [1]). With Ada2022, you should be able to
use Ada.Streams.Storage.Bounded[2].

Message'Write the record into the memory stream;
transmit the written contents as one datagram.

To read, create a memory stream large enough for the message you expect;
read a datagram into the memory stream;
Message'Read (Stream => the_memory_stream, Item => a_message);

You can use gnatbind's switch -xdr to "Use the target-independent XDR
protocol for stream oriented attributes instead of the default
implementation which is based on direct binary representations and is
therefore target-and endianness-dependent".

[1]
https://github.com/simonjwright/coldframe/blob/master/lib/coldframe-memory_streams.ads
[2] http://www.ada-auth.org/standards/22rm/html/RM-13-13-1.html#p25

[Mark Gardner]

On 31/12/2022 19:39, Simon Wright wrote:
> [...]

>
> The approach we adopted was to create a 'memory stream', which is a
> chunk of memory that you can treat as a stream (see for example
> ColdFrame.Memory_Streams at [1]). With Ada2022, you should be able to
> use Ada.Streams.Storage.Bounded[2].
>

Wait, so if I know what shape my data is, and use a memory_stream (like
the one in the Big Online Book of Linux Ada Programming chapter 11 [1]),
I'm fine using Stream, in conjunction with Get_Address? That's
wonderful. Not at all frustrated that I just wasted approximately three
working days looking for a solution to a problem that didn't exist.

> Message'Write the record into the memory stream;
> transmit the written contents as one datagram.

I'm guessing with Memory_Stream'Write(Socket_Stream, Buffer);?

>
> To read, create a memory stream large enough for the message you expect;
> read a datagram into the memory stream;
> Message'Read (Stream => the_memory_stream, Item => a_message);

Does this second buffer need to be added? If the datagram arrives (UDP),
shouldn't GNAT.Sockets.Stream() be able to handle it?

>
> You can use gnatbind's switch -xdr to "Use the target-independent XDR
> protocol for stream oriented attributes instead of the default
> implementation which is based on direct binary representations and is
> therefore target-and endianness-dependent".

Oh fun, I didn't think of that aspect. Thanks! Would I have to pass it
as a command line flag, or would there be some kind of pragma I could use?

Thanks for the help so far, and happy new year!

[1] http://www.pegasoft.ca/resources/boblap/11.html#11.12

[Dmitry A. Kazakov]

On 2022-12-31 20:36, Mark Gardner wrote:
> On 31/12/2022 19:39, Simon Wright wrote:

>> Message'Write the record into the memory stream;
>> transmit the written contents as one datagram.
>
> I'm guessing with Memory_Stream'Write(Socket_Stream, Buffer);?

No, you create a memory stream object. Then you write your packet into it:

My_Message'Write (My_Memory_Stream'Access);

Once written you use the accumulated stream contents to write it into
the socket. An implementation of a memory-resident stream is very
simple. E.g. see:

http://www.dmitry-kazakov.de/ada/strings_edit.htm#Strings_Edit.Streams

My advise would be not to do this. It is wasting resources and
complicated being indirect when 'Write and 'Read are compiler-generated.
If you implement 'Write and 'Read yourself, then why not calling these
implementations directly. It just does not make sense to me. I always
wonder why people always overdesign communication stuff.

Build messages directly in a Stream_Element_Array. Use
system-independent ways to encode packet data. E.g. chained codes for
integers. Mantissa + exponent for real numbers. If you have Booleans and
enumerations it is a good idea to pack them into one or two octets to
shorten the packets. All this is very straightforward and easy to implement.

You can also consider using some standard data representation format,
e.g. ASN.1. An Ada ASN.1 implementation is here:

http://www.dmitry-kazakov.de/ada/components.htm#ASN.1

You describe your message in ASN.1 as an Ada tagged type derived from
building blocks. Then you can encode and decode it directly from
Stream_Element_Array. I would not recommend that either. ASN.1 is quite
overblown.

Happy New Year!

[philip...@gmail.com]

On Saturday, December 31, 2022 at 5:11:14 AM UTC-8, Dmitry A. Kazakov wrote:

> And, well, do not use UDP, expect for broadcasting. There is no reason
> to use it. For multicast consider delivery-safe protocols like PGM. For
> single cast use TCP/IP. (If you need low latency see the socket NO_DELAY
> option)

I have to disagree here. UDP is perfectly fine for RPC-like (Remote Procedure Call) transactions on a local area network. And it is orders of magnitude easier to implement on microcontrollers than TCP. An Ada program using UDP to communicate with data collecting microcontrollers makes perfect sense in some contexts. I use it for my Remote I/O Protocol.

The only trick is that the server (or responder, as I like to call it) and client (or initiator) can't quite use the same code.

Here is my generic package for UDP with fixed length messages:

https://github.com/pmunts/libsimpleio/blob/master/ada/objects/messaging-fixed-gnat_udp.ads
https://github.com/pmunts/libsimpleio/blob/master/ada/objects/messaging-fixed-gnat_udp.adb

Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time. If somebody has a better idea, let me know.

[Jeffrey R.Carter]

On 2022-12-31 23:32, philip...@gmail.com wrote:
>
> Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time. If somebody has a better idea, let me know.

You should be able to use Unchecked_Conversion for that.

[Dmitry A. Kazakov]

On 2022-12-31 23:32, philip...@gmail.com wrote:

> I have to disagree here. UDP is perfectly fine for RPC-like (Remote Procedure Call) transactions on a local area network.

RPC and other synchronous exchange policies should be avoided as much as
possible.

Saying said that, implementation of RPC on top of streams is
incomparable more easier than on top of UDP.

> And it is orders of magnitude easier to implement on microcontrollers than TCP.

Not at all. You need:

- Safe transmission and error correction on top UDP;
- Buffering and sorting out incoming datagrams;
- Maintaining sequence numbers;
- Splitting messages that do not fit into a single datagram and
reassembling them on the receiver side;
- Buffering on the sender side to service resend requests.

This is extremely difficult and huge load for a microcontroller.

> Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time. If somebody has a better idea, let me know.

Use "in situ" conversion if you are concerned about copying. E.g.

pragma Import (Ada, Y);
for Y'Address use X'Address;

[Simon Wright]

"Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:

> My advise would be not to do this. It is wasting resources and
> complicated being indirect when 'Write and 'Read are
> compiler-generated. If you implement 'Write and 'Read yourself, then
> why not calling these implementations directly. It just does not make
> sense to me. I always wonder why people always overdesign
> communication stuff.
>
> Build messages directly in a Stream_Element_Array. Use
> system-independent ways to encode packet data. E.g. chained codes for
> integers. Mantissa + exponent for real numbers. If you have Booleans
> and enumerations it is a good idea to pack them into one or two octets
> to shorten the packets. All this is very straightforward and easy to
> implement.

It has to depend on the design criteria.

If you need something now, and it's not performance critical, and you
have control over both ends of the channel, why not go for a
low-brain-power solution?

On the other hand, when faced with e.g. SNTP, why not use Ada's
facilities (e.g. [1]) to describe the network packet and use unchecked
conversion to convert to/from the corresponding stream element array to
be sent/received?

I'd have thought that building messages directly in a stream element
array would be the least desirable way to do it.

[1] https://sourceforge.net/p/coldframe/adasntp/code/ci/default/tree/SNTP.impl/sntp_support.ads

[Dmitry A. Kazakov]

On 2023-01-01 00:41, Simon Wright wrote:
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>
>> My advise would be not to do this. It is wasting resources and
>> complicated being indirect when 'Write and 'Read are
>> compiler-generated. If you implement 'Write and 'Read yourself, then
>> why not calling these implementations directly. It just does not make
>> sense to me. I always wonder why people always overdesign
>> communication stuff.
>>
>> Build messages directly in a Stream_Element_Array. Use
>> system-independent ways to encode packet data. E.g. chained codes for
>> integers. Mantissa + exponent for real numbers. If you have Booleans
>> and enumerations it is a good idea to pack them into one or two octets
>> to shorten the packets. All this is very straightforward and easy to
>> implement.
>
> It has to depend on the design criteria.
>
> If you need something now, and it's not performance critical, and you
> have control over both ends of the channel, why not go for a
> low-brain-power solution?

Because it is still much simpler (and safer) than record layouts.

> On the other hand, when faced with e.g. SNTP, why not use Ada's
> facilities (e.g. [1]) to describe the network packet and use unchecked
> conversion to convert to/from the corresponding stream element array to
> be sent/received?

Well decoding NTP query from Stream_Element_Array takes 2 statements
(extracting two big-endian 32-bit numbers). You can check the
implementation here:

http://www.dmitry-kazakov.de/ada/components.htm#17.17

which is under-zero brained (:-)) compared to dealing with definition of
record layout and bit orders you suggest.

[Simon Wright]

It's obviously OK to use just the fields of interest in the received
packet.

But this strikes me as something I would have questioned at review:

Data : Stream_Element_Array (1..NTP_Packet_Size) :=
( 1 => 2#1110_0011#, -- LI, Version, Mode
2 => 0, -- Stratum, or type of clock
3 => 0, -- Polling Interval
4 => 16#EC#, -- Peer Clock Precision
13 => 49,
14 => 16#4E#,
15 => 49,
16 => 52,
others => 0
);

[Niklas Holsti]

On 2023-01-01 1:41, Simon Wright wrote:
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>
>> My advise would be not to do this. It is wasting resources and
>> complicated being indirect when 'Write and 'Read are
>> compiler-generated. If you implement 'Write and 'Read yourself, then
>> why not calling these implementations directly. It just does not make
>> sense to me. I always wonder why people always overdesign
>> communication stuff.
>>
>> Build messages directly in a Stream_Element_Array. Use
>> system-independent ways to encode packet data. E.g. chained codes for
>> integers. Mantissa + exponent for real numbers. If you have Booleans
>> and enumerations it is a good idea to pack them into one or two octets
>> to shorten the packets. All this is very straightforward and easy to
>> implement.
>
> It has to depend on the design criteria.
>
> If you need something now, and it's not performance critical, and you
> have control over both ends of the channel, why not go for a
> low-brain-power solution?
>
> On the other hand, when faced with e.g. SNTP, why not use Ada's
> facilities (e.g. [1]) to describe the network packet and use
> unchecked conversion to convert to/from the corresponding stream
> element array to be sent/received?
>

> [1]
https://sourceforge.net/p/coldframe/adasntp/code/ci/default/tree/SNTP.impl/sntp_support.ads


One reason is that the Scalar_Storage_Order representation aspect is
GNAT-specific, so that code is not really "Ada code", but "GNAT code".
With most processors now being little-endian, while network traffic is
still big-endian, byte-endianness conversion is usually necessary.

[Daniel Norte de Moraes]

Em Sat, 31 Dec 2022 14:11:55 +0200, Mark Gardner escreveu:
> At this point, I'm half-tempted to make my own binding, but as I've
> never done that sort of thing before, I thought I'd ask the wisdom of
> the Usenet if there is a way to convert a Stream_Element_Array into the
> exotic types of Unsigned_16 and String.

You are obliged to use gnat-sockets ?
No? then use https://github.com/danieagle/adare-net

And Be Happy! (Enjoy!!)

p.s.: see the client and server in udp in example diretory.
p.s.: nowdays Adare_Net has a manual in pdf, too

Thanks All,
Dani.

[Dmitry A. Kazakov]

On 2023-01-05 08:55, Daniel Norte de Moraes wrote:
> Em Sat, 31 Dec 2022 14:11:55 +0200, Mark Gardner escreveu:
>> At this point, I'm half-tempted to make my own binding, but as I've
>> never done that sort of thing before, I thought I'd ask the wisdom of
>> the Usenet if there is a way to convert a Stream_Element_Array into the
>> exotic types of Unsigned_16 and String.
>
> You are obliged to use gnat-sockets ?
> No? then use https://github.com/danieagle/adare-net

The OP's question was about Stream_Element_Array being used for payload.
In that regard there is no difference between GNAT.Sockets and the
library you suggest.

P.S. I am not sure if the library supports socket select and setting all
necessary socket flags (e.g. NODELAY).