Protocol Design - byte order
ron....@lewisinnovative.com
desires
our firm to implement the software solution for this protocol.
First thing to note is that this protocol has been designed by one
architect and has not gone through a peer review.
The first issue that is glaring at me is that the protocol has been
designed in Least Significant Byte (LSB) first. Most protocols that I
am aware of are MSB first.
I asked why it is in LSB format and the response was "So that it was
easier for the programmer. It has an x86 processor."
This does not seem right to me.
The customer is willing to listen for arguements as to why to change it
to MSB
first but I am not sure why most of the network protocols
are in "network byte order". Is it a historical thing in that the early
protocols
were on UNIX systems (Big Endian)?
Maybe a LSB first order will work, but it seems that there must be a
reason
why the standard protocols are MSB first. My initial thought is that
network sniffers may have a problem
analyzing a LSB first protocol.
Any ideas appreciated.
A side note, this new protocol will be implemented on top of a ASYNC
serial port, it is similar to PPP.
thanks
Ron
George Wicks
Usually, there is a set of library function which the application developer
can use to convert between Host and Network byte order. These functions are
htons() ["host to network short"], htonl() ["host to network long"], ntohs
["network to host short"], and ntohl() ["network to host long"]. If you can
do a man page lookup on them, or see the documentation, you can see how to
use them.
Network byte order is defined as Big Endian, whereas Host byte order is the
byte ordering native to the host's machine architecture, whether it is Big
or Little Endian. If the host machine is Big Endian and you call one of the
functions, there will be no conversion done since it is already in Big
Endian format from both the host and network perspective. If the host
machine is in Little Endian format, such as a x86 based system, then these
byte ordering functions will do the byte swapping for you.
Really, it's not a big deal - that's what this set of functions are here
for.
Hope this helps.
ron....@lewisinnovative.com wrote in news:1131467140.851016.285880
@g44g2000cwa.googlegroups.com:
Steve Horsley
> Hello all,I have a customer who has designed a new network protocol and
> desires
> our firm to implement the software solution for this protocol.
> First thing to note is that this protocol has been designed by one
> architect and has not gone through a peer review.
>
> The first issue that is glaring at me is that the protocol has been
> designed in Least Significant Byte (LSB) first. Most protocols that I
> am aware of are MSB first.
> I asked why it is in LSB format and the response was "So that it was
> easier for the programmer. It has an x86 processor."
> This does not seem right to me.
<snip>
A network protocol that goes LSB first will stand out as an
oddity, but if the designers aren't embarrassed about that, all
well and good. As long as the documentation is clear about the
exact byte order for all structures larger than a byte, then the
protocol spec is adequate.
As a programmer, you should ensure that your code will work
regardless of he processor it finds itself on. Since they have
chosen non-standard byte ordering, the standard byte-ordering
library routines, htons and friends, won't work for you and you
will have to to a lot of shifting and masking yourself. As the
programmer, you could point ou that their choice is actually
harder for you - maybe they will accept that and change the spec.
That's my tuppence worth.
Steve
Le Chaud Lapin
Steve Horsley wrote:
> ron....@lewisinnovative.com wrote:
> > Hello all,I have a customer who has designed a new network protocol and
> > desires
> > our firm to implement the software solution for this protocol.
> > First thing to note is that this protocol has been designed by one
> > architect and has not gone through a peer review.
> >
> > The first issue that is glaring at me is that the protocol has been
> > designed in Least Significant Byte (LSB) first. Most protocols that I
> > am aware of are MSB first.
> > I asked why it is in LSB format and the response was "So that it was
> > easier for the programmer. It has an x86 processor."
> > This does not seem right to me.
>
> <snip>
>
> A network protocol that goes LSB first will stand out as an
> oddity, but if the designers aren't embarrassed about that, all
> well and good. As long as the documentation is clear about the
> exact byte order for all structures larger than a byte, then the
> protocol spec is adequate.
True. And in addition, it should be pointed out that if the entire
world spoke only Hebrew, Arabic, or other left-to-right languages, we'd
all be using LSB, and MSB never would have existed.
> As a programmer, you should ensure that your code will work
> regardless of he processor it finds itself on. Since they have
> chosen non-standard byte ordering, the standard byte-ordering
> library routines, htons and friends, won't work for you and you
> will have to to a lot of shifting and masking yourself. As the
> programmer, you could point ou that their choice is actually
> harder for you - maybe they will accept that and change the spec.
This is the crux of this (50-year-old) argument. Those who could never
possibly get confused about what means what (once they take a glimpse
at the spec) are LSB types.
Those who would rather have the computer "think like them" rather than
the other way around choose MSB.
-Le Chaud Lapin-
George Wicks
@news.freedom2surf.net:
> ron....@lewisinnovative.com wrote:
>> Hello all,I have a customer who has designed a new network protocol and
>> desires
>> our firm to implement the software solution for this protocol.
>> First thing to note is that this protocol has been designed by one
>> architect and has not gone through a peer review.
>>
>> The first issue that is glaring at me is that the protocol has been
>> designed in Least Significant Byte (LSB) first. Most protocols that I
>> am aware of are MSB first.
>> I asked why it is in LSB format and the response was "So that it was
>> easier for the programmer. It has an x86 processor."
>> This does not seem right to me.
>
> <snip>
>
> A network protocol that goes LSB first will stand out as an
> oddity, but if the designers aren't embarrassed about that, all
> well and good. As long as the documentation is clear about the
> exact byte order for all structures larger than a byte, then the
> protocol spec is adequate.
>
Mea Culpa. I stand corrected. I was under the perhaps mistaken impression
that the original poster could in fact change the byte ordering for proper
transmission onto a network, so that both Big and Little Endian machines
would be able to use the data - by the way, if that is the <requirement>.
It may well be that this protocol is designed in a strictly proprietary
sense. Yes, if I were to design this, I would make the argument that the
protocol should be done strictly observing Network Byte Order, <but> that
might not be a requirement that the company desires or cares about. Should
the company care about it? Darn right. But try telling an 800 lb. Gorilla
to move..
> As a programmer, you should ensure that your code will work
> regardless of he processor it finds itself on. Since they have
> chosen non-standard byte ordering, the standard byte-ordering
> library routines, htons and friends, won't work for you and you
> will have to to a lot of shifting and masking yourself.
Hmmm, htons & the other byte-ordering library do <work> for you, IMHO. It
depends upon the data that is encapsulated within the packet. If the datum
are atomic, e.g. a sequence of shorts, for example, then the byte-ordering
functions are fine. Now, when you're talking about "shifting" and
"masking", that's another arena. If, for example, you are attempting to
transmit a block of data that is defined by, say, a C structure, in that
instance, of course you are going to have to be concerned with padding in
between member fields within the structure. And actually, with regards to
padding, that is <usually> not something that is encountered on x86 based
machines - virtually all of the C compilers for the x86 do in fact do byte
packing, i.e., no padding between fields. That of course is definitely not
the case with other architectures that may align fields on 32-bit
boundaries, 16-byte paragraphs, etc., in which case you will need to do the
specific data conversions like shifting and so forth. Take a look at a
typical TCP/IP stack, and you'll see a good deal of data conversion at
work.
> As the
> programmer, you could point ou that their choice is actually
> harder for you - maybe they will accept that and change the spec.
If the protocol is intended to be proprietary, and they've made the
requirement such that all of the machines will only use the same endianness
and the same alignment of fields, and I'd add that it will never change in
the future (of course we all know that will be true), then making the
argument for putting in the extra work for compatibility with other
architectures will most likely fall on deaf ears.
If, on the other hand, this protocol is destined for an RFC, or if any of
the above conditions are not always going to be true, then the poster
should tell the original designer that they better change the spec, because
it will certainly fail to gain acceptance, or they will have to deal with a
lot of customer support.
My hat off to the original poster, by the way.
sleb...@gmail.com
> > This does not seem right to me.
>
> <snip>
>
> regardless of he processor it finds itself on. Since they have
> chosen non-standard byte ordering, the standard byte-ordering
> library routines, htons and friends, won't work for you and you
> will have to to a lot of shifting and masking yourself. As the
> programmer, you could point ou that their choice is actually
> harder for you - maybe they will accept that and change the spec.
The company I work for got bitten by the x86 assumption recently. A lot
of our C code lazily assumed a little endian CPU without doing the
htons/ntohs thing. But our application got so popular that marketing
wanted us to port it to an embedded system and management chose ARM for
the platform due to popularity & cost. And what do you know.. the
application stopped working because ARM is big endian.
Now, htons/ntohs doesn't work transparently when compiled for both
architecture because the functions assumes that the network byte order
is big endian and we wanted it the other way around. So I had to write
my own library functions (actually just copied htons and friends) for
it. And then we had to retrofit the functions into the hundreds of
files of C code. This delayed product launch enough to cost us a big
customer.
So, the lesson is.. when transmitting data, always do it network byte
order (as programmers we should not care about lsb or msb, just network
and host byte order).
Craig Partridge
>The customer is willing to listen for arguements as to why to change it
>to MSB
>first but I am not sure why most of the network protocols
>are in "network byte order". Is it a historical thing in that the early
>protocols
>were on UNIX systems (Big Endian)?
This issue cropped up in the very early days -- the perception that
a byte ordering choice "harmed" whichever processors did it the other
way.
There's a famous paper on the subject: Danny Cohen's "On Holy Wars and
A Plea for Peace"
Over the years, there have been attempts to allow hosts to negotiate
which byte order to use (the NDR representation) and the conclusion is
that, while useful, it wasn't useful enough.
More generally, the penalty for defining your own byte representation
vs. using a standard one is very high in pain of deployment.
Craig
EventHelix.com
probably just historical.
It does reminder me of Gulliver's travels where one set of people
wanted to break an egg from the smaller side and the other side wanted
to break it from the larger side. They almost went to war over this. :)
Also note that the byte ordering is more a function of the processor
architecture than anything else.
The following article might help:
http://www.eventhelix.com/RealtimeMantra/ByteAlignmentAndOrdering.htm
--
EventStudio System Designer 2.5 - http://www.EventHelix.com/EventStudio
Design and Document Protocol Interactions as Sequence Diagrams
Charles Bryant
<ron....@lewisinnovative.com> wrote:
>Hello all,I have a customer who has designed a new network protocol and
>desires
>our firm to implement the software solution for this protocol.
>First thing to note is that this protocol has been designed by one
>architect and has not gone through a peer review.
>
>The first issue that is glaring at me is that the protocol has been
>designed in Least Significant Byte (LSB) first. Most protocols that I
>am aware of are MSB first.
LSB first is quite popular, particularly with ITU standards. This is
partly a side effect of it being more a logical choice in some
situations, nad partly historical accident.
Protocols for LANs and the Internet are normally MSB first, leading to
some rather inelegant bit ordering. For example, the Ethernet 'length'
field was specifid to be most significant octet first, even though the
bits are sent least significant first, so they come out:
8, 9, 10,11,12,13,14,15, 1, 2, 3, 4, 5, 6, 7
>I asked why it is in LSB format and the response was "So that it was
>easier for the programmer. It has an x86 processor."
>This does not seem right to me.
Indeed, it is not right. That is a very bad reason for an aspect of a
protocol. I would be afraid that anyone advancing it as a reason
intends to write code like this:
v = *(uint32_t *) p;
which is very bad since, not only does it fail on systems which are
not strictly little-endian, it also fails on little-endian systems
where the pointer is not appropriately aligned, so this:
v = ntohl( *(uint32_t *) p);
is nearly as bad. It's only better because an alignment error is
likely to raise an exception and be easily found, while a byte-order
problem can lurk unnoticed for a long time in fields which are rarely
used.
>The customer is willing to listen for arguements as to why to change it
>to MSB first but I am not sure why most of the network protocols
>are in "network byte order".
I believe it's purely historical accident, though it makes much more
sense for addresses to be sent most-significant-part first, so that
may have influenced protocol designers. Note that although the
Ethernet specification claims that octets are sent least significant
bit first, the most significant bit of the destination address field
is actually the fist bit in a frame, but the description of the
address field is worded in such a way as to conceal this.
Michael Wojcik
In article <1131623439.5...@g14g2000cwa.googlegroups.com>, "EventHelix.com" <event...@gmail.com> writes:
>
> It does reminder me of Gulliver's travels where one set of people
> wanted to break an egg from the smaller side and the other side wanted
> to break it from the larger side. They almost went to war over this. :)
That's where the terms "big-endian" and "little-endian" come from.
Danny Cohen may not have been the first to make the comparison, but
his "On Holy Wars and a Plea for Peace" (IEN 137) is probably the
most famous treatment of it. I believe that document has already
been mentioned in this thread.
(Incidentally, is there a current official repository of IENs?
rfc-editor doesn't seem to have them. normos.org apparently no
longer exists. ftp.apnic.net was the only site I could find that
seems to have a complete collection, under ietf/ien. I know they're
obsolete now, but they're interesting for historical reasons.)
--
Michael Wojcik michael...@microfocus.com
Is it any wonder the world's gone insane, with information come to be
the only real medium of exchange? -- Thomas Pynchon
Steve Horsley
>>
>> A network protocol that goes LSB first will stand out as an
>> oddity, but if the designers aren't embarrassed about that, all
>> well and good. As long as the documentation is clear about the
>> exact byte order for all structures larger than a byte, then the
>> protocol spec is adequate.
>>
>
> Mea Culpa. I stand corrected. I was under the perhaps mistaken impression
> that the original poster could in fact change the byte ordering for proper
> transmission onto a network, so that both Big and Little Endian machines
> would be able to use the data - by the way, if that is the <requirement>.
Are we at cross-purposes here? What I'm trying to say is that if
the protocol spec is specific about byte ordering, then it is a
complete spec. It may say that integers go LSB first, or it may
say that MSB goes first. Either way, the byte order is fully
defined, such that anyone can read the spec and from that, build
an implementation or an analyser. The choice of byte order may
have an impact on how easy it is to implement on different
hardware platforms, but an implementation is possible on ANY
hardware, regardless of its internal memory organisation.
>
>> As a programmer, you should ensure that your code will work
>> regardless of he processor it finds itself on. Since they have
>> chosen non-standard byte ordering, the standard byte-ordering
>> library routines, htons and friends, won't work for you and you
>> will have to to a lot of shifting and masking yourself.
>
> Hmmm, htons & the other byte-ordering library do <work> for you, IMHO.
Not if your protocol is not in Network Byte Order. Example, for a
16 bit short on x86, htons will swap the bytes, converting the
byte pair from internal endian-ness to NBO big-endian-ness. If
you want to send the bytes in little-endian-ness, htons is the
last thing you want to do. Equally, on a ??? processor, htons is
a no-op because the short already stored in NBO. No amount of
calling htons will swap the bytes to little-endian.
So htons specifically will NOT help convert your integers to
little-endian. You have to specifically say something like:
packet[0] = int1;
packet[1] = int1 >> 8;
packet[2] = int1 >> 16;
packet[3] = int1 >> 24;
> If the protocol is intended to be proprietary, and they've made the
> requirement such that all of the machines will only use the same endianness
> and the same alignment of fields, and I'd add that it will never change in
> the future (of course we all know that will be true), then making the
> argument for putting in the extra work for compatibility with other
> architectures will most likely fall on deaf ears.
>
Again, the machine archtecture and protocol spec are only loosely
connected by mild differences in ease of
constructing/deconstructing the message, i.e. whether the bytes
match the in-memory structure.
Steve
Le Chaud Lapin
Le Chaud Lapin wrote:
> Those who would rather have the computer "think like them" rather than
> the other way around choose MSB.
I came across the following web page via random searching for
information on reprogrammable routers. Obviously, the author of the
page would have no problem explaining endianess, but it appears that
the example is wrong:
http://www.bluewatersys.com/corporate/uni/unikit/other/bigendian.php
I would argue that little-endian mindsets are less prone to this type
of error than big-endian mindsets.
-Le Chaud Lapin-
Steve Horsley
It looks right to me. Why do you think it's wrong?
Steve
Le Chaud Lapin
It looks as if the ordering of the bytes in the C/C++ constant
(0x01234567) is not congruent to the ordering in the table, per his
discussion.
Disclaimer: I been out partying all night so the brains and eyes could
be conspiring against me at this very moment. Silly brains and eyes.
-Le Chaud Lapin-