TIP #129: New Format Codes for the [binary] Command
Arjen Markus
TIP #129: NEW FORMAT CODES FOR THE [BINARY] COMMAND
=====================================================
Version: $Revision: 1.1 $
Author: Arjen Markus <arjen.markus_at_wldelft.nl>
State: Draft
Type: Project
Tcl-Version: 8.5
Vote: Pending
Created: Friday, 14 March 2003
URL: http://purl.org/tcl/tip/129.html
WebEdit: http://purl.org/tcl/tip/edit/129
Post-History:
-------------------------------------------------------------------------
ABSTRACT
==========
This TIP proposes to add a set of new format codes to the [binary]
command to enhance its ability to deal with especially non-native
floating-point data. The assumption is that current limitations are due
to the distinction between little-endian and big-endian storage of such
data.
INTRODUCTION
==============
The current [binary] command can manipulate little-endian and
big-endian _integer_ data, but only _native_ floating-point data. This
means that binary data from other computer systems that use a different
representation of floating-point data can not be directly handled.
The lack of format codes to handle "native" integer data means that one
has to distinguish the current platform's byte ordering to be
platform-independent, whenever the [binary] command is used.
Most current computer systems use either little-endian or big-endian
byte order and the so-called IEEE representation for the exponent and
mantissa. So, the main variation to deal with is the endian-ness.
Some popular file formats, like ESRI's ArcInfo shape files, use both
types of byte order. It is difficult (though not impossible) to handle
these files with the current set of format codes.
It should be noted that there _is_ more variety among floating-point
representation than just the byte order. This TIP will not solve this
more general problem.
PROPOSED FORMAT CODES
=======================
Format codes should be available to catch the two main varieties of
byte ordering. There should, both for reasons of symmetry and for
practical purposes, also be a full set to deal with "native" data.
For integer types there are no codes to deal with native ordering. So:
* t (tiny) for short integers, using native ordering.
* n (normal) for ordinary integers, using native ordering.
* m (mirror of "w") for wide integers, using native ordering.
The floating-point types will be handled via:
* r/R (real) for single-precision reals.
* q/Q (mirror of "d") for double-precision reals.
where the lower-case is associated with little-endian order and the
upper-case with big-endian order.
IMPLEMENTATION NOTES
======================
The implementation for the integer types is simple:
* The new format codes are synonyms for the current ones, but
different ones for each endian-ness.
The implementation for the floating-point types is somewhat more
complicated, this involves adding byte swapping, if the ordering of the
platform does not correspond to that of the format code.
A reference implementation does _not_ yet exist.
COPYRIGHT
===========
This document is placed in the public domain.
-------------------------------------------------------------------------
TIP AutoGenerator - written by Donal K. Fellows
[[Send Tcl/Tk announcements to tcl-an...@mitchell.org
Announcements archived at http://groups.yahoo.com/group/tcl_announce/
Send administrivia to tcl-announ...@mitchell.org
Tcl/Tk at http://tcl.tk/ ]]
Andreas Leitgeb
> * t (tiny) for short integers, using native ordering.
> * n (normal) for ordinary integers, using native ordering.
> * m (mirror of "w") for wide integers, using native ordering.
network-byteorder (aka big-endian). for formats mainly used on PC-
platforms it tends to be little-endian independent of the current
platform that I actually use it on.
If you're bound to parse a binary stream whose byteorder is not fixed
but platform-dependent, it makes sense to make it explicitly
platform-dependent, as in:
if {$tcl_platform(byteOrder) eq "littleEndian"} {
set binformatstr "siw..."
} else {
set binformatstr "SIW..."
}
Still, I wouldn't oppose adding new chars to binary-formats,
although the proposed chars (t,n,m) do not make me very happy.
An alternative to this could be a switch to binary that will
change each format-character's meaning to (lower-case: host's
byteorder, Uppercase: reverse host's byteorder)
If we start adding new format-codes to binary, I'd also ask for
"unsigned" versions, for at least those integer-sizes where the
unsignedness can be seen afterwards. (e.g. the new code for "unsigned
char" applied to \377 should then yield 255, not -128)
> The floating-point types will be handled via:
I know too little about machine-float formats to comment on this.
--
Nichts ist schlimmer als zu wissen, wie das Unheil sich entwickelt,
und voll Ohnmacht zusehn muessen - das macht mich voellig krank...
-- (Musical Elisabeth; "Die Schatten werden laenger")
Arjen Markus
>
> Arjen Markus <arjen....@wldelft.nl> wrote:
> > * t (tiny) for short integers, using native ordering.
> > * n (normal) for ordinary integers, using native ordering.
> > * m (mirror of "w") for wide integers, using native ordering.
> Most binary formats have a defined byteorder: (often its the so called
> network-byteorder (aka big-endian). for formats mainly used on PC-
> platforms it tends to be little-endian independent of the current
> platform that I actually use it on.
>
Well, if you want to read a binary file created by, say, a C program
on the very same machine, then you have to know the byte order.
That is: the code should distinguish between i and I formats.
If, instead, you can use the native ordering (via a new "n" format),
then you do not need to distinguish this.
> If you're bound to parse a binary stream whose byteorder is not fixed
> but platform-dependent, it makes sense to make it explicitly
> platform-dependent, as in:
> if {$tcl_platform(byteOrder) eq "littleEndian"} {
> set binformatstr "siw..."
> } else {
> set binformatstr "SIW..."
> }
> Still, I wouldn't oppose adding new chars to binary-formats,
> although the proposed chars (t,n,m) do not make me very happy.
>
> An alternative to this could be a switch to binary that will
> change each format-character's meaning to (lower-case: host's
> byteorder, Uppercase: reverse host's byteorder)
>
That would break existing scripts!
Unless you make it optional:
- No option --> the old behaviour
- Turn the option on or off --> the new or the old behaviour.
I do not find that very Tclish though - it makes the behaviour
depend on an option outside the command.
> If we start adding new format-codes to binary, I'd also ask for
> "unsigned" versions, for at least those integer-sizes where the
> unsignedness can be seen afterwards. (e.g. the new code for "unsigned
> char" applied to \377 should then yield 255, not -128)
>
Right now, all format codes are strictly one character. I invented
the t/n/m ones because they have some (vague) mnemonic connotation
with their purpose. We would need a lot more! Or, introduce a prefix
like "u".
> > The floating-point types will be handled via:
> I know too little about machine-float formats to comment on this.
>
I have ample experience with them :) on various computer systems,
also ones that did not use the IEEE convention.
Thanks for your comments. I have given one or two thoughts to
the question of unsigned integer types, but that is not _my_
main concern. Still, with a good solution, they can be implemented
too.
Regards,
Arjen
Andreas Leitgeb
>> An alternative to this could be a switch to binary that will
>> change each format-character's meaning to (lower-case: host's
>> byteorder, Uppercase: reverse host's byteorder)
> - No option --> the old behaviour
> - Turn the option on or off --> the new or the old behaviour.
string cannot currently start with (or even contain) a dash ("-"),
we could compatibly add -switches just before the format string.
e.g.: binary scan $whatever -native "sS" ok ko
will scan four bytes from $whatever: two of them would be handled
as a native short int, the other two as a "anti-native" short int.
Yet another alternative:
we introduce (non alphanumeric) modifier-chars, that
modify (only) the next conversion specifier.
"+" for unsigned, "^" for native, "." may modify the next
conversion specifier to do something floating-point-related ...
to enable reuse of mnemonically rich characters :-)
Frank Pilhofer
>
> The current [binary] command can manipulate little-endian and
> big-endian _integer_ data, but only _native_ floating-point data. This
> means that binary data from other computer systems that use a different
> representation of floating-point data can not be directly handled.
>
Right. That's why the ability to read floating point values backwards
and forwards doesn't help much, if the encoding is still nonstandard.
I would like these new format codes to explicitly handle IEEE only. I
could prepare some C code (matter of fact, I could also supply Tcl code)
to encode/decode IEEE fp data, if you were interested.
>
> For integer types there are no codes to deal with native ordering. So:
>
> * t (tiny) for short integers, using native ordering.
> * n (normal) for ordinary integers, using native ordering.
> * m (mirror of "w") for wide integers, using native ordering.
>
Just wondering - do you have a use case for this?
Frank
--
Frank Pilhofer ........................................... f...@fpx.de
Live every day as if it were your last, because one of these days you
will be right! - Alfred E. Neuman
Arjen Markus
>
> Arjen Markus <arjen....@wldelft.nl> wrote:
> >
> > The current [binary] command can manipulate little-endian and
> > big-endian _integer_ data, but only _native_ floating-point data. This
> > means that binary data from other computer systems that use a different
> > representation of floating-point data can not be directly handled.
> >
>
> Right. That's why the ability to read floating point values backwards
> and forwards doesn't help much, if the encoding is still nonstandard.
> I would like these new format codes to explicitly handle IEEE only. I
> could prepare some C code (matter of fact, I could also supply Tcl code)
> to encode/decode IEEE fp data, if you were interested.
>
Hm, that would be very useful. I thought about this, but decided it
was not worth the trouble - as one would have to face a possibly unbound
set of cases. But I was kind of thinking the other way around - meaning
converting "any" floating-point data to something useful on the current
machine. Your solution is more limited than that, but covers mine for
most practical purposes.
What I would dearly like is the ability to deal with floating-point
data in a more flexible way.
> >
> > For integer types there are no codes to deal with native ordering. So:
> >
> > * t (tiny) for short integers, using native ordering.
> > * n (normal) for ordinary integers, using native ordering.
> > * m (mirror of "w") for wide integers, using native ordering.
> >
>
> Just wondering - do you have a use case for this?
>
>
I do for the 4-bytes case - files produced by computational programs
written in Fortran or C for instance. I threw in the others for reasons
of symmetry.
Arjen Markus
>
> Arjen Markus <arjen....@wldelft.nl> wrote:
> >> An alternative to this could be a switch to binary that will
> >> change each format-character's meaning to (lower-case: host's
> >> byteorder, Uppercase: reverse host's byteorder)
> > Unless you make it optional:
> > - No option --> the old behaviour
> of course!
> > - Turn the option on or off --> the new or the old behaviour.
> binary has not yet a proper "switch-hook", but since the format
> string cannot currently start with (or even contain) a dash ("-"),
> we could compatibly add -switches just before the format string.
>
> e.g.: binary scan $whatever -native "sS" ok ko
> will scan four bytes from $whatever: two of them would be handled
> as a native short int, the other two as a "anti-native" short int.
>
This would still mean knowing what is native in the case of
floating-point.
> Yet another alternative:
> we introduce (non alphanumeric) modifier-chars, that
> modify (only) the next conversion specifier.
> "+" for unsigned, "^" for native, "." may modify the next
> conversion specifier to do something floating-point-related ...
> to enable reuse of mnemonically rich characters :-)
>
I think it is a bit perverse to use a sign character to indicate no
sign :) but I do like this better than the first one, as it is localised
within the format string.
Regards,
Arjen
rene
scan] and creation of binary variables [binary format]. Still missing
is the ability to set a part of an already existing variable. To do
this task i propose a
[binary set variable formatString ?varName varName ...?] command.
The command performs the same operation as the [binary format] command
without creation of a new variable. Instead it uses the given
variable.
Rene
Arjen Markus
What exactly would the effect be? For instance:
binary variable A has the value "1 2 3 4" (of course in binary)
binary set A i 10 ==> 10 2 3 4 ?
But what if you want to change the second element? Use special format
codes?
- Mind you: I have thought about creating a binary string just to
allocate
memory that I can hand to some extension. (This can be done at the
script
level too).
Regards,
Arjen
rene
>
> binary variable A has the value "1 2 3 4" (of course in binary)
>
> binary set A i 10 ==> 10 2 3 4 ?
>
> But what if you want to change the second element? Use special format
> codes?
>
the variable. Writing values outside the size of the given variable
would result
in a error message.
Currently I have writen a special command to achieve these
functionality.
Write binary string over network sockets, use the values in "C" code
and
build the gui using Tk.
Regards,
Rene
Arjen Markus
:) I had forgotten about @ - that makes the implementation quite a bit
simpler than I had imagined (I think).
Well, there has been enough discussion to merit an update of my proposal
...
Regards,
Arjen