binary file parsing
Christopher
I am at a point in my binary file, where there is a 4 byte integral
value I need to read. I don't think I am guarenteed an int is 4 bytes
by C++, although it probably is. So, how do I read in 4 bytes and then
make it a numerical value?
Should I read char by char, put them together with some bitwise
operation magic, then try to convert it to an integer?
Or is there an easier way of telling the ifstream to extract exactly
four bytes into an actual int?
I am going to have the same problem in a few minutes with floats too.
joshua...@gmail.com
Google ios::binary.
Neelesh
The following code works for me (Uses reinterpret_cast):
#include <iostream>
#include <fstream>
int main()
{
std::ifstream f("a.txt", ios::in | ios::binary); //a.txt has an int
followed by a double (binary format)
char x[4];
char y[8];
f.read(x,4); //Read 4 bytes into x, this will read the int
f.read(y,8); //Read 8 bytes into y, this will read the float
int m = *(reinterpret_cast<int*>(x)); //reinterpret_cast will
simply reinterprete the bit patten
double n = *(reinterpret_cast<double*>(y)); //ditto
std::cout << m << std::endl; //prints the integer correctly
std::cout << n << std::endl; //prints the double correctly
}
Of couse since the mapping performed by reinterpret_cast is
implementation defined, I said that "this works for me", and I use
such a cast when I donot need to write portable code.
Christopher
> On May 3, 10:39 pm, Christopher <cp...@austin.rr.com> wrote:
Do you realize how many contradicting opinions come up in a google
search for something so basic, 95% of which are probably wrong?
Here is what I found in my google search:
1) make a template function that converts an array of bytes into an
integral type (neat but probably overkill)
2) use >> and extract to an int, don't worry about it (probably
wrong)
3) read byte by byte and bit shift into an integral result
(probably overly complicated and still depends on size)
4) let's use fstream.h and do it the deprecated way
(definatly wrong)
5) Well first we better check the indianess of the machine.... (I
can assume the same OS wrote it that is reading it in my case)
I bet most of those are either not necessary, overly complicated, or
just wrong.
I think a better source is:
http://www.parashift.com/c++-faq-lite/serialization.html#faq-36.6
However, it still fails to explain why we need to avoid the extraction
operator in favor of read and write, although that tid bit did answer
my question.
---------------------------
Here is my summation
1) There is no guarentee of the size of an integral type
2) The extraction operator, as I've always dealth with, is going to
extract a number of bytes from the stream that are equiv to the sizeof
the data type you are attempting to extract to, it will try to convert
it and if it fails, will set the fail bit
So, it wouldn't be safe to
int numBytes; // 4 bytes make this value
file >> numBytes;
Because the authoring tool that wrote the file wrote 4 bytes and
depends on you reading 4 bytes, if any implementation of an int comes
along that isn't 4 bytes, your parser is broken.
Regardless if it is incorrect or not, the fail bit gets set in my code
and I would like to understand why?
//---------------------------------------------------------------------------
void PolygonSetParser::ParseFile(const std::string & filepath,
ID3D10Device & device,
InputLayoutManager &
inputLayoutManager,
EffectManager & effectManager,
bool generateTangentData)
{
BaseException exception("Not Set",
"void PolygonSetParser::ParseFile(const
std::string & filepath)",
"PolygonSetParser.cpp");
// Open the file
std::ifstream file(filepath.c_str(), std::fstream::in |
std::fstream::binary);
// Check if the file was successfully opened
if( file.fail() )
{
exception.m_msg = std::string("Failed to open file: ") +
filepath;
throw exception;
}
// snip
// Read in a data identifier
unsigned char dataID;
file >> dataID;
while( file.good() )
{
switch( static_cast<int>(dataID) )
{
case MATERIAL_START:
{
ParseMaterial(file);
break;
}
default:
{
std::stringstream msg;
msg << "Unknown data identifier encountered in file: " <<
filepath << " at streampos " << file.tellg();
exception.m_msg = msg.str();
throw exception;
}
}
file >> dataID;
}
}
//---------------------------------------------------------------------------
void PolygonSetParser::ParseMaterial(std::ifstream & file)
{
BaseException exception("Not Set",
"void PolygonSetParser::ParseMaterial
(std::ifstream & file)",
"PolygonSetParser.cpp");
// Get how many bytes to parse
long numBytes; // 4 bytes make this value
file >> numBytes;
if( !file.good() ) // fails test
{
exception.m_msg = "Error parsing number of bytes to read for
material";
throw exception;
}
}
First few bytes of file in hex:
01 24 01 01 00 00 16 43 00 00 16 43 00 00 16 43
So, I am going to try it this way:
int numBytes;
fileread(static_cast<unsigned char *>(&numBytes), 4);
Which looks like it solves my problem of worrying about reading 4
bytes and the conversion.
I'd still like to know why the extraction operator fails though, if
anyone can explain.
Christopher
> So, I am going to try it this way:
> int numBytes;
> fileread(static_cast<unsigned char *>(&numBytes), 4);
Oops, spoke too soon. It looks like I do indeed have to
reinterpret_cast
// Read 4 byte unsigned integral value
unsigned numBytes;
fileread(reinterpret_cast<char *>(&numBytes), 4);
yucky.
James Kanze
> On May 3, 10:39 pm, Christopher <cp...@austin.rr.com> wrote:
> > Been awhile since I've done this.
> > I am at a point in my binary file, where there is a 4 byte
> > integral value I need to read. I don't think I am guarenteed
> > an int is 4 bytes by C++, although it probably is.
Not at all. I'm aware of implementations where int's are 16, 36
or 48 bits. 16 bit is, or at least used to be, quite common
(and the machines using 36 and 48 bits are still being sold).
> > So, how do I read in 4 bytes and then make it a numerical
> > value?
> > Should I read char by char, put them together with some
> > bitwise operation magic, then try to convert it to an
> > integer?
That's about it. I'm not aware of any other solution that
actually works, except in extremely limited cases.
> > Or is there an easier way of telling the ifstream to extract
> > exactly four bytes into an actual int?
No. The iostream abstraction is formatted text, and it only
supports formatting to and from text. If you have a binary
format, then you have to write the formatting yourself (or find
it somewhere on the net).
> > I am going to have the same problem in a few minutes with
> > floats too.
That can be a much larger problem, since the differences in
format can be much greater.
In practice, you need to do two things: define the format you
will be reading, and decide your portability requirements. If
you need to handle float, and need to be portable to just about
everything, the code is far from trivial. If the format you're
reading used IEEE floats, however (often the case), and you
don't have to worry about machines which use other floating
point formats (mainly---perhaps only---mainframes), then it is a
lot easier.
> Google ios::binary.
Which is totally irrelevant here. (The file must be opened in
binary mode, but that won't change the rules iostream uses for
formatting.)
--
James Kanze (GABI Software) email:james...@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
Pascal J. Bourguignon
I don't know.
If I were you, I'd check very closely what the standard says about
padding, and how it would be processed by reinterpret_cast.
Then I would check closely how well the implementations match the
specifications.
Now, the most important point is to know the format you have in your
file. How are the integers stored? Byte size, bytesex, bit order,
how signed numbers are encoded (two-complement, one-complement,
sign+magnitude, BCD, etc)? Same for your floating point numbers.
Beware that bytesex still matters for IEEE754.
Then the safest bet is to read the file byte by byte, and to build the
values from the bytes. I'd rather use arithmetic operations since
this would avoid the need for reinterpreting bits.
For example, for a 2-complement, 32-bit integer stored as four 8-bit
bytes in big-endian order, I would do:
#include <limits.h>
#if CHAR_BITS<8
#error "Won't be able to read the bytes"
#endif
#if (CHAR_BITS*sizeof(int))<32
#if (CHAR_BITS*sizeof(long int))<32
#error "int too small (would need to use a bignum library"
#else
typedef long int integer;
typedef unsigned long int cardinal;
#endif
#else
typedef int integer;
typedef unsigned int cardinal;
#endif
typedef unsigned char byte;
cardinal read_cardinal(stream){
cardinal u=0;
for(int i=0;i<4;i++){
byte b=read_byte(stream);
u=(u*256)+b; /* big endian */
}
return(u);
}
integer read_integer(stream){
cardinal u=read_cardinal(stream);
integer i;
if(u<=2147483647UL){ /* 2-complement */
i=u;
}else{
i=((integer)(u-2147483648UL))-2147483648L;
}
return(i);
}
--
__Pascal Bourguignon__
James Kanze
> On May 4, 1:21 am, joshuamaur...@gmail.com wrote:
> > On May 3, 10:39 pm, Christopher <cp...@austin.rr.com> wrote:
> > Google ios::binary.
> Do you realize how many contradicting opinions come up in a
> google search for something so basic, 95% of which are
> probably wrong?
:-)
> Here is what I found in my google search:
> 1) make a template function that converts an array of bytes
> into an integral type (neat but probably overkill)
I'm not sure I see the use of a template here, either. You
generally have at most two or three cases to deal with (2 bytes,
4 bytes, and once in a while, 8 bytes). And the meta
programmation necessary for the templates is probably more work
and more lines of code than just writing the three functions.
> 2) use >> and extract to an int, don't worry about it (probably
> wrong)
If the format is binary, certainly wrong.
> 3) read byte by byte and bit shift into an integral result
> (probably overly complicated and still depends on size)
It's the only solution I know. And it's not that complicated.
> 4) let's use fstream.h and do it the deprecated way
> (definatly wrong)
And doesn't work any better than with <fstream>.
> 5) Well first we better check the indianess of the machine....
> (I can assume the same OS wrote it that is reading it in my
> case)
Endianness doesn't really depend on the OS. I've seen it change
from one version of the compiler to the next. And of course,
you really can't assume that your users are never going to
upgrade the material, which depending on the evolution path,
could mean a lot of things. It's simpler just to do it right to
begin with.
> I bet most of those are either not necessary, overly
> complicated, or just wrong.
> I think a better source is:http://www.parashift.com/c++-faq-lite/serialization.html#faq-36.6
> However, it still fails to explain why we need to avoid the
> extraction operator in favor of read and write, although that
> tid bit did answer my question.
The extraction operator formats/unformats. For a specific
format. (You can control it somewhat via flags, but it always
handles text.) For binary input, *if* you use istream or
ostream (I wouldn't, for a file that is completely binary), you
want to bypass the iostream formatting, using unformatted input
and output (read/get and write/put), and do your own formatting.
> ---------------------------
> Here is my summation
> 1) There is no guarentee of the size of an integral type
> 2) The extraction operator, as I've always dealth with, is
> going to extract a number of bytes from the stream that are
> equiv to the sizeof the data type you are attempting to
> extract to, it will try to convert it and if it fails, will
> set the fail bit
No. The extraction operator will read bytes, interpret them as
characters, and convert the resulting string into the type you
want. If the format in the source file is not text based, they
simply won't work.
> So, it wouldn't be safe to
> int numBytes; // 4 bytes make this value
> file >> numBytes;
> Because the authoring tool that wrote the file wrote 4 bytes
That is *NOT* enough information. Four bytes doesn't mean
anything. You still have to know the format used.
> and depends on you reading 4 bytes, if any implementation of
> an int comes along that isn't 4 bytes, your parser is broken.
In general, if you try to read in a different format than that
was written, it's not going to work.
> Regardless if it is incorrect or not, the fail bit gets set in
> my code and I would like to understand why?
Probably, the extractor didn't find what looked like an int.
The extractor skips whitespace, then looks for an optional sign,
followed by one or more digits. In whatever encoding is imbued
into the stream (typically either ISO 8859-1 or UTF-8 in the
environments I work in, but YMMV).
> //---------------------------------------------------------------------------
> void PolygonSetParser::ParseFile(const std::string & filepath,
> ID3D10Device & device,
> InputLayoutManager &
> inputLayoutManager,
> EffectManager & effectManager,
> bool generateTangentData)
> {
> BaseException exception("Not Set",
> "void PolygonSetParser::ParseFile(const
> std::string & filepath)",
> "PolygonSetParser.cpp");
> // Open the file
> std::ifstream file(filepath.c_str(), std::fstream::in |
> std::fstream::binary);
> // Check if the file was successfully opened
> if( file.fail() )
> {
> exception.m_msg = std::string("Failed to open file: ") +
> filepath;
> throw exception;
> }
You also want to imbue the "C" locale, to ensure that no code
translation occurs. (This is the sort of thing that works in
your test programs, because the "C" locale is the default, and
your test programs don't need to change it, but fails in actual
code, because the application has switched to some other
locale.)
> // snip
> // Read in a data identifier
> unsigned char dataID;
> file >> dataID;
Note that I'd use get for this. And pass through an
intermediate int:
int dataId = file.get() ;
if ( dataId == EOF ) {
// bad format...
}
> while( file.good() )
No. ios::good() is generally useless.
If you're reading into an int, as above, you can use:
while ( dataID != EOF )
Otherwise (and more generally):
while ( file )
This is one of the reasons I wouldn't use istream, but would
implement my own ibinstream, or whatever. The ibinstream (which
would still derive from ios) would then define the extraction
operator to handle the format you're reading, setting the
various status bits as appropriate.
> {
> switch( static_cast<int>(dataID) )
> {
> case MATERIAL_START:
> {
> ParseMaterial(file);
> break;
> }
> default:
> {
> std::stringstream msg;
> msg << "Unknown data identifier encountered in file: " <<
> filepath << " at streampos " << file.tellg();
>
> exception.m_msg = msg.str();
> throw exception;
> }
> }
> file >> dataID;
> }
> }
> //---------------------------------------------------------------------------
> void PolygonSetParser::ParseMaterial(std::ifstream & file)
> {
> BaseException exception("Not Set",
> "void PolygonSetParser::ParseMaterial
> (std::ifstream & file)",
> "PolygonSetParser.cpp");
> // Get how many bytes to parse
> long numBytes; // 4 bytes make this value
Not on my machines. Long is usually 64 bits today (although 32,
36 and 48 bits are not unknown, and 32 bits was common in the
past).
Note that this is one of the reasons why you must pass through
explicit serialization. Even if long is 32 bits on your machine
today, it's likely that if your user upgrades, it will be 64
bits. Whereas the format of the data file will not change.
> file >> numBytes;
This reads ASCII.
> if( !file.good() ) // fails test
And again, this may fail even if the input succeeded. Just use:
if ( ! file )
> {
> exception.m_msg = "Error parsing number of bytes to read for
> material";
> throw exception;
> }
> }
> First few bytes of file in hex:
> 01 24 01 01 00 00 16 43 00 00 16 43 00 00 16 43
> So, I am going to try it this way:
> int numBytes;
> fileread(static_cast<unsigned char *>(&numBytes), 4);
> Which looks like it solves my problem of worrying about
> reading 4 bytes and the conversion.
Except that it doesn't. It may seem to work in a particular
case, but it doesn't provide a general solution, and may fail
the next time you recompile your code.
> I'd still like to know why the extraction operator fails
> though, if anyone can explain.
A better question is why you would expect it to work.
James Kanze
The reinterpret_cast is necessary because you're doint something
that won't normally work.
James Kanze
> #include <iostream>
> #include <fstream>
Only because you've been lucky (or unlucky). It will generally
work if the exact same executable is doing both the reading and
the writing. As soon as you recompile one or the other, it may
fail.
Christopher
wrote:
I am trying to expand on that. I am having trouble though and got
stuck.
I want to make it a template function, because the program writing
this binary file frequently writes 1,2, or 4 byte integral values,
however a client of my library is not going to necessarily be storing
these values in the same number of bytes. Often a 1 byte value maybe
be stored in a 4 byte type, or perhaps even more than 4 bytes
considering I am doing this in 64bit....
So here is what I have so far:
template typename<IntegralType>
IntegralType IntegralTypeFromBytes(const char * bytes, const unsigned
numBytes, const bool isSigned)
{
if( CHAR_BIT != 8 || sizeof(char) != 1 )
{
// Error
}
if( sizeof(IntegralType) < numBytes )
{
// Error
}
if( !isSigned )
{
IntegralType u = 0;
// Little Endian
for(unsigned i = numBytes; i > 0; --i)
{
char b = bytes[i];
u = u * 256 + b;
}
}
}
else
{
}
}
I don't know what to do in the signed case. Can you explain your
example a little more?
Does my code so far look OK?
Pascal J. Bourguignon
> [...]
> I am trying to expand on that. I am having trouble though and got
> stuck.
>
> I want to make it a template function, because the program writing
> this binary file frequently writes 1,2, or 4 byte integral values,
> however a client of my library is not going to necessarily be storing
> these values in the same number of bytes. Often a 1 byte value maybe
> be stored in a 4 byte type, or perhaps even more than 4 bytes
> considering I am doing this in 64bit....
I would keep separate the physical file format from the application
level types.
There's little point in writting templates to handle the different
integer size you may have in the file, after all, there's only three
or four sizes.
On the other hand, when you convert to or from application types to
file format types, you want to check boundaries, so you may use
boost::lexical_cast<> which will throw an exception if the value is
out of the range of the target type.
> So here is what I have so far:
>
> template typename<IntegralType>
> IntegralType IntegralTypeFromBytes(const char * bytes, const unsigned
> numBytes, const bool isSigned)
> {
> if( CHAR_BIT != 8 || sizeof(char) != 1 )
sizeof(char) is always 1, by definition. If it wasn't 1, it would
mean the compiler is broken.
> {
> // Error
> }
>
> if( sizeof(IntegralType) < numBytes )
> {
> // Error
> }
>
> if( !isSigned )
> {
> IntegralType u = 0;
>
> // Little Endian
> for(unsigned i = numBytes; i > 0; --i)
> {
> char b = bytes[i];
> u = u * 256 + b;
> }
> }
>
> }
> else
> {
> }
> }
>
> I don't know what to do in the signed case. Can you explain your
> example a little more?
The principle of two-complement representation is to offset the
negative values above the positive ones, where the most significant
bit of the word is 1.
For example, if the word is 8 bit wide, then we can code 256 (2^8)
different values. Unsigned will go from 0 to 255. We split the range
in two, keeping positive codes from 0 to 127 to encode the positive
values from 0 to 127, and the remaining codes, from 127 to 255 are
offsetted by -256 to cover the range of values from -128 to -1.
0 ... 127, 128 ... 255
00000000 ... 01111111, 10000000 ... 11111111
0 ... 127, -128 ... -1
-128 = 128 - 256
-1 = 255 - 256
So when you have a code in two-complement on w bits, to convert it
into a signed value, you check whether the code is between 0 and
2^(w-1) exclusive, in which case the value is equal to the code, and
if the code is greater or equal to 2^(w-1), then the value is the code
minus 2^w.
> Does my code so far look OK?
I would do something like the following (not tested, not even compiled):
typedef unsigned char byte;
template <class NativeUnsignedInt,int ByteSize,int FileUnsignedIntBytes>
NativeUnsignedInt read_big_endian_cardinal(FILE* stream){
assert(ByteSize<=CHAR_BITS);
assert(ByteSize*FileUnsignedIntBytes<=sizeof(NativeUnsignedInt));
NativeUnsignedInt u=0;
for(int i=0;i<FileUnsignedIntBytes;i++){
byte b=read_byte(stream);
u=(u*256)+b; /* big endian */
}
return(u);
}
template <class NativeUnsignedInt,int ByteSize,int FileUnsignedIntBytes>
NativeUnsignedInt read_little_endian_cardinal(FILE* stream){
assert(ByteSize<=CHAR_BITS);
assert(ByteSize*FileUnsignedIntBytes<=CHAR_BITS*sizeof(NativeUnsignedInt));
NativeUnsignedInt u=0;
NativeUnsignedInt w=1<<(ByteSize*(FileUnsignedIntBytes-1));
for(int i=0;i<FileUnsignedIntBytes;i++){
byte b=read_byte(stream);
u+=(b*w); /* little endian */
w>>=ByteSize;
}
return(u);
}
#include <limits.h>
template<char> unsigned long long maximumOf(){ return(CHAR_MAX); }
template<unsigned char> unsigned long long maximumOf(){ return(UCHAR_MAX); }
template<short> unsigned long long maximumOf(){ return(SHRT_MAX); }
template<unsigned short> unsigned long long maximumOf(){ return(USHRT_MAX); }
template<int> unsigned long long maximumOf(){ return(INT_MAX); }
template<unsigned int> unsigned long long maximumOf(){ return(UINT_MAX); }
template<long int> unsigned long long maximumOf(){ return(LONG_MAX); }
template<unsigned long int> unsigned long long maximumOf(){ return(ULONG_MAX); }
template<long long int> unsigned long long maximumOf(){ return(LLONG_MAX); }
template<unsigned long long int> unsigned long long maximumOf(){ return(ULLONG_MAX); }
template<char> long long minimumOf(){ return(CHAR_MIN); }
template<unsigned char> long long minimumOf(){ return(0); }
template<short> long long minimumOf(){ return(SHRT_MIN); }
template<unsigned short> long long minimumOf(){ return(0); }
template<int> long long minimumOf(){ return(INT_MIN); }
template<unsigned int> long long minimumOf(){ return(0); }
template<long int> long long minimumOf(){ return(LONG_MIN); }
template<unsigned long int> long long minimumOf(){ return(0); }
template<long long int> long long minimumOf(){ return(LLONG_MIN); }
template<unsigned long long int> long long minimumOf(){ return(0); }
template <class NativeSignedInt,class NativeUnsignedInt,int ByteSize,int FileUnsignedIntBytes>
NativeSignedInt two_complement(NativeUnsignedInt u){
/* 2-complement */
NativeSignedInt i;
if(u<=(1ULL<<(ByteSize*FileUnsignedIntBytes-1)-1)){
assert(u<=maximumOf<NativeSignedInt>());
i=u;
}else{
assert((1UL<<(ByteSize*FileUnsignedIntBytes-1))+minimumOf<NativeSignedInt>()
<=(u-(1UL<<(ByteSize*FileUnsignedIntBytes-1))));
i=((NativeSignedInt)(u-(1ULL<<(ByteSize*FileUnsignedIntBytes-1))))-(1LL<<(ByteSize*FileUnsignedIntBytes-1));
}
return(i);
}
template <class NativeSignedInt,class NativeUnsignedInt,int ByteSize,int FileUnsignedIntBytes>
NativeSignedInt read_big_endian_integer(FILE* stream){
return(two_complement<NativeUnsignedInt,ByteSize,FileUnsignedIntBytes>(
read_big_endian_cardinal<NativeUnsignedInt,ByteSize,FileUnsignedIntBytes>(stream)));
}
template <class NativeSignedInt,class NativeUnsignedInt,int ByteSize,int FileUnsignedIntBytes>
NativeSignedInt read_little_endian_integer(FILE* stream){
return(two_complement<NativeUnsignedInt,ByteSize,FileUnsignedIntBytes>(
read_little_endian_cardinal<NativeUnsignedInt,ByteSize,FileUnsignedIntBytes>(stream)));
}
Here, the asserts in two_complement check for the native type range
when going from unsigned to signed. You could call:
short s=read_little_endian_integer<short,unsigned long long,8,8>(stream);
if you had a short stored over a 64-bit two-complement in little-endian order coding.
--
__Pascal Bourguignon__
Pete Becker
>
> I want to make it a template function, because the program writing
> this binary file frequently writes 1,2, or 4 byte integral values,
> however a client of my library is not going to necessarily be storing
> these values in the same number of bytes. Often a 1 byte value maybe
> be stored in a 4 byte type, or perhaps even more than 4 bytes
> considering I am doing this in 64bit....
>
A template seems like overkill. Use an unsigned long or unsigned long
long to accumulate the values from the bytes and return the result. Yes,
stupid compilers might warn you that the value is being truncated if you
store in into a smaller integer type, but you're not writing code to
please some compiler writer's notion of good coding style.
--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of
"The Standard C++ Library Extensions: a Tutorial and Reference"
(www.petebecker.com/tr1book)
James Kanze
wrote:
> Christopher <cp...@austin.rr.com> writes:
[...]
> I'd rather use arithmetic operations since
> this would avoid the need for reinterpreting bits.
I fully agree with the rest of what you wrote, but I'm curious
about this. If the format is specified in terms of bits
(usually the case), it would seem to me that the bit operations
are more appropriate, in the sense that they are closer to the
specification. IOW, for a 32 bit unsigned integer: if the
format specification says that the first octet contains the
value divided by 16777216, the second the value divided by
65536, modulo 256, etc., then I'd use arithmetic operators. If
it says that the first octet contains the bits 24-31, the second
the bits 16-23, etc., I'd probably use bit operations (shifting
and masking).
Of course, the distinction isn't always that clear. For a
floating point value, the format probably specifies where the
sign, exponant and mantissa fields are located in terms of bits,
but the semantic value of each field will be specified
mathematically. So I'd naturally use bitwise operations to
extract the fields from the input, then mathematical functions
(e.g. ldexp) to merge them into the final value. Something like
(assume that I've set up source to handle errors like unexpected
end of file correctly):
unsigned char byte = source.get() ;
bool isNegative = (byte & 0x80) != 0 ;
int exponent = (byte & 0x7F) << 1 ;
int mantissa = 0 ;
byte = source.get() ;
exponent |= byte >> 7 ;
mantissa = (byte & 0x7F) << 16 ;
mantissa |= source.get() << 8 ;
mantissa |= source.get() ;
float result = 0.0F ;
if ( exponent != 0 ) {
result = ldexp( mantissa | 0x00800000,
exponent -126 - 24 ) ;
} else {
result = ldexp( mantissa, -126 - 24 ) ;
}
if ( isNegative ) {
result = -result ;
}
(This supposes input conform to XDR, and should work on any
machine where the results don't overflow. It doesn't support
NaN or Infinity, however---that would require extra handling.)
> For example, for a 2-complement, 32-bit integer stored as four 8-bit
> bytes in big-endian order, I would do:
> #include <limits.h>
> #if CHAR_BITS<8
> #error "Won't be able to read the bytes"
#error "Isn't a legal C/C++ implementation"
> #endif
> #if (CHAR_BITS*sizeof(int))<32
> #if (CHAR_BITS*sizeof(long int))<32
> #error "int too small (would need to use a bignum library"
> #else
> typedef long int integer;
> typedef unsigned long int cardinal;
> #endif
> #else
> typedef int integer;
> typedef unsigned int cardinal;
> #endif
> typedef unsigned char byte;
I'm lazy; for the moment, I just include <stdint.h>, and punt if
uint32_t and int32_t aren't defined. Otherwise, if <stdint.h>
(or <cstdint> in C++0x) is available, uint_fast_32_t and
int_fast32_t Should do the job. (I don't like the name
"cardinal", because C++'s unsigned doesn't really behave like a
"cardinal", which should be a sub-range of "integer", and behave
rationally when compared with "integer".)
> cardinal read_cardinal(stream){
> cardinal u=0;
> for(int i=0;i<4;i++){
> byte b=read_byte(stream);
> u=(u*256)+b; /* big endian */
> }
> return(u);
> }
> integer read_integer(stream){
> cardinal u=read_cardinal(stream);
> integer i;
> if(u<=2147483647UL){ /* 2-complement */
> i=u;
> }else{
> i=((integer)(u-2147483648UL))-2147483648L;
> }
> return(i);
> }
In the end, the real question is how portable do you want (or
need) to be. For a lot of people, it's probably acceptable to
suppose that 1) there is a 32 bit integral type, using 2's
complement, and 2) that conversion unsigned to signed int
doesn't change the bit pattern. In such cases, you can make
reading an integer a lot simpler. Similarly, if in addition you
can suppose IEEE floating point, my floating point read can be
made a lot, lot simpler---just put the four bytes in an array,
and memcpy into the double. (Currently, there are only a few
exotic machines which don't have a 32 bit 2's complement
integral type. None of the mainframes I know use IEEE floating
point, however.) I would stress, however, that if you make such
simplifying assumptions, you document them clearly and
explicitly.
James Kanze
> Christopher wrote:
> > I want to make it a template function, because the program
> > writing this binary file frequently writes 1,2, or 4 byte
> > integral values, however a client of my library is not going
> > to necessarily be storing these values in the same number of
> > bytes. Often a 1 byte value maybe be stored in a 4 byte
> > type, or perhaps even more than 4 bytes considering I am
> > doing this in 64bit....
> A template seems like overkill. Use an unsigned long or
> unsigned long long to accumulate the values from the bytes and
> return the result. Yes, stupid compilers might warn you that
> the value is being truncated if you store in into a smaller
> integer type, but you're not writing code to please some
> compiler writer's notion of good coding style.
He still needs different versions for different lengths in the
source file. He could use a loop for that, but it's probably
not much more difficult to maintain the 3 or 4 functions
separately, each returning the correct type. (This also makes
handling the sign a little bit easier.)
As for range checking, this really depends on higher level code
anyway. If the protocol is encoding e.g. Sudoko boards, it's
likely that the legal range for some integral values in the file
will be 1-9 (or 0-9, if you use 0 for an open square). You
really don't want to encode this sort of information in the
routine reading an int (or an unsigned char, or whatever).
Pascal J. Bourguignon
> On May 4, 11:16 am, p...@informatimago.com (Pascal J. Bourguignon)
> wrote:
>> Christopher <cp...@austin.rr.com> writes:
>
> [...]
>> I'd rather use arithmetic operations since
>> this would avoid the need for reinterpreting bits.
>
> I fully agree with the rest of what you wrote, but I'm curious
> about this. If the format is specified in terms of bits
> (usually the case), it would seem to me that the bit operations
> are more appropriate, in the sense that they are closer to the
> specification. IOW, for a 32 bit unsigned integer: if the
> format specification says that the first octet contains the
> value divided by 16777216, the second the value divided by
> 65536, modulo 256, etc., then I'd use arithmetic operators. If
> it says that the first octet contains the bits 24-31, the second
> the bits 16-23, etc., I'd probably use bit operations (shifting
> and masking).
For integers, I assume it wouldn't make a lot of difference. The
compiler probably generate the same code for n<<8 and for n*256
anyways. You'd just have to be careful not to leave uninitialized
bits in the case where the word size of the host is bigger than that
of the file (eg. reading a 32-bit integer on a 36-bit host),
particularly for signed integers where you may need to do a sign
extend. That's where using the substration to convert a
two-complement seems to me to be more interesting than just using bit-or.
Absolutely. The file format will be specified at the bit or byte
level and will have to be processed with bit operations. But to build
the host value, integer or floating-point, it's safer to use
arithmetic operations such as ldexp() and negation.
>> For example, for a 2-complement, 32-bit integer stored as four 8-bit
>> bytes in big-endian order, I would do:
>
>> #include <limits.h>
>> #if CHAR_BITS<8
>> #error "Won't be able to read the bytes"
>
> #error "Isn't a legal C/C++ implementation"
Are you sure? AFAIK, char may be 6 bits, there are trigraphs to deal
with these hosts. But I haven't read the recent C standards.
> In the end, the real question is how portable do you want (or
> need) to be. For a lot of people, it's probably acceptable to
> suppose that 1) there is a 32 bit integral type, using 2's
> complement, and 2) that conversion unsigned to signed int
> doesn't change the bit pattern. In such cases, you can make
> reading an integer a lot simpler. Similarly, if in addition you
> can suppose IEEE floating point, my floating point read can be
> made a lot, lot simpler---just put the four bytes in an array,
> and memcpy into the double. (Currently, there are only a few
> exotic machines which don't have a 32 bit 2's complement
> integral type. None of the mainframes I know use IEEE floating
> point, however.) I would stress, however, that if you make such
> simplifying assumptions, you document them clearly and
> explicitly.
Of course, this is a question of specification.
If all is called is to be able to save and load binary data on the
same computer, we could just memory map the file.
However, I see one problem in relying on specifications: they are not
formally defined, and cannot automatically be checked (in general, I'm
yet to see a real project using formal specification tools, seems I'm
not in the right industry for this kind of tools :-( ). So the
problem is that you will clearly document that your lateral
accelerometre outputs 16-bit values, you will document it and write it
all over. But when another team will reuse your module, and embed it
in hardware able to support lateral accelerations that need to be
expressed in more than 16 bits, no compiler will be there to check the
specification mismatch and you'll lose costly hardware and perhaps
lives. Ok, you've specified and documented the limits of your module,
so nobody can reproach you anything, nonetheless, this didn't prevent
a catastrophe.
So perhaps, that's my point of view, if you didn't expect a restricted
context, you could write your code to be able to handle the unexpected
environment and avoid such problems, or at least, have the mismatch be
detected at compilation time.
--
__Pascal Bourguignon__
joshua...@gmail.com
wrote:
> James Kanze <james.ka...@gmail.com> writes:
> > On May 4, 11:16 am, p...@informatimago.com (Pascal J. Bourguignon)
> > wrote:
> >> For example, for a 2-complement, 32-bit integer stored as four 8-bit
> >> bytes in big-endian order, I would do:
>
> >> #include <limits.h>
> >> #if CHAR_BITS<8
> >> #error "Won't be able to read the bytes"
>
> > #error "Isn't a legal C/C++ implementation"
>
> Are you sure? AFAIK, char may be 6 bits, there are trigraphs to deal
> with these hosts. But I haven't read the recent C standards.
It's technically nonconforming, at least for C++. I don't have the C
standard handy, but C++03 specifically says that CHAR_BITS must be 8
or larger.
Alf P. Steinbach
Assuming you meant CHAR_BIT, where did you find that requirement in C++0x?
The C++ standard has an implicit size requirement on the bit size of 'char' via
�3.9.1/1 which requires a char to be able to hold any character in the
implementation's basic character set, which is a bit unclear but anyways AFAIUI
must refer to one of the character sets denoted "basic" in �2.2, any any of
those have at least 96 characters, which implies at least 7 bits. However, the
C++ standard refers to the C standard. And the C standard requires that CHAR_BIT
is mimimum 8, in �5.2.4.2.1/1.
So, summing up, as far as I can see it's CHAR_BIT, not CHAR_BITS, and it's the C
standard, not the C++ standard.
Cheers,
- Alf
--
Due to hosting requirements I need visits to <url: http://alfps.izfree.com/>.
No ads, and there is some C++ stuff! :-) Just going there is good. Linking
to it is even better! Thanks in advance!
James Kanze
wrote:
> James Kanze <james.ka...@gmail.com> writes:
> > On May 4, 11:16 am, p...@informatimago.com (Pascal J. Bourguignon)
> > wrote:
> >> Christopher <cp...@austin.rr.com> writes:
> > [...]
> >> I'd rather use arithmetic operations since
> >> this would avoid the need for reinterpreting bits.
> > I fully agree with the rest of what you wrote, but I'm
> > curious about this. If the format is specified in terms
> > of bits (usually the case), it would seem to me that the
> > bit operations are more appropriate, in the sense that
> > they are closer to the specification. IOW, for a 32 bit
> > unsigned integer: if the format specification says that
> > the first octet contains the value divided by 16777216,
> > the second the value divided by 65536, modulo 256, etc.,
> > then I'd use arithmetic operators. If it says that the
> > first octet contains the bits 24-31, the second the bits
> > 16-23, etc., I'd probably use bit operations (shifting
> > and masking).
> For integers, I assume it wouldn't make a lot of
> difference.
For anything: if the specification says "this byte
corresponds to bits 8-15", I find shifting left 8 more
intuitive (closer to what is written in the specification)
than multiplying by 256.
> The compiler probably generate the same code
> for n<<8 and for n*256 anyways. You'd just have to be
> careful not to leave uninitialized bits in the case where
> the word size of the host is bigger than that of the file
> (eg. reading a 32-bit integer on a 36-bit host),
> particularly for signed integers where you may need to do
> a sign extend. That's where using the substration to
> convert a two-complement seems to me to be more
> interesting than just using bit-or.
That's a different issue. You use shifting, etc. to extract
the sign bit and the value bits, because that's how their
location in the input data is specified: by the bit position
of the data. You use arithmetic operations to create the
final value from the extracted fields, because you're now
dealing with mathematical identities.
The difference is obviously much more evident in floating
point: you're not going to be able to extract individual
fields using arithmetic operators on floating point, and
anything you do using bitwise operators to assemble the
fields will be very implementation dependent---for that, you
want the mathematical functions and operators.
But I see that later, after my example:
[...]
> Absolutely. The file format will be specified at the bit
> or byte level and will have to be processed with bit
> operations. But to build the host value, integer or
> floating-point, it's safer to use arithmetic operations
> such as ldexp() and negation.
You actually agree with me. Bitwise for extracting (and
inserting when writing) the individual fields, arithmetic
for manipulating the values. (It seems reasonable to
consider each field an unsigned int of a specific size.)
> >> For example, for a 2-complement, 32-bit integer stored
> >> as four 8-bit bytes in big-endian order, I would do:
> >> #include <limits.h>
> >> #if CHAR_BITS<8
> >> #error "Won't be able to read the bytes"
> > #error "Isn't a legal C/C++ implementation"
> Are you sure? AFAIK, char may be 6 bits, there are
> trigraphs to deal with these hosts. But I haven't read
> the recent C standards.
C90 required that UCHAR_MAX be at least 255, and that the
representation be pure binary. For an implementation to be
conform with less than 8 bits, it whould have to be able to
fit 255 in less than 8 bits, using a binary representation.
Which is impossible.
Admitted, it's a pretty indirect way to specify the minimum
size, but it's what C90 used, and it hasn't changed in C++
or C99.
> > In the end, the real question is how portable do you
> > want (or need) to be. For a lot of people, it's
> > probably acceptable to suppose that 1) there is a 32 bit
> > integral type, using 2's complement, and 2) that
> > conversion unsigned to signed int doesn't change the bit
> > pattern. In such cases, you can make reading an integer
> > a lot simpler. Similarly, if in addition you can
> > suppose IEEE floating point, my floating point read can
> > be made a lot, lot simpler---just put the four bytes in
> > an array, and memcpy into the double. (Currently, there
> > are only a few exotic machines which don't have a 32 bit
> > 2's complement integral type. None of the mainframes I
> > know use IEEE floating point, however.) I would stress,
> > however, that if you make such simplifying assumptions,
> > you document them clearly and explicitly.
> Of course, this is a question of specification.
> If all is called is to be able to save and load binary
> data on the same computer, we could just memory map the
> file.
Not necessarily. As I said, I've actually experienced the
case where the byte order of a long changed from one version
of the compiler to the next, and on the machine I currently
use, the size of a long depends on a compiler option (and
there are two different system API's, depending on which
compiler option was used).
> However, I see one problem in relying on specifications:
> they are not formally defined, and cannot automatically be
> checked (in general, I'm yet to see a real project using
> formal specification tools, seems I'm not in the right
> industry for this kind of tools :-( ). So the problem is
> that you will clearly document that your lateral
> accelerometre outputs 16-bit values, you will document it
> and write it all over. But when another team will reuse
> your module, and embed it in hardware able to support
> lateral accelerations that need to be expressed in more
> than 16 bits, no compiler will be there to check the
> specification mismatch and you'll lose costly hardware and
> perhaps lives. Ok, you've specified and documented the
> limits of your module, so nobody can reproach you
> anything, nonetheless, this didn't prevent a catastrophe.
I see you've read the details of the Ariane 5 explosion as
well:-). In practice, this is a real problem. The boss
says that the code will never have to be ported to another
machine, and the week following its delivery, he installs a
new machine, with different characteristics. I've found
that it is often worthwhile being "portable", even if it
isn't part of the specifications. How portable depends on
the extra effort required, but I'd certainly attempt to
handle at least byte order and the size of the basic
integral types. Beyond that, a lot depends---the type of
projects I usually work on might conceivably be run on a
mainframe, so I take mainframe architectures into
consideration (none of the mainframes I know use IEEE, and
some have 36 or 48 bit ints, padding bits in int, and 1's
complement or signed magnitude), but if I were writing, say,
a GUI interface, I probably wouldn't bother: I'd suppose
IEEE floating point, for example, because all current
workstations and PC's use it, and I can't imagine that
changing.
Jorgen Grahn
...
> In practice, you need to do two things: define the format you
> will be reading, and decide your portability requirements.
Yes. I hate maintaining code where the external data format is defined
by "whatever the first implementation ended up generating, on the
first machine it happened to run on."
> If
> you need to handle float, and need to be portable to just about
> everything, the code is far from trivial. If the format you're
> reading used IEEE floats, however (often the case), and you
> don't have to worry about machines which use other floating
> point formats (mainly---perhaps only---mainframes), then it is a
> lot easier.
It seems to me that XDR would be useful here -- it defines storage of
floating-point types. Or better, don't insist on an unreadable data
format; use plain text instead.
/Jorgen
--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.se> R'lyeh wgah'nagl fhtagn!
James Kanze
> On Mon, 4 May 2009 01:35:40 -0700 (PDT), James Kanze
> <james.ka...@gmail.com> wrote:
> ...
> > In practice, you need to do two things: define the format you
> > will be reading, and decide your portability requirements.
> Yes. I hate maintaining code where the external data format is
> defined by "whatever the first implementation ended up
> generating, on the first machine it happened to run on."
Especially when they've been around for a while. Some of those
older machines had some really wierd formats. (Byte order of a
long 2301, for example.)
> > If you need to handle float, and need to be portable to just
> > about everything, the code is far from trivial. If the
> > format you're reading used IEEE floats, however (often the
> > case), and you don't have to worry about machines which use
> > other floating point formats (mainly---perhaps
> > only---mainframes), then it is a lot easier.
> It seems to me that XDR would be useful here -- it defines
> storage of floating-point types.
It defines it to be IEEE. If your portability needs are
limited to machines with IEEE (PC's and most mid-sized Unix,
but not mainframes), then using IEEE in the external format is
particularly easy: just copy the bytes into an unsigned integer
of the same size, and output it as usual. If you need to
support other machines, however, you'll need to extract the
bits for each field, as they're defined by the format (which
defines them by reference to IEEE), and reassemble them using
things like ldexp.
> Or better, don't insist on an unreadable data format; use
> plain text instead.
That's always to be preferred. If only because it makes
debugging several orders of magnitude easier.
Note that if you want a portable format, even if it is a text
format, you'll still have to open the file as a binary file.
And output whatever the format requires for line endings. For
that matter, if you really want to be portable, you'll also have
to consider the possibility that your implementation of C++ uses
a different encoding internally than that required by the
format. Alternatively, you define the format as pure text, in
the native format for text, and require translation when moving
between systems. Historically, this is the traditional
solution, but it doesn't work very well if the machines are
physically connected or if you're sharing disks.
Mateusz Loskot
> On May 4, 9:13 am, Christopher <cp...@austin.rr.com> wrote:
>> On May 4, 1:46 am, Christopher <cp...@austin.rr.com> wrote:
>
>>> So, I am going to try it this way:
>>> int numBytes;
>>> fileread(static_cast<unsigned char *>(&numBytes), 4);
>
>> Oops, spoke too soon. It looks like I do indeed have to
>> reinterpret_cast
>
>> // Read 4 byte unsigned integral value
>> unsigned numBytes;
>> fileread(reinterpret_cast<char *>(&numBytes), 4);
>
> The reinterpret_cast is necessary because you're doint something
> that won't normally work.
It can be polished with some utility function templates:
// Name the cast properly, to self-document the code
template<typename T>
inline char* as_buffer(T& d)
{
return static_cast<char*>(static_cast<void*>(&d));
}
// Read binary data from input stream
template <typename T>
inline void read_n(T& d, std::istream& s, std::streamsize const& n)
{
if (!s)
throw std::runtime_error("input stream not readable");
s.read(as_buffer(d), n);
// good idea to check state of s
// one may need to handle endianness and swap bytes
}
Example, read 4 bytes from file stream ifs
int n = 0;
read_n(n, ifs, sizeof(n));
It should be also possible to specialize read_n function so it's
not necessary to give number of bytes explicitly.
Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
James Kanze
> James Kanze wrote:
> > On May 4, 9:13 am, Christopher <cp...@austin.rr.com> wrote:
> >> On May 4, 1:46 am, Christopher <cp...@austin.rr.com> wrote:
> >>> So, I am going to try it this way:
> >>> int numBytes;
> >>> fileread(static_cast<unsigned char *>(&numBytes), 4);
> >> Oops, spoke too soon. It looks like I do indeed have to
> >> reinterpret_cast
> >> // Read 4 byte unsigned integral value
> >> unsigned numBytes;
> >> fileread(reinterpret_cast<char *>(&numBytes), 4);
> > The reinterpret_cast is necessary because you're doint
> > something that won't normally work.
> It can be polished with some utility function templates:
> // Name the cast properly, to self-document the code
> template<typename T>
> inline char* as_buffer(T& d)
> {
> return static_cast<char*>(static_cast<void*>(&d));
> }
I think that there are cases where It still won't work (at least
in some cases, with g++)---even though the standard says it
should. (I think that there is an option in g++ which will make
it work. But since the overall approach is broken anyway,
there's no point.)
> // Read binary data from input stream
> template <typename T>
> inline void read_n(T& d, std::istream& s, std::streamsize const& n)
> {
> if (!s)
> throw std::runtime_error("input stream not readable");
> s.read(as_buffer(d), n);
> // good idea to check state of s
> // one may need to handle endianness and swap bytes
> }
> Example, read 4 bytes from file stream ifs
> int n = 0;
> read_n(n, ifs, sizeof(n));
Of course, you won't necessarily read the same value that was
written, unless it was written by the same binary, running on
the same machine.
> It should be also possible to specialize read_n function so
> it's not necessary to give number of bytes explicitly.
Or just use sizeof in the function.
The only real problem is that the code only works in very
special cases.
Mateusz Loskot
> On May 24, 11:04 pm, Mateusz Loskot <see...@signature.net> wrote:
>>
>> // Name the cast properly, to self-document the code
>> template<typename T>
>> inline char* as_buffer(T& d)
>> {
>> return static_cast<char*>(static_cast<void*>(&d));
>> }
>
> I think that there are cases where It still won't work (at least
> in some cases, with g++)---even though the standard says it
> should. (I think that there is an option in g++ which will make
> it work. But since the overall approach is broken anyway,
> there's no point.)
Do you mean the cast is broken or the template?
Could you explain what's broken in as_buffer?
>> // Read binary data from input stream
>> template <typename T>
>> inline void read_n(T& d, std::istream& s, std::streamsize const& n)
>> {
>> if (!s)
>> throw std::runtime_error("input stream not readable");
>
>> s.read(as_buffer(d), n);
>
>> // good idea to check state of s
>> // one may need to handle endianness and swap bytes
>> }
>
>> Example, read 4 bytes from file stream ifs
>
>> int n = 0;
>> read_n(n, ifs, sizeof(n));
>
> Of course, you won't necessarily read the same value that was
> written, unless it was written by the same binary, running on
> the same machine.
Of course. I've indicated byte order may be an issue.
>> It should be also possible to specialize read_n function so
>> it's not necessary to give number of bytes explicitly.
>
> Or just use sizeof in the function.
>
> The only real problem is that the code only works in very
> special cases.
I'm curious, because I've found it quite portable across various
implementations of C++ on Windows and Unix.
joshua...@gmail.com
Now, casting hackery is bit of a black art according to the guarantees
we have by the standard.
#include <iostream>
int main()
{ int x = 4;
std::cout.write(reinterpret_cast<char*>(&x), sizeof(x));
}
The above is not clearly guaranteed to work by the standard. There are
many passages alluding to and strongly suggesting that you can read
and write to anything through a char*, but nothing that actually says
the above program does what you want it to.
#include <iostream>
int main()
{ int x = 4;
std::cout.write(static_cast<char*>(static_cast<void*>(&x)), sizeof
(x));
}
The above program becomes even more iffy. I think by the letter and
intent the standard, this has undefined behavior. When casting to and
from a void*, you may only cast back to exactly the same type (maybe
with CV qualifier differences). In practice, I believe that
static_cast<char*>(static_cast<void*>(&x))
reinterpret_cast<char*>(&x)
are pretty much equivalent, though I don't think there's a passage in
the standard to support that.
Moreover, you should prefer
reinterpret_cast<char*>(&x)
over
static_cast<char*>(static_cast<void*>(&x))
They both have the same effect in practice, but the reinterpret_cast
says more clearly what you're doing. Going through void* is just an
obfuscation, especially when the static_casts are not in the same line
of code.
Other reasons why that code might not work as intended.
Endianness was mentioned.
The int might not be in two's complement. It might be stored as one's
complement or some other more bizarre format.
Your compiler isn't standards compliant. :)
However, I also don't quite follow what James Kanze is saying.
template<typename T>
inline char* as_buffer(T& d)
{ return static_cast<char*>(static_cast<void*>(&d));
}
James Kanze, what versions of g++ in what cases is it "where It still
won't work (at least
> > in some cases, with g++)---even though the standard says it should." I'm also very curious as my company's code does lots of hackery like this.
James Kanze
> James Kanze wrote:
> > On May 24, 11:04 pm, Mateusz Loskot <see...@signature.net> wrote:
> >> // Name the cast properly, to self-document the code
> >> template<typename T>
> >> inline char* as_buffer(T& d)
> >> {
> >> return static_cast<char*>(static_cast<void*>(&d));
> >> }
> > I think that there are cases where It still won't work (at
> > least in some cases, with g++)---even though the standard
> > says it should. (I think that there is an option in g++
> > which will make it work. But since the overall approach is
> > broken anyway, there's no point.)
> Do you mean the cast is broken or the template?
> Could you explain what's broken in as_buffer?
The fact that it just copies memory, instead of formatting.
> >> // Read binary data from input stream
> >> template <typename T>
> >> inline void read_n(T& d, std::istream& s, std::streamsize const& n)
> >> {
> >> if (!s)
> >> throw std::runtime_error("input stream not readable");
> >> s.read(as_buffer(d), n);
> >> // good idea to check state of s
> >> // one may need to handle endianness and swap bytes
> >> }
> >> Example, read 4 bytes from file stream ifs
> >> int n = 0;
> >> read_n(n, ifs, sizeof(n));
> > Of course, you won't necessarily read the same value that
> > was written, unless it was written by the same binary,
> > running on the same machine.
> Of course. I've indicated byte order may be an issue.
Amongst other things. Size, representation and padding are also
issues.
> >> It should be also possible to specialize read_n function so
> >> it's not necessary to give number of bytes explicitly.
> > Or just use sizeof in the function.
> > The only real problem is that the code only works in very
> > special cases.
> I'm curious, because I've found it quite portable across
> various implementations of C++ on Windows and Unix.
You've just not tested it.
James Kanze
> On May 25, 7:29 am, Mateusz Loskot <see...@signature.net> wrote:
> > James Kanze wrote:
> > > On May 24, 11:04 pm, Mateusz Loskot <see...@signature.net>
> > > wrote:
[...]
> However, I also don't quite follow what James Kanze is saying.
> template<typename T>
> inline char* as_buffer(T& d)
> { return static_cast<char*>(static_cast<void*>(&d));
>
> }
> James Kanze, what versions of g++ in what cases is it "where It still
> won't work (at least
> > > in some cases, with g++)---even though the standard says
> > > it should." I'm also very curious as my company's code
> > > does lots of hackery like this.
I don't know all of the exact details, but roughly speaking,
when g++ sees two pointers to different types, *even* if one of
those types is a character type, it assumes no aliasing. So,
for example, if you are writing through a char*, and reading
through an int*, the optimizer will assume that the write
doesn't change the value referenced by the int*, and may reorder
the read and the write.
As I said, I think that there's an option somewhere to turn this
off, and it really shouldn't affect you unless you are using
higher levels of optimization (I think).
Mateusz Loskot
> On May 25, 4:29 pm, Mateusz Loskot <see...@signature.net> wrote:
>> James Kanze wrote:
>>> On May 24, 11:04 pm, Mateusz Loskot <see...@signature.net> wrote:
>>>>
>>>> // Name the cast properly, to self-document the code
>>>> template<typename T>
>>>> inline char* as_buffer(T& d)
>>>> {
>>>> return static_cast<char*>(static_cast<void*>(&d));
>>>> }
>>>
>>> I think that there are cases where It still won't work (at
>>> least in some cases, with g++)---even though the standard
>>> says it should. (I think that there is an option in g++
>>> which will make it work. But since the overall approach is
>>> broken anyway, there's no point.)
>>
>> Do you mean the cast is broken or the template?
>> Could you explain what's broken in as_buffer?
>
> The fact that it just copies memory, instead of formatting.
I understand.
>>>> It should be also possible to specialize read_n function so
>>>> it's not necessary to give number of bytes explicitly.
>>>
>>> Or just use sizeof in the function.
>>>
>>> The only real problem is that the code only works in very
>>> special cases.
>>>
>> I'm curious, because I've found it quite portable across
>> various implementations of C++ on Windows and Unix.
>
> You've just not tested it.
Possible. I can see where is the problem now. Thanks!
joshua...@gmail.com
What version(s)? Are you referring to strict aliasing? I've written
tests on this for gcc 3.4.3, and it appears to correctly assume char*
can alias anything, and it optimizes assuming a short* does not alias
a int*. Is the option you refer to -fno-strict-aliasing?
James Kanze
> > [...]
I think so. As I say, I don't remember all of the details; I
just remember people having problems with it.
In general, it's a difficult issue. Possible aliasing reduces
optimization potential significantly---that's why C98 introduced
restrict. So a compiler rightfully tries to assume as much as
possible. On the other hand, something like:
union S
{
double d ;
int i ;
} ;
int f( double* d, int* i )
{
int result = *i ;
*d = 3.1415 ;
}
int
main()
{
S s ;
s.i = 42 ;
f( &s.d, &s.i ) ;
}
is clearly legal, but if f is in a different translation unit
than the union, there's no way the compiler can know that it
can't invert the read and write in f. (I think that there's
even a defect report about this in C. Because such things
pretty much negate what little guarantees there are concerning
aliasing.)
joshua...@gmail.com
> On the other hand, something like:
>
> union S
> {
> double d ;
> int i ;
> } ;
>
> int f( double* d, int* i )
> {
> int result = *i ;
> *d = 3.1415 ;
> }
>
> int
> main()
> {
> S s ;
> s.i = 42 ;
> f( &s.d, &s.i ) ;
> }
>
> is clearly legal, but if f is in a different translation unit
> than the union, there's no way the compiler can know that it
> can't invert the read and write in f. (I think that there's
> even a defect report about this in C. Because such things
> pretty much negate what little guarantees there are concerning
> aliasing.)
Odd that there is a defect report for the C language when I think your
example technically has undefined behavior in C. A defect report for
gcc maybe? It's my understanding that type punning with unions is
basically supported as an extension to the C89 standard by nearly all
C compilers and not by the C89 standard itself, and I don't know about
C99.
As for C++03, I think it does allow for type punning through a union,
but the wording in the standard is not the most clear.
You're right that this is quite unexpected and kills a lot of
guarantees we have. For example, take:
char getFirstByte(int arg)
{ union { int i; char c; }
i = x;
return c;
}
can be rewritten equivalently as:
char getFirstByte(int arg)
{ union { int i; char c; }
int * ip = & i;
*ip = x;
char * cp = & c;
char rv = *cp;
return rv;
}
I expect analogous code transformations when compiling that function's
analogue for user defined types. To implement type punning with a
union, the compiler assumes that for all unions in scope, the types of
members of each union can alias the other types of that union? Good to
know. Thanks.
James Kanze
> On May 26, 12:33 am, James Kanze <james.ka...@gmail.com> wrote:
> > On the other hand, something like:
> > union S
> > {
> > double d ;
> > int i ;
> > } ;
> > int f( double* d, int* i )
> > {
> > int result = *i ;
> > *d = 3.1415 ;
> > }
> > int
> > main()
> > {
> > S s ;
> > s.i = 42 ;
> > f( &s.d, &s.i ) ;
> > }
> > is clearly legal, but if f is in a different translation
> > unit than the union, there's no way the compiler can know
> > that it can't invert the read and write in f. (I think that
> > there's even a defect report about this in C. Because such
> > things pretty much negate what little guarantees there are
> > concerning aliasing.)
> Odd that there is a defect report for the C language when I
> think your example technically has undefined behavior in C.
No it's not. As the standard is currently written, it's
perfectly legal code, with well defined behavior.
> A defect report for gcc maybe? It's my understanding that type
> punning with unions is basically supported as an extension to
> the C89 standard by nearly all C compilers and not by the C89
> standard itself, and I don't know about C99.
Type punning with unions is undefined behavior, according to all
versions of both the C and the C++ standard. Any read access
may only refer to the last element written. But there's no type
punning in my example: the code writes S::i, then reads it, and
only then writes S::d. Unless the compiler reorders the reads
and writes in f, assuming no aliasing.
> As for C++03, I think it does allow for type punning through a
> union, but the wording in the standard is not the most clear.
The wording definitely could be clearer, but I'm pretty sure
that the intent is to do what C does.
> You're right that this is quite unexpected and kills a lot of
> guarantees we have. For example, take:
> char getFirstByte(int arg)
> { union { int i; char c; }
> i = x;
> return c;
> }
That's undefined behavior. Take a very close look at my
example, and the order of the various accesses.
Note that if I pass f an S*, instead of the int* and the
double*, there's no problem:
int
f( S* s )
{
int result = s->i ;
s->d = 3.1415 ;
return result ;
}
In this case, the aliasing is clearly visible. (Although I've
seen compilers which get this wrong as well. Early versions of
Microsoft C, for example.)
> can be rewritten equivalently as:
> char getFirstByte(int arg)
> { union { int i; char c; }
> int * ip = & i;
> *ip = x;
> char * cp = & c;
> char rv = *cp;
> return rv;
> }
> I expect analogous code transformations when compiling that
> function's analogue for user defined types. To implement type
> punning with a union, the compiler assumes that for all unions
> in scope, the types of members of each union can alias the
> other types of that union? Good to know. Thanks.
But of course, the C and the C++ standards don't approuve of
type punning with a union. The "standard sanctioned" way of
type punning is reinterpret_cast. But here, too, unless the
reinterpret_cast is visible in the function where the punning
takes place, I don't think it reasonable to expect it to work.
Regardless of the actual wording in the standard.
Bart van Ingen Schenau
The C standards committee does not agree with you.
They ruled that your example breaks the aliasing rules and has UB. See
http://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_236.htm
> --
> James Kanze (GABI Software) email:james...@gmail.com
Bart v Ingen Schenau
--
a.c.l.l.c-c++ FAQ: http://www.comeaucomputing.com/learn/faq
c.l.c FAQ: http://c-faq.com/
c.l.c++ FAQ: http://www.parashift.com/c++-faq-lite/
Mateusz Loskot
The term "perfectly" sounded strange to me too, though I was trying to
figure out if I understand it well or not, so I pulled out my voice.
Perhaps, James considers legality of this code as follows:
"However, it is an extremely common idiom and is well-supported
by all major compilers"
[1]
http://www.cellperformance.com/mike_acton/2006/06/understanding_strict_aliasing.html
joshua...@gmail.com
I apologize. I missed your point. You are indeed correct. That is a
defect in the compiler and the C standard as well. Remind me to use
unions with great caution now, and thoroughly test the results.
Giovanni Deretta
>
> I don't know all of the exact details, but roughly speaking,
> when g++ sees two pointers to different types, *even* if one of
> those types is a character type, it assumes no aliasing. So,
> for example, if you are writing through a char*, and reading
> through an int*, the optimizer will assume that the write
> doesn't change the value referenced by the int*, and may reorder
> the read and the write.
>
GCC definitely allows char pointers to alias anything, as required by
the standard. The documentation for -fstrict-aliasing explicitly says
that a character type may alias anything.
Anyways, future versions of GCC might have a more conservative TBAA:
every store (except for char) will be considered to change the dynamic
type of the stored location. This still allow many type based
optimizations, but will not break some legal usages of unions and
placement new that are sometimes miscompiled by gcc.
See http://gcc.gnu.org/wiki/MemoryModel for details.
> As I said, I think that there's an option somewhere to turn this
> off, and it really shouldn't affect you unless you are using
> higher levels of optimization (I think).
>
It is enabled by default at -O2.
HTH,
--
Giovanni P. Deretta
James Kanze
wrote:
Yes, but the text they cite to defend that position doesn't say
what they try to make it say. For their argument to hold,
§6.5/7 (in the C standard) would have to say something about
complete types. As it holds, the code above fulfills the
requirements of §6.5/7, since the object s.i is only accessed
through an lvalue expression with type i, and the object s.d
is only accessed through an lvalue expression with type d.
Of course, what they are arguing may very well be what they
meant to say. If so, then the wording in §6.5/7 is defective.
--
James Kanze (GABI Software) email:james...@gmail.com
James Kanze
No. The reason I raised this is because the above example does
NOT work with g++ (at least in some versions, with some
optimization options).
There are practical reasons why it doesn't work, and the authors
of the standard may not have meant that it should work, but as
currently worded, the standard guarantees that it does work (but
most compilers don't).
--
James Kanze (GABI Software) email:james...@gmail.com
James Kanze
> On May 26, 4:15 am, James Kanze <james.ka...@gmail.com> wrote:
[...]
> I apologize. I missed your point. You are indeed correct. That
> is a defect in the compiler and the C standard as well.
I think it is a defect in the standard, and that the compiler is
just implementing what was intended (which is not what is
written).
> Remind me to use unions with great caution now, and thoroughly
> test the results.
In general, use any aliasing with great caution. If the
aliasing is visible to the compiler (in the function body), I
would expect, from a quality of implementation point of view,
the code to do what is expected. If the aliasing isn't visible
(and most compilers don't look beyond the limits of a single
function), then all bets are off. Regardless of what the
standard actually says.
Bart van Ingen Schenau
On re-reading the issue and the committee argumentation, I have to agree
with you.
>
> --
> James Kanze (GABI Software) email:james...@gmail.com
Bart v Ingen Schenau
Mateusz Loskot
James,
I understand.
Thank you for patiently explaining this issue.