[QT creator on "nix"] - getting a strict 8 bit (1 byte) array with no padding

[Soviet_Mario]

I am trying to create a statically linked library in the
said IDE.
I would not want to include many QT features for now
so, what is the STANDARD C++ way to get
unsigned 8 bits integer type ?

I fear that unsigned char is system dependent (and even if
there are "wider" explicitely supported types, no plain char
type seems to warranty to be 8 bits wide).

The solution of creating an unnamed structure with a "bit
field like
struct {unsigned Pixel : 8} Mask;
gives me a warning like : 3 byte PADDING used to align the
struct Mask.

Now I don't know whether this padding would exist also in
ARRAY dynamic allocation (I'm not speaking for just the
first address of the array, but for EVERY SINGLE ITEM in it,
wasting as much as 3 times the space required !)

How to get rid of such alignment feature ?

in Visual Studio there were align / pad directives, but here
on QT I dunno anything.

I don't care a lot about fastness of access, but need a true
compact byte array (QT has its own, but I will need to
communicate with a Gambas program using the static library,
so I'd be not happy to interface QT internals with Gambas,
I'd prefer a standard C++ types-based solution).

long ago CHAR was 8 bit wide, but now it seems to be
platform dependent, and more often "a word" wide.

QT has its explicitely sized integers (even if no 8-bit sized)

TY


--
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
Soviet_Mario - (aka Gatto_Vizzato)

[Soviet_Mario]

Il 20/11/19 19:55, Soviet_Mario ha scritto:

Uhm, from some search, it seems that a "pragma pack" exist.
I still have to read it, though

[Soviet_Mario]

Il 20/11/19 19:55, Soviet_Mario ha scritto:

this

#pragma pack(push,1) // pushes the current alignment setting
on an internal stack and then optionally sets the new alignment.
typedef struct clsByte
{
union
{
unsigned PixelMask : 8;
};
} clsByte;
#pragma pack(pop) // restores former default value

seems to have suppressed the warning

A further question (general)

as I am accessing BYTES (not multibyte object), apart from
performance issues, do you think there could be a
portability issue ?
I am thinking that addressing strict bytes should require
accessing EVERY non aligned address in general (string
operations on ascii chars and odd length would be impossible
otherwise).

Is it correct ?

[Jorgen Grahn]

On Wed, 2019-11-20, Soviet_Mario wrote:
>
> I am trying to create a statically linked library in the
> said IDE.

The IDE doesn't matter; the Qt libraries do matter.

> I would not want to include many QT features for now
> so, what is the STANDARD C++ way to get
> unsigned 8 bits integer type ?

uint8_t, just like in C. Or, I guess it's really std::uint8_t.

> I fear that unsigned char is system dependent (and even if
> there are "wider" explicitely supported types, no plain char
> type seems to warranty to be 8 bits wide).

...

> long ago CHAR was 8 bit wide, but now it seems to be
> platform dependent, and more often "a word" wide.

No; a char is rarely more than 8 bits nowadays. You may be thinking
of wchar_t, but that's something different.

> QT has its explicitely sized integers (even if no 8-bit sized)

Qt has its own versions of most things ... are you sure you need it?

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[David Brown]

On 20/11/2019 19:55, Soviet_Mario wrote:
>
> I am trying to create a statically linked library in the said IDE.
> I would not want to include many QT features for now
> so, what is the STANDARD C++ way to get
> unsigned 8 bits integer type ?

uint8_t from <stdint.h>, or std::uint8_t from <cstdint>.

>
> I fear that unsigned char is system dependent (and even if there are
> "wider" explicitely supported types, no plain char type seems to
> warranty to be 8 bits wide).

> -based solution).
>

<snip>

> long ago CHAR was 8 bit wide, but now it seems to be platform dependent,
> and more often "a word" wide.
>

The size of "char" has always been platform dependent. Whether plain
"char" is signed or unsigned has also always been platform dependent.

On most systems, "char" is 8-bit, but the signedness varies - always
specify it explicitly ("signed char" or "unsigned char") when it is
relevant. Typically you only find char bigger than 8-bit on DSP's or
other specialist processors.

But if you stick to the standard type "uint8_t", you get exactly what
you ask for. (If the platform doesn't support 8-bit char, the type
won't exist and the code won't compile. But very little code needs to
be /that/ portable.)

[Paavo Helde]

On 20.11.2019 20:55, Soviet_Mario wrote:
>
> I am trying to create a statically linked library in the said IDE.
> I would not want to include many QT features for now
> so, what is the STANDARD C++ way to get
> unsigned 8 bits integer type ?

Why do you need an exactly 8-bit type? And how this is related to QT or
static linking, or to an IDE?

In general, if you need an unsigned 8-bit type, use std::uint8_t.

[James Kuyper]

On Wednesday, November 20, 2019 at 1:55:45 PM UTC-5, Soviet_Mario wrote:
> I am trying to create a statically linked library in the
> said IDE.
> I would not want to include many QT features for now
> so, what is the STANDARD C++ way to get
> unsigned 8 bits integer type ?

You should use uint8_t, a typedef declared in <cstdint>. The meaning of
this type is specified in the C standard, and incorporated into the C++
standard only by reference. The type is optional, you can check whether
or not UINT8_MAX is #defined to determine whether or not the type is
supported. As a practical matter, an implementation will fail to
support uint8_t only when compiling code to run on a platform that where
an 8-bit data type cannot be implemented, at least not efficiently.

You can also check CHAR_BIT, a macro #defined in <climits>. If
CHAR_BIT==8, then unsigned char can also be used, and uint8_t will
almost certainly be a typedef for unsigned char. As with uint8_t, if
CHAR_BIT!=8, then you an safely assume that the target platform is one
where an 8-bit type cannot be efficiently supported.

> I fear that unsigned char is system dependent (and even if
> there are "wider" explicitely supported types, no plain char
> type seems to warranty to be 8 bits wide).

Correct.

> The solution of creating an unnamed structure with a "bit
> field like
> struct {unsigned Pixel : 8} Mask;
> gives me a warning like : 3 byte PADDING used to align the
> struct Mask.

That approach will not work. The size of that struct is guaranteed to
be a positive integer number of bytes. On any platform where unsigned
char is NOT an 8-bit type, that struct is guaranteed to have a size
bigger than 8 bits. Even on a platform where CHAR_BIT==8, that struct
might be larger than 8 bits.

...

> long ago CHAR was 8 bit wide, but now it seems to be
> platform dependent, and more often "a word" wide.

There is no CHAR type in C. There is a char type - C++ is a case-
sensitive language, so that distinction is important. The number of bits
in a char object has always been platform dependent.

[Soviet_Mario]

Il 20/11/19 20:34, Jorgen Grahn ha scritto:

> On Wed, 2019-11-20, Soviet_Mario wrote:
>>
>> I am trying to create a statically linked library in the
>> said IDE.
>
> The IDE doesn't matter; the Qt libraries do matter.
>
>> I would not want to include many QT features for now
>> so, what is the STANDARD C++ way to get
>> unsigned 8 bits integer type ?
>
> uint8_t, just like in C. Or, I guess it's really std::uint8_t.

I opened the headers, i.g. sys/types.h and stdint.h
and they finally stem from a typedef that turns out to be a
"delusional" unsigned char.
I write delusional as I still do not have found a strict
warranty that a char is really 8 bit wide

>
>> I fear that unsigned char is system dependent (and even if
>> there are "wider" explicitely supported types, no plain char
>> type seems to warranty to be 8 bits wide).
> ...
>> long ago CHAR was 8 bit wide, but now it seems to be
>> platform dependent, and more often "a word" wide.
>
> No; a char is rarely more than 8 bits nowadays. You may be thinking
> of wchar_t, but that's something different.

ok, note taken. But is it a warranty or a very common guess ?

>
>> QT has its explicitely sized integers (even if no 8-bit sized)
>
> Qt has its own versions of most things ... are you sure you need it?

Yes it has many types redecorated its own way.

But as I wrote I'm not much confident on them on one side
(not on themselves, on my skills obv.) and I often cannot
even guess what they could be like from "outside".
With gambas I just got to work with integral types, even
"strings" are still very difficult to me
So I'd feel better the less "wrappers" around types are put.

Now I'm exploring the solution with a struct containing only
a 8 bit field, and the #pragma pack

>
> /Jorgen

[Soviet_Mario]

Il 20/11/19 22:03, David Brown ha scritto:

> On 20/11/2019 19:55, Soviet_Mario wrote:
>>
>> I am trying to create a statically linked library in the
>> said IDE.
>> I would not want to include many QT features for now
>> so, what is the STANDARD C++ way to get
>> unsigned 8 bits integer type ?
>
> uint8_t from <stdint.h>, or std::uint8_t from <cstdint>.

i inquired, and they expand to a typedef related to unsigned
char

>
>>
>> I fear that unsigned char is system dependent (and even if
>> there are "wider" explicitely supported types, no plain
>> char type seems to warranty to be 8 bits wide).
>> -based solution).
>>
>
> <snip>
>
>> long ago CHAR was 8 bit wide, but now it seems to be
>> platform dependent, and more often "a word" wide.
>>
>
> The size of "char" has always been platform dependent.
> Whether plain "char" is signed or unsigned has also always
> been platform dependent.
>
> On most systems, "char" is 8-bit, but the signedness varies
> - always specify it explicitly ("signed char" or "unsigned
> char") when it is relevant.  Typically you only find char
> bigger than 8-bit on DSP's or other specialist processors.
>
> But if you stick to the standard type "uint8_t", you get
> exactly what you ask for.  (If the platform doesn't support
> 8-bit char, the type won't exist and the code won't
> compile.  But very little code needs to be /that/ portable.)

uhm ... maybe. But I am trying to mirror a byte-wise mask of
opacity/transparency from images, so I have to get two
different pieces of programs (a front end in gambas and this
would-be static library to speak the same language and have
the same data layout).

To be very very tight fisted, I could even have devoted a
SINGLE BIT to the transparency mask, but addressing would
have become far slower. So I waste 7 bits ! but no more.

The pictures themselves are also made of bit fields, but are
four fields 8 bit each, so exactly 32 bit / pixel and the
packing was not necessary there (and also align issue did
not show up).

the gambas frontend has a native type BYTE (0-255 range)
for the transparency mask so I have to replicate it in the
library

[Öö Tiib]

On Thursday, 21 November 2019 01:54:39 UTC+2, Soviet_Mario wrote:
> Il 20/11/19 20:34, Jorgen Grahn ha scritto:
> >

> > No; a char is rarely more than 8 bits nowadays. You may be thinking
> > of wchar_t, but that's something different.
>
> ok, note taken. But is it a warranty or a very common guess ?

Use std::uint8_t from <cstdint> or uint8_t from <stdint.h> anyway
just to document what you do. Both must be only present when the
platform supports 8 bit unsigned integers. I don't know existence
of any platforms with conforming C++ compilers that do not support
those or where those are not unsigned char but so what?

>
> Now I'm exploring the solution with a struct containing only
> a 8 bit field, and the #pragma pack

An uint8_t directly or wrapped into struct feels better.
Your unsigned bitfield in anonymous union in packed struct
is possibly not very well tested / profiled in all situations on
all compilers and so may have odd differences of performance
or the like.

[Manfred]

On 11/21/19 12:54 AM, Soviet_Mario wrote:
>> uint8_t, just like in C.  Or, I guess it's really std::uint8_t.
>
> I opened the headers, i.g. sys/types.h and stdint.h
> and they finally stem from a typedef that turns out to be a "delusional"
> unsigned char.
> I write delusional as I still do not have found a strict warranty that a
> char is really 8 bit wide

It's not delusional, it's how it works.
You opened some standard header of your implementation, and since in
your implementation a byte is an unsigned char, this is the correct typedef.
If you happen to port the code to an implementation (if any) where a
byte has a different representation, then the standard header of that
implementation would have the corresponding definition.

This way of working is most evident with stuff like uint32_t and
uint64_t (as many have said, most of the time uint8_t is an unsigned char).

If porting to an implementation where no byte is available, then uint8_t
would not be defined (also said by others).

Gist of the story is, yes, uint8_t is guaranteed to be an 8-bit unsigned
integer, as its name suggests.

[James Kuyper]

On Wednesday, November 20, 2019 at 6:54:39 PM UTC-5, Soviet_Mario wrote:
> Il 20/11/19 20:34, Jorgen Grahn ha scritto:
> > On Wed, 2019-11-20, Soviet_Mario wrote:

...

> >> I would not want to include many QT features for now
> >> so, what is the STANDARD C++ way to get
> >> unsigned 8 bits integer type ?
> >
> > uint8_t, just like in C. Or, I guess it's really std::uint8_t.
>
> I opened the headers, i.g. sys/types.h and stdint.h
> and they finally stem from a typedef that turns out to be a
> "delusional" unsigned char.
> I write delusional as I still do not have found a strict
> warranty that a char is really 8 bit wide

You won't find it, because there is no such warranty in general. The
minimum value for CHAR_BIT is quite explicitly 8, but there is no
maximum, and it's certainly not required to be 8. Values other than 8
are rare, but not unheard of.

However, any conforming implementation of C++ for which <cstdint>
contains a typedef of uint8_t as unsigned char IS guaranteed to be one
for which unsigned char is an 8 bit type. That's not because of anything
the standard says directly about unsigned char; it's because it's not
permitted to for uint8_t to be a typedef for a type that isn't exactly 8
bits. The relevant guarantee is in the C standard, and incorporated into
the C++ standard by reference, but it is still a guarantee.

If CHAR_BIT != 8, or #ifndef UINT8_MAX, then there will be no way to
declare an 8-bit type for that implementation (except for bit-fields,
and you can't make arrays of bit-fields).

The key question the OP needs to ask himself is "What do I want my
program to do when compiled on a platform where there is no way to
declare an 8-bit type?". There's many options: failing to compile with
an explicit error message is one possibility. Depending upon what he
actually needs to do, uint_least8_t might be an acceptable alternative,
though I doubt it.

...
> >> long ago CHAR was 8 bit wide, but now it seems to be
> >> platform dependent, and more often "a word" wide.
> >
> > No; a char is rarely more than 8 bits nowadays. You may be thinking
> > of wchar_t, but that's something different.
>
> ok, note taken. But is it a warranty or a very common guess ?

It's very common, not guaranteed, and there's no need to guess about it.
Just check the value of CHAR_BIT.

...

> Now I'm exploring the solution with a struct containing only

> a 8 bit field, and the #pragma pack.

On a platform where CHAR_BIT==8, unsigned char will work. On a platform
where CHAR_BIT != 8, that approach won't work, nor will any other.

[Reinhardt Behm]

Of course it has: Q_UINT8

The standard way would be (u)int8_t, if it is defined in your compiler.

[Soviet_Mario]

Il 21/11/19 01:48, Manfred ha scritto:

mmm, all changes then, as it was the warranty I was looking
for !

[Soviet_Mario]

Il 21/11/19 02:44, James Kuyper ha scritto:

> On Wednesday, November 20, 2019 at 6:54:39 PM UTC-5, Soviet_Mario wrote:
>> Il 20/11/19 20:34, Jorgen Grahn ha scritto:
>>> On Wed, 2019-11-20, Soviet_Mario wrote:
> ...
>>>> I would not want to include many QT features for now
>>>> so, what is the STANDARD C++ way to get
>>>> unsigned 8 bits integer type ?
>>>
>>> uint8_t, just like in C. Or, I guess it's really std::uint8_t.
>>
>> I opened the headers, i.g. sys/types.h and stdint.h
>> and they finally stem from a typedef that turns out to be a
>> "delusional" unsigned char.
>> I write delusional as I still do not have found a strict
>> warranty that a char is really 8 bit wide
>
> You won't find it, because there is no such warranty in general. The
> minimum value for CHAR_BIT is quite explicitly 8, but there is no
> maximum, and it's certainly not required to be 8. Values other than 8
> are rare, but not unheard of.
>
> However, any conforming implementation of C++ for which <cstdint>
> contains a typedef of uint8_t as unsigned char IS guaranteed to be one
> for which unsigned char is an 8 bit type. That's not because of anything
> the standard says directly about unsigned char; it's because it's not
> permitted to for uint8_t to be a typedef for a type that isn't exactly 8
> bits. The relevant guarantee is in the C standard, and incorporated into
> the C++ standard by reference, but it is still a guarantee.

very clear (and very good for me!)
tnx

>
> If CHAR_BIT != 8, or #ifndef UINT8_MAX, then there will be no way to
> declare an 8-bit type for that implementation (except for bit-fields,
> and you can't make arrays of bit-fields).

actually I made an array of the enclosing struct of the bit
field (which was the only member but this may not be relevant).
I noticed that the "padding warning disappeared" with the
#pragma pack(1)
which might be a proof that the system is capable of
addressing single bytes regardless of their alignment in memory

but all this now is unnecessary, after what you all have stated

>
> The key question the OP needs to ask himself is "What do I want my
> program to do when compiled on a platform where there is no way to
> declare an 8-bit type?".

none, I won't possibly port it anywhere, just run here locally

> There's many options: failing to compile with
> an explicit error message is one possibility. Depending upon what he
> actually needs to do, uint_least8_t might be an acceptable alternative,
> though I doubt it.
>
> ...
>>>> long ago CHAR was 8 bit wide, but now it seems to be
>>>> platform dependent, and more often "a word" wide.
>>>
>>> No; a char is rarely more than 8 bits nowadays. You may be thinking
>>> of wchar_t, but that's something different.
>>
>> ok, note taken. But is it a warranty or a very common guess ?
>
> It's very common, not guaranteed, and there's no need to guess about it.
> Just check the value of CHAR_BIT.
>
> ...
>> Now I'm exploring the solution with a struct containing only
>> a 8 bit field, and the #pragma pack.
>
> On a platform where CHAR_BIT==8, unsigned char will work. On a platform
> where CHAR_BIT != 8, that approach won't work, nor will any other.

understood. So the workaround worked only because the system
was just able to address single bytes

[Keith Thompson]

Soviet_Mario <Sovie...@CCCP.MIR> writes:
> I am trying to create a statically linked library in the
> said IDE.
> I would not want to include many QT features for now
> so, what is the STANDARD C++ way to get
> unsigned 8 bits integer type ?

If uint8_t exists, then both it and unsigned char are unsigned 8-bit
integer types (and very likely uint8_t is an alias for unsigned char).
Using uint8_t makes it more explicit that you want an unsigned 8-bit
integer type.

If uint8_t doesn't exist, then CHAR_BIT > 8 and there is no 8-bit
integer type. You're unlikely to encounter such an implementation
(I've only heard of very old systems and DSPs) -- and I'd be
astonished if such a system were able to support Qt.

(There's probably some wiggle room there. An implementation with
CHAR_BIT==8 with signed char using a representation other than
2's-complement won't define int8_t and, IIRC, therefore shouldn't define
uint8_t -- but such implementations are, I think, even rarer.)

> I fear that unsigned char is system dependent (and even if
> there are "wider" explicitely supported types, no plain char
> type seems to warranty to be 8 bits wide).

Correct. The three char types are CHAR_BIT bits, or 1 byte.

Your best bet is probably to use uint8_t, which guarantees that your
code won't compile if the implementation doesn't meet its requirements.

[...]

--
Keith Thompson (The_Other_Keith) ks...@mib.org <http://www.ghoti.net/~kst>
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */

[Keith Thompson]

Soviet_Mario <Sovie...@CCCP.MIR> writes:
> Il 20/11/19 19:55, Soviet_Mario ha scritto:
> Uhm, from some search, it seems that a "pragma pack" exist.
> I still have to read it, though

pragma pack is a common extension, but it's non-standard. If you're
worried enough about portability that you're unwilling to assume
CHAR_BIT==8, you probably don't want to depend on extensions.

[Keith Thompson]

Keith Thompson <ks...@mib.org> writes:
> Soviet_Mario <Sovie...@CCCP.MIR> writes:
>> I am trying to create a statically linked library in the
>> said IDE.
>> I would not want to include many QT features for now
>> so, what is the STANDARD C++ way to get
>> unsigned 8 bits integer type ?
>
> If uint8_t exists, then both it and unsigned char are unsigned 8-bit
> integer types (and very likely uint8_t is an alias for unsigned char).
> Using uint8_t makes it more explicit that you want an unsigned 8-bit
> integer type.
>
> If uint8_t doesn't exist, then CHAR_BIT > 8 and there is no 8-bit
> integer type. You're unlikely to encounter such an implementation
> (I've only heard of very old systems and DSPs) -- and I'd be
> astonished if such a system were able to support Qt.

[...]

I'm curious about something. Do you have a realistic expectation that
your code might need to work under an implementation with CHAR_BIT > 8?
Do you have such a target system in mind?

C++ with the assumption that CHAR_BIT==8 is realistically at least 99%
as portable as C++ without that assumption -- and far more portable than
C++ with the assumption that Qt is available.

[Soviet_Mario]

Il 21/11/19 07:43, Keith Thompson ha scritto:

> Keith Thompson <ks...@mib.org> writes:
>> Soviet_Mario <Sovie...@CCCP.MIR> writes:
>>> I am trying to create a statically linked library in the
>>> said IDE.
>>> I would not want to include many QT features for now
>>> so, what is the STANDARD C++ way to get
>>> unsigned 8 bits integer type ?
>>
>> If uint8_t exists, then both it and unsigned char are unsigned 8-bit
>> integer types (and very likely uint8_t is an alias for unsigned char).
>> Using uint8_t makes it more explicit that you want an unsigned 8-bit
>> integer type.
>>
>> If uint8_t doesn't exist, then CHAR_BIT > 8 and there is no 8-bit
>> integer type. You're unlikely to encounter such an implementation
>> (I've only heard of very old systems and DSPs) -- and I'd be
>> astonished if such a system were able to support Qt.
> [...]
>
> I'm curious about something. Do you have a realistic expectation that
> your code might need to work under an implementation with CHAR_BIT > 8?

no, not at all.
My only requirement is that the plain native type BYTE used
in gambas code that calls the C++ static library and that
expects a 8-bit wide unsigned integral, find its matching
type in the C++ code, as they need to share bitmaps and
transparency maps.
The bitmaps use 32-bit packed colors+alpha channel, so I was
assuming no difficulty in exchange.
The transparency mask (due to blend different images
pattern-wise) theoretically would need just ONE BIT (not
byte, bit) per pixel, but this would have made much more
complex (and slow) the single pixel addressing and even
worse calculation in gambas who does not have a good support
for single bit R/W
so I decided to waste some space and to use an addressable
unit of 8 bits for a boolean value

> Do you have such a target system in mind?

No, and I wont' even try to figure out. I need the program
to work in just two machines (and AMD 10 y old and an INTEL
1 month old), both with linux.

My only problem is to get the GUI client (gambas) to be able
to call consistently the C++ static lib

I still have not decided whether to exchange big bitmap-like
rectangular arrays directly via pointers or via memory files.
I have to assess if and how the FILE type for gambas is seen
outside ...

>
> C++ with the assumption that CHAR_BIT==8 is realistically at least 99%
> as portable as C++ without that assumption -- and far more portable than
> C++ with the assumption that Qt is available.

I have put QT on every distro in use, but I hope that
choosing the static library solution would produce a binary
with all the needed QT essentials hard wired in it and no
further dependencies (just a guess, all to be verified).

I'm moving just the first steps

Initially I was writing all in gambas (which is very good
for GUI), but I got stuck, its object pattern is too
different from C/C++ representation and regularly when I
lose control of data representation, I tend to resort to C++
way of design it, which is hugely more flexible and precise.
Then the problem of mixing code arises, though :\ :\

[David Brown]

On 21/11/2019 01:00, Soviet_Mario wrote:
> Il 20/11/19 22:03, David Brown ha scritto:
>> On 20/11/2019 19:55, Soviet_Mario wrote:
>>>
>>> I am trying to create a statically linked library in the said IDE.
>>> I would not want to include many QT features for now
>>> so, what is the STANDARD C++ way to get
>>> unsigned 8 bits integer type ?
>>
>> uint8_t from <stdint.h>, or std::uint8_t from <cstdint>.
>
> i inquired, and they expand to a typedef related to unsigned char
>

That is because <stdint.h> and <cstdint> (and other standard library
headers) are adapted to suit the target processor.

uint8_t is a bad example here, because (AFAIK) if a compiler supports a
type that could be used for uint8_t, then "unsigned char" is always
going to be suitable for it. (That does not mean it is the only choice.)

Consider uint32_t. This type is guaranteed to be 32-bit, unsigned, with
no padding bits. If a C99/C++11 compiler has a type that can fit these
requirements (along with a matching int32_t), it /must/ provide uint32_t
and int32_t types. But it is up to the implementation - the compiler
and the standard headers - to decide how to do that. On one platform,
it might have "typedef unsigned int uint32_t;". On another, it could
have "typedef unsigned long int uint32_t;". On another, it could have
"typedef __UINT32_TYPE__ uint32_t;". The standard headers, like
<stdint.h>, are /not/ portable - they are fixed for the given compiler
implementation.

Thus when you want a 32-bit unsigned integer, you use "uint32_t" and let
the C/C++ implementation worry about how it makes that work on this
particular target.

The fixed size types exist precisely to let you write code in a portable
way with specific requirements for the types, without having to know
details of the underlying target.

>>
>>>
>>> I fear that unsigned char is system dependent (and even if there are
>>> "wider" explicitely supported types, no plain char type seems to
>>> warranty to be 8 bits wide).
>>> -based solution).
>>>
>>
>> <snip>
>>
>>> long ago CHAR was 8 bit wide, but now it seems to be platform
>>> dependent, and more often "a word" wide.
>>>
>>
>> The size of "char" has always been platform dependent.  Whether plain
>> "char" is signed or unsigned has also always been platform dependent.
>>
>> On most systems, "char" is 8-bit, but the signedness varies - always
>> specify it explicitly ("signed char" or "unsigned char") when it is
>> relevant.  Typically you only find char bigger than 8-bit on DSP's or
>> other specialist processors.
>>
>> But if you stick to the standard type "uint8_t", you get exactly what
>> you ask for.  (If the platform doesn't support 8-bit char, the type
>> won't exist and the code won't compile.  But very little code needs to
>> be /that/ portable.)
>
> uhm ... maybe. But I am trying to mirror a byte-wise mask of
> opacity/transparency from images, so I have to get two different pieces
> of programs (a front end in gambas and this would-be static library to
> speak the same language and have the same data layout).

Use uint8_t.

>
> To be very very tight fisted, I could even have devoted a SINGLE BIT to
> the transparency mask, but addressing would have become far slower. So I
> waste 7 bits ! but no more.

uint8_t is always 8 bits. No more, no less.

>
> The pictures themselves are also made of bit fields, but are four fields
> 8 bit each, so exactly 32 bit / pixel and the packing was not necessary
> there (and also align issue did not show up).
>
> the gambas frontend has a native type BYTE  (0-255 range) for the
> transparency mask so I have to replicate it in the library
>

C and C++ have an equivalent type. It is called "uint8_t".

[David Brown]

And if it is not defined by the compiler, there is no Q_UINT8 type
either. In fact, there is no QT for the target.

(The exception here would be a very old version of a poor quality
compiler that does not support C++11 or C99 - then you might be missing
<stdint.h> and <cstdint>.)

[Paavo Helde]

On 21.11.2019 13:53, Soviet_Mario wrote:
> Il 21/11/19 07:43, Keith Thompson ha scritto:
>>

>> I'm curious about something. Do you have a realistic expectation that
>> your code might need to work under an implementation with CHAR_BIT > 8?
>
> no, not at all.

Good. I suspect that without lots of porting and testing efforts none of
these software pieces would work with CHAR_BIT>8 - neither gambas,
neither QT and neither your code.

> My only requirement is that the plain native type BYTE used in gambas
> code that calls the C++ static library and that expects a 8-bit wide
> unsigned integral, find its matching type in the C++ code, as they need
> to share bitmaps and transparency maps.

This is easy, use unsigned char or std::uint8_t.

> The bitmaps use 32-bit packed colors+alpha channel, so I was assuming no
> difficulty in exchange.

This is more tricky because the size of int or long varies much more
than char. For example, long is 32 bits in 64-bit MSVC and 64 bits with
64-bit gcc on Linux.

However, for your purposes std::uint32_t should work fine.

> The transparency mask (due to blend different images pattern-wise)
> theoretically would need just ONE BIT (not byte, bit) per pixel, but
> this would have made much more complex (and slow) the single pixel
> addressing and even worse calculation in gambas who does not have a good
> support for single bit R/W
> so I decided to waste some space and to use an addressable unit of 8
> bits for a boolean value

This is a common choice, ok.

> I have put QT on every distro in use, but I hope that choosing the
> static library solution would produce a binary with all the needed QT
> essentials hard wired in it and no further dependencies (just a guess,
> all to be verified).

Qt is mostly a GUI library as far as I have understood. Of course it has
all bells and whistles like threading support and datetime library and
whatever, but these things are nowadays also present in standard C++. So
if you are writing some back-end library with no GUI interface involved,
you actually should not need Qt at all.

[Soviet_Mario]

Il 21/11/19 17:20, Paavo Helde ha scritto:

> On 21.11.2019 13:53, Soviet_Mario wrote:

CUT

>
> Qt is mostly a GUI library as far as I have understood. Of
> course it has all bells and whistles like threading support
> and datetime library and whatever, but these things are
> nowadays also present in standard C++. So if you are writing
> some back-end library with no GUI interface involved, you
> actually should not need Qt at all.
>
>

uhm ... this might sound silly ... but I had not thought a
lot about this choice. And actually I find it a darn good
advice.
I am using an elephant to tow a feather :\
I used it without much thought, but now that you said it,
CodeBlocks would be more than enough.
Ciao !

[Mr Flibble]

Qt has too much legacy cruft: QVector? LOL. My alternative, neoGFX, will ultimately win out as I have the distinct advantage of being able to support modern C++ as I have no legacy user-base to worry about.

/Flibble

--
"Snakes didn't evolve, instead talking snakes with legs changed into snakes." - Rick C. Hodgin

“You won’t burn in hell. But be nice anyway.” – Ricky Gervais

“I see Atheists are fighting and killing each other again, over who doesn’t believe in any God the most. Oh, no..wait.. that never happens.” – Ricky Gervais

"Suppose it's all true, and you walk up to the pearly gates, and are confronted by God," Byrne asked on his show The Meaning of Life. "What will Stephen Fry say to him, her, or it?"
"I'd say, bone cancer in children? What's that about?" Fry replied.
"How dare you? How dare you create a world to which there is such misery that is not our fault. It's not right, it's utterly, utterly evil."
"Why should I respect a capricious, mean-minded, stupid God who creates a world that is so full of injustice and pain. That's what I would say."

[Keith Thompson]

David Brown <david...@hesbynett.no> writes:
[...]

> uint8_t is always 8 bits. No more, no less.

[...]

Yes. Expanding on that, uint8_t is always 8 bits *if it exists*.

(And in the real world, it almost always exists. You're far more
likely to encounter a very old implementation that doesn't support
<stdint.h> or <cstdint> than to encounter one that does have those
headers but has CHAR_BIT>8.)

[David Brown]

On 21/11/2019 20:30, Keith Thompson wrote:
> David Brown <david...@hesbynett.no> writes:
> [...]
>> uint8_t is always 8 bits. No more, no less.
> [...]
>
> Yes. Expanding on that, uint8_t is always 8 bits *if it exists*.
>
> (And in the real world, it almost always exists. You're far more
> likely to encounter a very old implementation that doesn't support
> <stdint.h> or <cstdint> than to encounter one that does have those
> headers but has CHAR_BIT>8.)
>

That is certainly true if by "in the real world" you mean "platforms
that support QT" :-)

I've worked with a couple of devices that have 16-bit CHAR, and been
indirectly connected with a project using one with 32-bit CHAR. These
were DSP devices, but are current devices, with new family members being
added regularly. But it is very rare that you want much software that
will be portable between these sorts of chips and other more "normal"
processors.

Balancing that, I have a few compilers that I still have occasion to use
that don't support <stdint.h> (or C++, for that matter).

[Christian Gollwitzer]

Am 21.11.19 um 17:20 schrieb Paavo Helde:

> On 21.11.2019 13:53, Soviet_Mario wrote:
>> My only requirement is that the plain native type BYTE used in gambas
>> code that calls the C++ static library and that expects a 8-bit wide
>> unsigned integral, find its matching type in the C++ code, as they need
>> to share bitmaps and transparency maps.
>
> This is easy, use unsigned char or std::uint8_t.
>
>> The bitmaps use 32-bit packed colors+alpha channel, so I was assuming no
>> difficulty in exchange.
>
> This is more tricky because the size of int or long varies much more
> than char. For example, long is 32 bits in 64-bit MSVC and 64 bits with
> 64-bit gcc on Linux.
>
> However, for your purposes std::uint32_t should work fine.

Why not

#pragma pack(1)
struct pixel {
uint8_t red, green, blue, alpha;
}

? That would make more sense to me than uint32_t and bit-twiddling to
get the channel values.

Christian

[Keith Thompson]

Christian Gollwitzer <auri...@gmx.de> writes:
> Am 21.11.19 um 17:20 schrieb Paavo Helde:

[...]

>> However, for your purposes std::uint32_t should work fine.
>
> Why not
>
> #pragma pack(1)
> struct pixel {
> uint8_t red, green, blue, alpha;
> }
>
> ? That would make more sense to me than uint32_t and bit-twiddling to
> get the channel values.

Because #pragma pack is non-standard?

If you only care about implementations that support it as an extension,
that's fine, but the OP seemed very concerned about portability.

An array of uint8_t could also work:

typedef uint8_t pixel[4];
enum { red, green, blue, alpha };

Or an array wrapped in a struct to allow assignment. (An array of 4
uint8_t elements is guaranteed to be 32 bits, but a struct containing
such an array is not -- but it's a nearly safe assumption that there
will be no padding between the elements or at the end.)

[David Brown]

If you are using "pragma pack" (or gcc "packed" attribute), you are
probably doing something wrong. It can have some uses when dealing with
file formats or network packets with unaligned fields. But most of the
cases I see - such as this one (and the OP's first suggestion) are
misunderstandings and misuses.

When you use "pragma pack", you are making code that is non-standard,
non-portable, and can cause trouble interacting with other code. (What
happens when you mix pointers to a packed version of your struct, and a
non-packed version? Are they compatible? Will your compiler warn you
of mixups? My guess is you don't know, and that is not a good thing.)

They can lead to inefficiencies - on some compilers and targets, packed
fields are accessed by byte. Even when the fields are accessed by their
full size, packed means you can get misalignments, which can have a very
significant impact on efficiency in some situations.

They can lead to restrictions in the code. For some tools, you can't
take the address of a packed field, or you might see problems using it
because the compiler assumes pointers are properly aligned and that
might not be the case for packed fields.

In short, "packed" can lead to a lot of extra complications and
muck-ups. Perhaps the most insidious aspect is that /usually/ you don't
see problems with them. Things will work while you are writing /this/
code, and testing it with /this/ compiler - the problems will turn up
later when someone else gets trouble using the code in a different
project, and won't know why.

So use "packed" only when you /really/ need to - when it is clearly the
best solution to the problem. And only when you /really/ understand
what it does, and why it might be useful.

And clearly, in this case, it is utterly pointless. Just write the
struct directly - it is impossible for "pack" to be of any help here.
And if you don't understand that, you certainly should not be using
"pack". (And you can ask here for more help in understanding, if this
post is not sufficient.)

[David Brown]

On 22/11/2019 09:37, Keith Thompson wrote:
> Christian Gollwitzer <auri...@gmx.de> writes:
>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
> [...]
>>> However, for your purposes std::uint32_t should work fine.
>>
>> Why not
>>
>> #pragma pack(1)
>> struct pixel {
>> uint8_t red, green, blue, alpha;
>> }
>>
>> ? That would make more sense to me than uint32_t and bit-twiddling to
>> get the channel values.
>
> Because #pragma pack is non-standard?
>
> If you only care about implementations that support it as an extension,
> that's fine, but the OP seemed very concerned about portability.
>
> An array of uint8_t could also work:
>
> typedef uint8_t pixel[4];
> enum { red, green, blue, alpha };
> > Or an array wrapped in a struct to allow assignment. (An array of 4
> uint8_t elements is guaranteed to be 32 bits, but a struct containing
> such an array is not -- but it's a nearly safe assumption that there
> will be no padding between the elements or at the end.)
>

While it is true in theory that a struct can have extra padding at the
end (beyond what is needed for an array of the struct to have proper
alignment for all members), are there any compilers that have done so in
practice?

Another option is to cover all bases with a union:

typedef union {
uint32_t combined;
uint8_t rgba[4];
struct {

uint8_t red, green, blue, alpha;

};
} pixel;

_Static_assert(sizeof(pixel) == 4, "This only works on real systems!");

(Watch out for endian issues giving different orders within these parts
- that's always a concern for portable code when matching external file
formats.)

[Öö Tiib]

On Friday, 22 November 2019 10:37:59 UTC+2, David Brown wrote:
>
> When you use "pragma pack", you are making code that is non-standard,
> non-portable, and can cause trouble interacting with other code. (What
> happens when you mix pointers to a packed version of your struct, and a
> non-packed version? Are they compatible? Will your compiler warn you
> of mixups? My guess is you don't know, and that is not a good thing.)

Both gcc and clang have -Waddress-of-packed-member but AFAIK you
won't get that warning when compiling for platform on what it does not
matter.

> They can lead to inefficiencies - on some compilers and targets, packed
> fields are accessed by byte. Even when the fields are accessed by their
> full size, packed means you can get misalignments, which can have a very
> significant impact on efficiency in some situations.

Huh? Packed structs usually lead to raise of storage efficiency.
Say there is struct containing a char and double. Normally it is 16 bytes
but packed it is 9 bytes. 43.75% of storage saved!
Where we have to process large array of those there it is often impossible
to predict without profiling on target platform if that will cause performance
gain or loss.

[Soviet_Mario]

Il 22/11/19 09:37, Keith Thompson ha scritto:

> Christian Gollwitzer <auri...@gmx.de> writes:
>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
> [...]
>>> However, for your purposes std::uint32_t should work fine.
>>
>> Why not
>>
>> #pragma pack(1)
>> struct pixel {
>> uint8_t red, green, blue, alpha;
>> }
>>
>> ? That would make more sense to me than uint32_t and bit-twiddling to
>> get the channel values.
>
> Because #pragma pack is non-standard?
>
> If you only care about implementations that support it as an extension,
> that's fine, but the OP seemed very concerned about portability.
>
> An array of uint8_t could also work:
>
> typedef uint8_t pixel[4];

intresting idea !!!

> enum { red, green, blue, alpha };

btw since here #pragma pack seems supported, I find the
"dot" member selector not only clearer (more readable) but
more amenable to "intellisense" when typing (type less, no
typos as autocompletion pops up the members).

but your solution is elegant in that uses more intrinsic
features

>
> Or an array wrapped in a struct to allow assignment. (An array of 4
> uint8_t elements is guaranteed to be 32 bits, but a struct containing

actually I have the struct with individual "channels"
blended in a UNION with a u_int_32_t 32 bit value, which I
use when I have to do fast memberwise copy (operator =,
copy-constructor) or even apply bitwise operators on all
members at a time (like NOT, XOR etc).
I dunno whether being aligned to the same initial address
changes anything ... after having used u_int_8_t (four
distinct members) even without the #pragma directive the
compiler does not detect further PADDING issues (a feature
that it has just proved itself to be well aware of despite
my ignoring that before)

> such an array is not -- but it's a nearly safe assumption that there
> will be no padding between the elements or at the end.)
>


--

[Öö Tiib]

It usually works in practice but is undefined behavior in C++, so Bjarne
Stroustrup and Herb Sutter suggest against it in C++ Core Guidelines.

http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Ru-pun

[Soviet_Mario]

Il 22/11/19 09:37, David Brown ha scritto:

> On 22/11/2019 07:26, Christian Gollwitzer wrote:
>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
>>> On 21.11.2019 13:53, Soviet_Mario wrote:
>>>> My only requirement is that the plain native type BYTE used in gambas
>>>> code that calls the C++ static library and that expects a 8-bit wide
>>>> unsigned integral, find its matching type in the C++ code, as they need
>>>> to share bitmaps and transparency maps.
>>>
>>> This is easy, use unsigned char or std::uint8_t.
>>>
>>>> The bitmaps use 32-bit packed colors+alpha channel, so I was assuming no
>>>> difficulty in exchange.
>>>
>>> This is more tricky because the size of int or long varies much more
>>> than char. For example, long is 32 bits in 64-bit MSVC and 64 bits
>>> with 64-bit gcc on Linux.
>>>
>>> However, for your purposes std::uint32_t should work fine.
>>
>> Why not
>>
>> #pragma pack(1)
>> struct pixel {
>>     uint8_t red, green, blue, alpha;
>> }
>>
>> ? That would make more sense to me than uint32_t and bit-twiddling to
>> get the channel values.
>>
>> Christian
>
> If you are using "pragma pack" (or gcc "packed" attribute), you are
> probably doing something wrong. It can have some uses when dealing with
> file formats

it is exactly this second case : two programs have to
communicate and store picture files in a compact way (and if
I'll manage to, even exchange such pictures loaded in
memory) upon a single big binary "fread" from disk or save
with a single raw "fwrite" to disk.
I can't find it acceptable that the DISK layout and RAM
layout may differ as to packing aspects.
Non only for the waste of space (a large waste as pictures
could be some tens megapixels), but as in such case I would
have to read/write only single elements at a time.
Unacceptable (*).
I must be allowed to load and save fast and in a single
shot, so 1:1 mapping RAM<=>DISK.
This requirement is underogatable, even if I agree with all
other remarks you make afterwards ...


(*) well actually I do not make strictly a single load/save
as stated. First I save/load a very small descriptor with
WIDTH/HEIGHT stored as 16 bit integers, then, aware of the
array size of the buffer, alloc and fread or fwrite down.

It's a very "stupid" non standard format, but I don't need
to export this directly. When in gambas I save using its
native functions. The C++ engine is just supposed to
transform pictures and save in raw format.

[David Brown]

On 22/11/2019 10:37, Öö Tiib wrote:
> On Friday, 22 November 2019 10:37:59 UTC+2, David Brown wrote:
>>
>> When you use "pragma pack", you are making code that is non-standard,
>> non-portable, and can cause trouble interacting with other code. (What
>> happens when you mix pointers to a packed version of your struct, and a
>> non-packed version? Are they compatible? Will your compiler warn you
>> of mixups? My guess is you don't know, and that is not a good thing.)
>
> Both gcc and clang have -Waddress-of-packed-member but AFAIK you
> won't get that warning when compiling for platform on what it does not
> matter.

I wasn't saying that /I/ don't know the answer to these questions, or
that other more experienced developers don't know. I was saying that
that folks suggesting "pack" here don't know the answers. And that is
what matters - they are recommending a technique that they do not fully
understand. (If they did understand it fully, they would understand
that it is completely unnecessary in this situation.)

For the gcc warnings:

"-Waddress-of-packed-member" warns when you take the address of a packed
member, regardless of alignment and regardless of whether the target
supports unaligned data. It is enabled by default.

"-Wpacked" warns if you have made a struct "packed" but it does not
affect the size or layout of the struct. ("packed" attribute will still
affect the alignment of the struct.) It would trigger for code like the
the packed "pixel" struct, since the "packed" attribute does not affect
the structure.

Another useful warning here is "-Wpadded", which will tell you if a
struct has padding for alignment of members or the struct itself. If
"-Wpadded" is not triggered for your structs, making them "packed" is
not going to be at all useful (unless you need to change their alignment).

>
>> They can lead to inefficiencies - on some compilers and targets, packed
>> fields are accessed by byte. Even when the fields are accessed by their
>> full size, packed means you can get misalignments, which can have a very
>> significant impact on efficiency in some situations.
>
> Huh? Packed structs usually lead to raise of storage efficiency.

No, they don't - not usually, not significantly, and not helpfully.
/Occasionally/ they can help, but more often you can get close to the
same efficiency by simply re-arranging the members more sensibly. (And
if you don't have enough members to make this work, you don't have
enough data for the memory efficiency to be relevant.)

> Say there is struct containing a char and double. Normally it is 16 bytes
> but packed it is 9 bytes. 43.75% of storage saved!
> Where we have to process large array of those there it is often impossible
> to predict without profiling on target platform if that will cause performance
> gain or loss.
>

If memory efficiency or performance here is important, you would be
better off having two arrays - one of doubles, one of chars. Sure, that
would involve a little extra code - but C++ excels in letting you write
wrappers so that you have a good clean interface while keeping the
internal structures intact. And now you have all your doubles aligned,
giving significantly better cache usage and access speeds, and allowing
the possibility of SIMD vector instructions for massive improvements.
And since a structure like this is typically for a "valid, value" pair,
you now have 8 times the efficiency in dealing with invalid data over
the array since you only need to access the big doubles when the data is
valid. And if you only actually needed a single bit (bool) of the char,
you could potentially get another factor of 8 here (depending on the
ratio of valid to invalid values).

Packed /can/ be useful, but very often it is used when it is unnecessary
or directly detrimental, and often there are better ways to deal with
the situation.

[David Brown]

You have a struct with byte data - making it packed is /useless/. There
is /never/ any advantage in that.

If the structure contains data of different sizes, then first check if
there really are padding or alignment issues. Most structures used for
file formats are "naturally aligned" - the people who designed the
structures are smart enough to make sure that every 2-byte data item is
at an even address, every 4-byte data item is at a multiple of 4
address, and so on. There are exceptions, especially for some older
formats from the days of 16-bit cpus, but it is often the case.

And if it is the case here, then making it "packed" is useless at best,
and can easily be a significant disadvantage.

All of what you have written here is "premature optimisation" - it is
knee-jerk thinking "I've got a file structure, I must use packed" rather
than looking at what is actually relevant or helpful. At the moment,
you haven't the faintest clue as to whether using a packed struct makes
a real and measurable difference that is worth the cost - even if we
assume the file format details are not aligned. You haven't done any
timings of real-world usage of your system to see if this is a
bottleneck. You haven't tested alternatives like "mmap", or existing
libraries. You haven't compared the time it takes to load or save the
file with the time it takes to do your processing. You haven't compared
the time it takes to pack or unpack the format to the time it takes to
read or write it from the disk. You haven't compared the time the
processing takes for a "packed" struct to how it would run with an
unpacked struct.

(Yes, I realise I have made a lot of claims here about what you have and
haven't done. From your posts, I think they are almost certainly true.
If I am wrong there, let me know.)


If you want to tell me that using "pack" makes your development process
simpler and the code easier, that's fine - I'll believe that, and it is
a perfectly good reason for using it. But if you try to tell me you are
doing it for run-time speed reasons, then you are wrong. Measure first,
find out if it really is too slow, find your /real/ bottlenecks, /then/
start looking for ways to improve the throughput.

>
>
> (*) well actually I do not make strictly a single load/save as stated.
> First I save/load a very small descriptor with WIDTH/HEIGHT stored as 16
> bit integers, then, aware of the array size of the buffer, alloc and
> fread or fwrite down.
>
> It's a very "stupid" non standard format, but I don't need to export
> this directly. When in gambas I save using its native functions. The C++
> engine is just supposed to transform pictures and save in raw format.
>

So don't make stupid formats. Make /simple/ formats, and non-standard
formats, if that is what is best for the job. But don't make a stupid
format. If the fields in the struct are not properly ("naturally")
aligned, change the format so that they are. That will help the gambas
side, help the C++ side, and mean you don't have to faff around with
"pack" extensions /or/ with extra packing and unpacking stages.

[David Brown]

You are right - I was thinking in C terms, where this /is/ defined
behaviour. For C++, you'd need to put it all in a class and make
appropriate accessor functions. On the plus side, that would let you
make an endian-independent solution, and if it is done well the
compiler's optimiser will eliminate all overhead.

On some compilers, like gcc, the union type-punning is guaranteed to
work in C++ too. But then the code is not portable.

[Soviet_Mario]

Il 22/11/19 12:23, David Brown ha scritto:

sure I have no idea, I put the pragma there just for
"precaution"
what cost is that supposed to have ?

> even if we
> assume the file format details are not aligned. You haven't done any
> timings of real-world usage of your system to see if this is a
> bottleneck. You haven't tested alternatives like "mmap", or existing
> libraries.

I'm writing from scratch almost, but It will be a very
simple library

> You haven't compared the time it takes to load or save the
> file with the time it takes to do your processing.

sure, but I'll have to do both in any case :)
It's a non-sense comparison though
I just thought of
read/write all-at-once vs pixel by pixel (and choose I)

no use in knowing if I spend more time applying
transformations on the loaded picture vs saving or loading
it. They are sequential mandatory operations

> You haven't compared
> the time it takes to pack or unpack the format to the time it takes to

chosen not to pack/unpack anything actually, as I'm hoping
the RAM / disk layout are the very same raw binary.
I was asking help in making sure this was (not if it was
useful or not)

> read or write it from the disk. You haven't compared the time the
> processing takes for a "packed" struct to how it would run with an
> unpacked struct.

forgive me : how to read an all packed file in an array of
UNpacked data of different layout, in a single read
operation ? I can't figure out this

>
> (Yes, I realise I have made a lot of claims here about what you have and
> haven't done. From your posts, I think they are almost certainly true.
> If I am wrong there, let me know.)

no problem, most are right.
Some are irrelevant imho

>
>
> If you want to tell me that using "pack" makes your development process
> simpler and the code easier, that's fine

I don't know. Just searching a way to do binary read/write
in one single shot directly to/from RAM without any conversions.
And have a RAM layout simple enough not to require bitwise
masking to locate fields (the drawback of a pure "bit
fields" solution)

> - I'll believe that, and it is
> a perfectly good reason for using it. But if you try to tell me you are
> doing it for run-time speed reasons, then you are wrong. Measure first,
> find out if it really is too slow, find your /real/ bottlenecks, /then/
> start looking for ways to improve the throughput.

I don't have any running stuff by now, as I'm still in the
design time.
But I could not redesign the BASIC disk and data structure
later, so i MUST make some premature optimization now as it
affects the very base of the pillar.

>
>>
>>
>> (*) well actually I do not make strictly a single load/save as stated.
>> First I save/load a very small descriptor with WIDTH/HEIGHT stored as 16
>> bit integers, then, aware of the array size of the buffer, alloc and
>> fread or fwrite down.
>>
>> It's a very "stupid" non standard format, but I don't need to export
>> this directly. When in gambas I save using its native functions. The C++
>> engine is just supposed to transform pictures and save in raw format.
>>
>
> So don't make stupid formats. Make /simple/ formats,

I meant the same actually.
I store 16 bit width, 16 bit height, and then a compact
rectangular array o 32 bit pixels (8+8+8+8 for 3 color
channels + transparency) per each pixel.

it is is not completely "naturally aligned", but the header
and the "body" are placed apart, in memory, and read/written
in distinct file operations.

So I ask you, when I call NEW operator for an array of
"pixel" structs, is it naturally aligned or not ?

> and non-standard
> formats, if that is what is best for the job. But don't make a stupid
> format.

I called it stupid in place of the simplest as possible.
There are no "tags" included as this is only an "internal"
(and personal) format for intermediate raw data.
It does not seem to be more sophisticated than this.

> If the fields in the struct are not properly ("naturally")
> aligned, change the format so that they are. That will help the gambas
> side, help the C++ side, and mean you don't have to faff around with
> "pack" extensions /or/ with extra packing and unpacking stages.
>
>

[Öö Tiib]

On Friday, 22 November 2019 12:40:21 UTC+2, David Brown wrote:
> On 22/11/2019 10:37, Öö Tiib wrote:
> > On Friday, 22 November 2019 10:37:59 UTC+2, David Brown wrote:
> >
> >> They can lead to inefficiencies - on some compilers and targets, packed
> >> fields are accessed by byte. Even when the fields are accessed by their
> >> full size, packed means you can get misalignments, which can have a very
> >> significant impact on efficiency in some situations.
> >
> > Huh? Packed structs usually lead to raise of storage efficiency.
>
> No, they don't - not usually, not significantly, and not helpfully.
> /Occasionally/ they can help, but more often you can get close to the
> same efficiency by simply re-arranging the members more sensibly. (And
> if you don't have enough members to make this work, you don't have
> enough data for the memory efficiency to be relevant.)

But I did nowhere say that the effect of raised storage efficiency is
guaranteed to be significant or helpful anyhow? I only said that
there usually is that effect. Period. Lack of that effect is less usual.
Careful design and arrangement of data in way that padding is
minimal is effort and so is rarely invested. But what you say about
lot of members clearly contradicts with example that I brought in
next sentence.

> > Say there is struct containing a char and double. Normally it is 16 bytes
> > but packed it is 9 bytes. 43.75% of storage saved!
> > Where we have to process large array of those there it is often impossible
> > to predict without profiling on target platform if that will cause performance
> > gain or loss.
> >
>
> If memory efficiency or performance here is important, you would be
> better off having two arrays - one of doubles, one of chars.

It is third option that will save memory like packing on this example
It is worth to try out as well but unfortunately it is also not a silver
bullet.
It can even sometimes perform worse than packed structs depending
on platform it runs on, on context and what kind of processing is
performed on that data. Trying that alternative out may involve way
more significant refactorings than adding two pragmas.

> would involve a little extra code - but C++ excels in letting you write
> wrappers so that you have a good clean interface while keeping the
> internal structures intact. And now you have all your doubles aligned,
> giving significantly better cache usage and access speeds, and allowing
> the possibility of SIMD vector instructions for massive improvements.

Those are also available only on some (I agree that common) platforms
and usefulness of those assumes that by "processing" we did mean
(I also agree that those are common) sequential for loops.

> And since a structure like this is typically for a "valid, value" pair,
> you now have 8 times the efficiency in dealing with invalid data over
> the array since you only need to access the big doubles when the data is
> valid. And if you only actually needed a single bit (bool) of the char,
> you could potentially get another factor of 8 here (depending on the
> ratio of valid to invalid values).

Validity is more commonly encoded into values of doubles (in typical
64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
so I assumed that the char under question carries more than one bit
of information in it.

> Packed /can/ be useful, but very often it is used when it is unnecessary
> or directly detrimental, and often there are better ways to deal with
> the situation.

Correct. I wrote nowhere something that contradicts with it.
I only said that it is often impossible to predict without profiling on
target platform if the effect of those is positive or negative as
answer to your claim that these can lead only to significant
negative effects. Also it is often cheapest alternative to try
and test what the two pragmas added to code cause.

[David Brown]

I've told you in several posts what the costs are. "Packed" is /not/ a
"precaution", nor is using a struct with a bitfield instead of uint8_t.
If you don't know what you are doing, fine - if I understand you
correctly, you are experienced with gambas programming but new to C++.
So /ask/ (and please read the answers). Don't guess, or grab the first
thing that you find from a google search. And when the more experienced
people in a group like this all say "don't use packed" and "use
uint8_t", then that is what you should do. Ask /why/, and try to learn
- and once you do understand "pack", when it can be useful, when it is
useless, when it is worse than useless, what it does, and what it costs
- /then/ you can decide that because you know the full situation better
than we do, you still want to use it against general advice.

"Packed" is not a precaution - it is a last choice solution when you
have nothing better and know you can't write the code as well without
it. (That might mean you can't write the code as quickly or clearly
without it, which is fair enough - /if/ you are sure it is necessary.)

>
>> even if we
>> assume the file format details are not aligned.  You haven't done any
>> timings of real-world usage of your system to see if this is a
>> bottleneck.  You haven't tested alternatives like "mmap", or existing
>> libraries.
>
> I'm writing from scratch almost, but It will be a very simple library
>
>> You haven't compared the time it takes to load or save the
>> file with the time it takes to do your processing.
>
> sure, but I'll have to do both in any case :)
> It's a non-sense comparison though
> I just thought of
> read/write all-at-once vs pixel by pixel (and choose I)
>
> no use in knowing if I spend more time applying transformations on the
> loaded picture vs saving or loading it. They are sequential mandatory
> operations

Of course there is use in knowing this, if speed is an issue. If the
processing takes 1000 times as long as the loading and saving, then you
know that putting the slightest effort into speed optimisation for file
access is a waste of effort - your time would be better spent trying to
shave 0.1% off the processing time than increasing the speed of the file
access by orders of magnitude.

And if your file format contains non-aligned members, then the simplest
and easiest way to improve and simplify the code, and improve the speed
of several parts, is to fix the file format. Make sure the members are
aligned (add explicit dummy members if necessary) - then forget you ever
heard of "pragma pack".

>
>
>> You haven't compared
>> the time it takes to pack or unpack the format to the time it takes to
>
> chosen not to pack/unpack anything actually, as I'm hoping the RAM /
> disk layout are the very same raw binary.
> I was asking help in making sure this was (not if it was useful or not)

No one can tell you if it is the same without knowing the file format.

There is nothing wrong with having the ram and disk layout the same -
often that is a very convenient and efficient choice. But if the disk
layout is not directly usable in ram, then it often is /not/ a
convenient and efficient choice - and if it /is/ usable directly in ram,
you don't need or want "pack".

However you look at it, "pack" is usually (but not always) the wrong
choice. You should only consider it once you have eliminated better
alternatives.

As for asking for help here, you are getting the help you need even if
it is not exactly what you asked for. It's as though you said "I need a
new toothbrush. Can you tell me how to get to the toy shop?" It is
conceivable that a toy shop might have toothbrushes, but it is very
unlikely to be a good choice. Would you not rather have people say "You
don't want to go to a toy shop. You want to go to a chemist - this is
how you get there" ?

>
>> read or write it from the disk.  You haven't compared the time the
>> processing takes for a "packed" struct to how it would run with an
>> unpacked struct.
>
> forgive me : how to read an all packed file in an array of UNpacked data
> of different layout, in a single read operation ? I can't figure out this
>

First, you take a step back. You look to see if the packed file really
is "packed", in the sense of having unaligned data that would have
padding in a normal struct. I have seen no answer to this (unless it is
later in this post), despite asking this /crucial/ question.

If the format is already good, you are done. One memcpy, or read from
the file directly to a suitable buffer, or memmap to the file directly
(no reading or writing needed).

If the format is bad, the next step is to consider changing the format.
This would be good for all parts of the system. When the format is
changed, you are done. Again, I have had no answer to whether this is
possible.

If the format is bad, and cannot be fixed, then you have to write reader
and writer functions that unpack and pack the data into the right
formats. Reader and writer functions are common for all sorts of
reasons - you already have something to handle starting with the file
header and then allocating memory depending on the size. It is very
normal to have a function that reads in the data format and uses it to
populate an internal format. Perhaps your file format uses 8-bit
unsigned data for colour channels, but your internal format uses
floating point because that suits your processing needs. Perhaps the
file format uses run-length encoding for smaller files, and you want to
expand it to a raw array. There are all sorts of things that go on in
typical reader functions. Making them a "single operation", whatever
that might mean, is rarely relevant - code that uses these functions has
a "single operation" of calling the reader function.

>>
>> (Yes, I realise I have made a lot of claims here about what you have and
>> haven't done.  From your posts, I think they are almost certainly true.
>>   If I am wrong there, let me know.)
>
> no problem, most are right.
> Some are irrelevant imho

Perhaps - but not necessarily the ones you think are irrelevant. IMHO,
of course.

(And to be clear - I did not right this to be rude, or insulting, or
demoralising, and I hope it did not come across that way. I am writing
to push you to think about things that are important here, but which I
feel you are skipping. The aim is that you learn more, and that you can
write better code that is clear, correct and efficient without wasting
effort on poorer techniques.)

>
>>
>>
>> If you want to tell me that using "pack" makes your development process
>> simpler and the code easier, that's fine
>
> I don't know. Just searching a way to do binary read/write in one single
> shot directly to/from RAM without any conversions.

Doing this all "in one single shot" should not be an aim. Doing it in a
way that makes the code simple and clear, easy to develop, easy to read
and easily correct should be your aim. Quite possibly your real goal
would be easiest achieved by using a standard format (like PNG) and
existing libraries for gambas and C++. Call the one function - simple
and clear.

> And have a RAM layout simple enough not to require bitwise masking to
> locate fields (the drawback of a pure "bit fields" solution)
>

Why would you want to bring bitfields into this?

>> - I'll believe that, and it is
>> a perfectly good reason for using it.  But if you try to tell me you are
>> doing it for run-time speed reasons, then you are wrong.  Measure first,
>> find out if it really is too slow, find your /real/ bottlenecks, /then/
>> start looking for ways to improve the throughput.
>
> I don't have any running stuff by now, as I'm still in the design time.
> But I could not redesign the BASIC disk and data structure later, so i
> MUST make some premature optimization now as it affects the very base of
> the pillar.
>

Fair enough. So get it /right/ by designing the right data structure
/now/, rather than picking a bad structure from the start.

>>
>>>
>>>
>>> (*) well actually I do not make strictly a single load/save as stated.
>>> First I save/load a very small descriptor with WIDTH/HEIGHT stored as 16
>>> bit integers, then, aware of the array size of the buffer, alloc and
>>> fread or fwrite down.
>>>
>>> It's a very "stupid" non standard format, but I don't need to export
>>> this directly. When in gambas I save using its native functions. The C++
>>> engine is just supposed to transform pictures and save in raw format.
>>>
>>
>> So don't make stupid formats.  Make /simple/ formats,
>
> I meant the same actually.
> I store 16 bit width, 16 bit height, and then a compact rectangular
> array o 32 bit pixels (8+8+8+8 for 3 color channels + transparency) per
> each pixel.
>
> it is is not completely "naturally aligned", but the header and the
> "body" are placed apart, in memory, and read/written in distinct file
> operations.

So make it "naturally aligned". What will be the cost? A couple of
extra unsused padding bytes, added manually?

struct pixel {
uint8_t red, green, blue, alpha;

};

struct picture_format {
uint64_t key; // Always 0x0064727574636950ull
uint8_t file_type_version;
uint8_t padding1;
uint16_t height;
uint16_t width;
uint16_t padding2;

pixel pixeldata[];
};

Pick an alignment for the structure - 32-bit or 64-bit are good choices.
Start with a member of that size for convenience - I like a constant
marker to make it easy to tell that you are using the right file type.
(The value here is the string "Picture".) Make sure everything is
aligned sensibly. Add padding members manually and explicitly to keep
it clear - then there is never any doubt. (These may later be re-used
for extra information in later file format versions.) Make sure
separate blocks, such as the header and the data section, each are
aligned to the main large alignment for efficiency.

>
> So I ask you, when I call NEW operator for an array of "pixel" structs,
> is it naturally aligned or not ?

Yes, as long as you don't faff about with "packed" stuff. Then you
don't know.

>
>
>> and non-standard
>> formats, if that is what is best for the job.  But don't make a stupid
>> format.
>
> I called it stupid in place of the simplest as possible.
> There are no "tags" included as this is only an "internal" (and
> personal) format for intermediate raw data.
> It does not seem to be more sophisticated than this.

Then it is inexcusable to have a format that needs "packed". That just
means totally pointless complications, inefficiencies, and non-portable
code.

If you have learned nothing else from this conversation, hopefully you
have learned that much.

[David Brown]

On 22/11/2019 13:55, Öö Tiib wrote:
> On Friday, 22 November 2019 12:40:21 UTC+2, David Brown wrote:
>> On 22/11/2019 10:37, Öö Tiib wrote:
>>> On Friday, 22 November 2019 10:37:59 UTC+2, David Brown wrote:
>>>
>>>> They can lead to inefficiencies - on some compilers and targets, packed
>>>> fields are accessed by byte. Even when the fields are accessed by their
>>>> full size, packed means you can get misalignments, which can have a very
>>>> significant impact on efficiency in some situations.
>>>
>>> Huh? Packed structs usually lead to raise of storage efficiency.
>>
>> No, they don't - not usually, not significantly, and not helpfully.
>> /Occasionally/ they can help, but more often you can get close to the
>> same efficiency by simply re-arranging the members more sensibly. (And
>> if you don't have enough members to make this work, you don't have
>> enough data for the memory efficiency to be relevant.)
>
> But I did nowhere say that the effect of raised storage efficiency is
> guaranteed to be significant or helpful anyhow? I only said that
> there usually is that effect. Period. Lack of that effect is less usual.

Fair enough.

> Careful design and arrangement of data in way that padding is
> minimal is effort and so is rarely invested.

I would expect anyone who cares about the storage efficiency to put in
that effort.

> But what you say about
> lot of members clearly contradicts with example that I brought in
> next sentence.

Yes, but see below about that one.

>
>>> Say there is struct containing a char and double. Normally it is 16 bytes
>>> but packed it is 9 bytes. 43.75% of storage saved!
>>> Where we have to process large array of those there it is often impossible
>>> to predict without profiling on target platform if that will cause performance
>>> gain or loss.
>>>
>>
>> If memory efficiency or performance here is important, you would be
>> better off having two arrays - one of doubles, one of chars.
>
> It is third option that will save memory like packing on this example
> It is worth to try out as well but unfortunately it is also not a silver
> bullet.

If programming was full of silver bullet solutions, it would not be
nearly as much fun :-)

> It can even sometimes perform worse than packed structs depending
> on platform it runs on, on context and what kind of processing is
> performed on that data. Trying that alternative out may involve way
> more significant refactorings than adding two pragmas.
>
>> would involve a little extra code - but C++ excels in letting you write
>> wrappers so that you have a good clean interface while keeping the
>> internal structures intact. And now you have all your doubles aligned,
>> giving significantly better cache usage and access speeds, and allowing
>> the possibility of SIMD vector instructions for massive improvements.
>
> Those are also available only on some (I agree that common) platforms
> and usefulness of those assumes that by "processing" we did mean
> (I also agree that those are common) sequential for loops.

Of course. It is only a possibility, not something that is always useful.

>
>> And since a structure like this is typically for a "valid, value" pair,
>> you now have 8 times the efficiency in dealing with invalid data over
>> the array since you only need to access the big doubles when the data is
>> valid. And if you only actually needed a single bit (bool) of the char,
>> you could potentially get another factor of 8 here (depending on the
>> ratio of valid to invalid values).
>
> Validity is more commonly encoded into values of doubles (in typical
> 64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
> so I assumed that the char under question carries more than one bit
> of information in it.

I was thinking of something roughly like a std::optional<double>, but
where you want greater storage efficiency.

Of course there are endless use-cases for structures similar to this,
and many ways to handle it. I am just showing some ideas that can often
be better than throwing "packed" into the mix (while still accepting
that /sometimes/ "packed" is the best choice).

>
>> Packed /can/ be useful, but very often it is used when it is unnecessary
>> or directly detrimental, and often there are better ways to deal with
>> the situation.
>
> Correct. I wrote nowhere something that contradicts with it.
> I only said that it is often impossible to predict without profiling on
> target platform if the effect of those is positive or negative as
> answer to your claim that these can lead only to significant
> negative effects. Also it is often cheapest alternative to try
> and test what the two pragmas added to code cause.
>

It might be that there has been a little mixup about which claims
referred to which uses of "packed" (without saying who has mixed things
up, as it was likely to be me). In general, I agree that "packed" may
have positive effects, negative effects, or no effects, and any effects
may be significant or insignificant. I say there are usually, but not
always, alternatives that could do better. But I agree that "packed" is
often an easy choice from the development viewpoint.

However, I would say it is almost always a bad choice to start off with
a design that /requires/ "packed" - the OP has no reason at all for
doing that.

The case where I said there could never be any positive effects, but
might be negative effects, was a poster who suggested "packing" a struct
consisting of 4 uint8_t members. There, I think we can agree, it could
never lead to increased storage or runtime efficiency on any real
platform. (There is, I believe, a very hypothetical possibility of the
struct having extra padding.) And it is certainly possible that on some
platforms with some compilers, there could be negative effects.

[Scott Lurndal]

=?UTF-8?B?w5bDtiBUaWli?= <oot...@hot.ee> writes:
>On Friday, 22 November 2019 10:37:59 UTC+2, David Brown wrote:
>>
>> When you use "pragma pack", you are making code that is non-standard,
>> non-portable, and can cause trouble interacting with other code. (What
>> happens when you mix pointers to a packed version of your struct, and a
>> non-packed version? Are they compatible? Will your compiler warn you
>> of mixups? My guess is you don't know, and that is not a good thing.)
>
>Both gcc and clang have -Waddress-of-packed-member but AFAIK you
>won't get that warning when compiling for platform on what it does not
>matter.
>
>> They can lead to inefficiencies - on some compilers and targets, packed
>> fields are accessed by byte. Even when the fields are accessed by their
>> full size, packed means you can get misalignments, which can have a very
>> significant impact on efficiency in some situations.
>
>Huh? Packed structs usually lead to raise of storage efficiency.
>Say there is struct containing a char and double. Normally it is 16 bytes
>but packed it is 9 bytes. 43.75% of storage saved!

Of course, accesses become less efficient (particularly if the
unaligned accesses trap to the kernel for resolution).

[Öö Tiib]

On Friday, 22 November 2019 16:18:15 UTC+2, David Brown wrote:
> On 22/11/2019 13:55, Öö Tiib wrote:
> > On Friday, 22 November 2019 12:40:21 UTC+2, David Brown wrote:
> >
> > Validity is more commonly encoded into values of doubles (in typical
> > 64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
> > so I assumed that the char under question carries more than one bit
> > of information in it.
>
> I was thinking of something roughly like a std::optional<double>, but
> where you want greater storage efficiency.

There we slightly mixed up in our thinkings ... I myself imagined
something like array of "quantities" where double is value and char
selects its magnitude (or physical unit) from limited set of such.

> Of course there are endless use-cases for structures similar to this,
> and many ways to handle it. I am just showing some ideas that can often
> be better than throwing "packed" into the mix (while still accepting
> that /sometimes/ "packed" is the best choice).

Yes, and splitting struct between different arrays can be OK for
storage optimization. But it can also increase cache misses
because locality of members of object is now gone ... needs to
be profiled.

> >> Packed /can/ be useful, but very often it is used when it is unnecessary
> >> or directly detrimental, and often there are better ways to deal with
> >> the situation.
> >
> > Correct. I wrote nowhere something that contradicts with it.
> > I only said that it is often impossible to predict without profiling on
> > target platform if the effect of those is positive or negative as
> > answer to your claim that these can lead only to significant
> > negative effects. Also it is often cheapest alternative to try
> > and test what the two pragmas added to code cause.
> >
>
> It might be that there has been a little mixup about which claims
> referred to which uses of "packed" (without saying who has mixed things
> up, as it was likely to be me). In general, I agree that "packed" may
> have positive effects, negative effects, or no effects, and any effects
> may be significant or insignificant. I say there are usually, but not
> always, alternatives that could do better. But I agree that "packed" is
> often an easy choice from the development viewpoint.
>
> However, I would say it is almost always a bad choice to start off with
> a design that /requires/ "packed" - the OP has no reason at all for
> doing that.

I also replied to OP that his unsigned bitfields in anonymous unions in
packed structs can only harm compared to good old uint8_t.

> The case where I said there could never be any positive effects, but
> might be negative effects, was a poster who suggested "packing" a struct
> consisting of 4 uint8_t members. There, I think we can agree, it could
> never lead to increased storage or runtime efficiency on any real
> platform. (There is, I believe, a very hypothetical possibility of the
> struct having extra padding.) And it is certainly possible that on some
> platforms with some compilers, there could be negative effects.

True. Benefits are only fictional but real bad effects
can emerge in practice. Such code looks confused and so
experienced reader becomes careful ... hmm I bet here
will be major crocodiles somewhere. ;)

[David Brown]

On 22/11/2019 16:23, Öö Tiib wrote:
> On Friday, 22 November 2019 16:18:15 UTC+2, David Brown wrote:
>> On 22/11/2019 13:55, Öö Tiib wrote:
>>> On Friday, 22 November 2019 12:40:21 UTC+2, David Brown wrote:
>>>
>>> Validity is more commonly encoded into values of doubles (in typical
>>> 64-bit IEEE 754 doubles there is huge supply of NaNs plus two Infs)
>>> so I assumed that the char under question carries more than one bit
>>> of information in it.
>>
>> I was thinking of something roughly like a std::optional<double>, but
>> where you want greater storage efficiency.
>
> There we slightly mixed up in our thinkings ... I myself imagined
> something like array of "quantities" where double is value and char
> selects its magnitude (or physical unit) from limited set of such.

OK. The gains from having these as separate arrays (rather than packed
structs) might not be as much, but they could still be relevant.

>
>> Of course there are endless use-cases for structures similar to this,
>> and many ways to handle it. I am just showing some ideas that can often
>> be better than throwing "packed" into the mix (while still accepting
>> that /sometimes/ "packed" is the best choice).
>
> Yes, and splitting struct between different arrays can be OK for
> storage optimization. But it can also increase cache misses
> because locality of members of object is now gone ... needs to
> be profiled.

You still have as much locality, especially when iterating over the
arrays, just split into two blocks. Random access might be worse - you
will always have to pull in two cache lines, rather than only sometimes
needing to do so. (But if you are doing mainly random access and speed
is important, unpacked structs will be best.) Misalignment penalties
for the packed struct can vary from nothing to a great deal, depending
on the target platform.

Yes - you need to profile and test.

[James Kuyper]

On 11/22/19 4:10 AM, David Brown wrote:
> On 22/11/2019 09:37, Keith Thompson wrote:

...

>> Or an array wrapped in a struct to allow assignment. (An array of 4
>> uint8_t elements is guaranteed to be 32 bits, but a struct containing
>> such an array is not -- but it's a nearly safe assumption that there
>> will be no padding between the elements or at the end.)
>>
>
> While it is true in theory that a struct can have extra padding at the
> end (beyond what is needed for an array of the struct to have proper
> alignment for all members), are there any compilers that have done so in
> practice?

I can't name names, but my understanding is that there have been (and
might still be) implementations where all struct types have the same
alignment requirement as the most strictly aligned type supported by
that implementation. If that alignment requirement is greater than the
requirements of any of the member types, that will sometimes require
padding at the end of the struct.

[James Kuyper]

I said that wrong. Unless all members of the struct have the maximum
alignment requirement, it's possible that such an implementation would
have to put padding at the end.

[James Kuyper]

On 11/22/19 4:37 AM, Öö Tiib wrote:
> On Friday, 22 November 2019 10:37:59 UTC+2, David Brown wrote:

...

>> They can lead to inefficiencies - on some compilers and targets, packed
>> fields are accessed by byte. Even when the fields are accessed by their
>> full size, packed means you can get misalignments, which can have a very
>> significant impact on efficiency in some situations.
>
> Huh? Packed structs usually lead to raise of storage efficiency.

He said efficiency, not "storage efficiency". If there's any difference
between the packed and unpacked version of a struct, the packed version
usually requires slower access. If that weren't the case, there'd be no
motive for the implementation to make the unpacked version different.

[Manfred]

It is defined behavior in C++ too, but you have to pay extra care
(compared to C) in order to ensure that only the active member is accessed.

[David Brown]

I was assuming Öö meant that type punning via the union - accessing the
same data via the different union members - was undefined behaviour (as
it is, in C++). Using the union while only accessing the active member
is of course well defined - otherwise the union concept would be pretty
useless!

But it would have been worth making that point more clearly.

[Keith Thompson]

Soviet_Mario <Sovie...@CCCP.MIR> writes:
> Il 22/11/19 09:37, Keith Thompson ha scritto:
>> Christian Gollwitzer <auri...@gmx.de> writes:
>>> Am 21.11.19 um 17:20 schrieb Paavo Helde:
>> [...]
>>>> However, for your purposes std::uint32_t should work fine.
>>>
>>> Why not
>>>
>>> #pragma pack(1)
>>> struct pixel {
>>> uint8_t red, green, blue, alpha;
>>> }
>>>
>>> ? That would make more sense to me than uint32_t and bit-twiddling to
>>> get the channel values.
>>
>> Because #pragma pack is non-standard?
>>
>> If you only care about implementations that support it as an extension,
>> that's fine, but the OP seemed very concerned about portability.
>>
>> An array of uint8_t could also work:
>>
>> typedef uint8_t pixel[4];
>
> intresting idea !!!
>
>> enum { red, green, blue, alpha };
>
> btw since here #pragma pack seems supported, I find the
> "dot" member selector not only clearer (more readable) but
> more amenable to "intellisense" when typing (type less, no
> typos as autocompletion pops up the members).

If you have a struct consisting of 4 uint8_t members, it's very
likely that you don't *need* #pragma pack. There might be some
odd compiler out there that allocates more than 32 bits for such
a struct -- but that odd compiler might not support #pragma pack.

struct pixel {
uint8_t red;
uint8_t green;
uint8_t blue;
uint8_t alpha;
};

If you need to write 100% portable code, that might not be good
enough -- but you very likely don't need to write 100% portable code.
(And if you're willing to consider #pragma pack, it seems you
already know that.)

My point in suggesting an array was simply to approach that 100%
portability, but I don't think it's necessary.

[...]

[Soviet_Mario]

Il 22/11/19 15:04, David Brown ha scritto:

put is alas irregular verb : I meant its PAST form actually.
I have abandoned the unsigned Bit_Field : 8 solution as soon
as you (a lot of people) stated that u_int_8 MUST have
exactly 8 bits.

In that case either pragma is USELESS (and hence does not
produce any reasonable effect, neither slowness), either is
NECESSARY (and so the slowing effect is a unavoidable price
that I MUST pay either)

> "Packed" is /not/ a
> "precaution", nor is using a struct with a bitfield instead of uint8_t.

the second part has been abandoned early

> If you don't know what you are doing

I'm rather weak in higher level C++ constructs, and I know
that, but not particularly so in low level (some couple of
tens of years ago I used to stuff C code with inline "asm"
statements or even __emit() calls).

>, fine - if I understand you
> correctly, you are experienced with gambas

no, I am rather new to gambas actually.
I was used to VB (Visual Studio, but not even recently).
Going to Gambas has been actually a pain (not saying it is
bad, absolutely not : it's me that knows much less than .net
framework). On QT I am even more newby.

> programming but new to C++.

well, I am more of outdated than new, I had used C++ (and C)
a lot in 1995-2000 roughly, then less and less afterwards.
But strangely enough, I find it more easy to think in C++
terms when I met problems in basic.

> So /ask/ (and please read the answers). Don't guess, or grab the first
> thing that you find from a google search.

but I DID follow the answers :)

> And when the more experienced
> people in a group like this all say "don't use packed" and "use
> uint8_t",

I didn't read or desume the two things should be mutually
exclusive.

Anyway, there are three scenarios

1) added pragma (to the four u_int_8 members in a struct) is
USELESS, as the struct was just packed natively (and hence
does not produce any reasonable effect, neither slowness),
either
2) is NECESSARY, as the struct would not have been natively
packed otherwise (and so the slowing effect is a unavoidable
price that I MUST pay either)
either
3) it is NOT SUFFICIENT (in this case, luckily unlikely,
nothing would have worked fine at all)

as for ALIGNMENT problem, I should not meet any I guess.
The packed (natively or forcefully) struct is 32 bit wide,
so allocating an array should allow nice addressing of any
item even on a 64 bit OS
And no item would cross critical addressing boundary

> then that is what you should do. Ask /why/, and try to learn
> - and once you do understand "pack", when it can be useful, when it is
> useless, when it is worse than useless,

If you have some further patience, tell me if the 3
scenarios fit the present situation (not a more general one,
but just this)

> what it does, and what it costs
> - /then/ you can decide that because you know the full situation better
> than we do, you still want to use it against general advice.

I had given all details sufficient : the data layout is what
I said, so no obscure aspects

>
> "Packed" is not a precaution -

I call it a precaution as it reduces the chances of a non
packed data layout. I was not saying it would be successful
necessarily in every scenario (even sizes sequence is
relevant, and I know this, but my members were all 1 byte long)

> it is a last choice solution when you
> have nothing better and know you can't write the code as well without
> it. (That might mean you can't write the code as quickly or clearly
> without it, which is fair enough - /if/ you are sure it is necessary.)

let's say it explicitely

if I have
union
{
struct
{
u_int_8 Red, Green, Blue, Alpha;
};
u_int_32 AllChannels;
};

without PRAGMA, do I have any explicit or implicit warranty
that my union be 32 bit long ? If so, PRAGMA is useless. If
not, it seems it might be useful.
what damages could PRAGMA PACK produce ?
If the union would be just packed natively, I guess none.
If it wasn't, at best it would pack it (and maybe slow it,
but this is irrelevant, I need precise disk layout). At
worsk it would be insufficient to actually pack it, so my
problem is unsolved.
In this last worst case, are there any other means of
getting the thing to work ?
What are them ?

>
>>
>>> even if we
>>> assume the file format details are not aligned.  You haven't done any
>>> timings of real-world usage of your system to see if this is a
>>> bottleneck.  You haven't tested alternatives like "mmap", or existing
>>> libraries.
>>
>> I'm writing from scratch almost, but It will be a very simple library
>>
>>> You haven't compared the time it takes to load or save the
>>> file with the time it takes to do your processing.
>>
>> sure, but I'll have to do both in any case :)
>> It's a non-sense comparison though
>> I just thought of
>> read/write all-at-once vs pixel by pixel (and choose I)
>>
>> no use in knowing if I spend more time applying transformations on the
>> loaded picture vs saving or loading it. They are sequential mandatory
>> operations
>
> Of course there is use in knowing this, if speed is an issue. If the
> processing takes 1000 times as long as the loading and saving,

this could be true only if load/store is "optimized" like I
stated. If it was done an item at a time it would become
very very slow (and depending on smartness of OS caching
anyway, but not even so much even then, as I read/write the
whole file, which can be 4-12 MB long and one time only, so
caching cannot be as useful as when reusing a small region
many times).

> then you
> know that putting the slightest effort into speed optimisation for file
> access is a waste of effort

when I made some elaboration all in gambas the times were
"mixed" : some basic transformations with few or no floating
point calculations went actually FASTER than disk
read/write, with some more f.p. intensive calculations disk
load/save were faster.
But in any case, the same "order of magnitude", not
100-1000-etc times slower or faster.

> - your time would be better spent trying to
> shave 0.1% off the processing time than increasing the speed of the file
> access by orders of magnitude.
>
> And if your file format contains non-aligned members,

why making general guess when I reported precisely the layout ?
I said :
16 bit (width) - 16 bit (height) - N x 32 bit (R,G,B,alpha)
where N = width x height = number of "pixels"

every pixel is multiple of 32 address on file (but this
seems irrelevant to me).
In the ram layout,
the BMP descriptor has width and height as "local" members,
and a pointer to an array of struct aligned in memory by the
NEW [] operator who is aware of alignment requirements.

Where are the possible problems ?
What problems are affected by a pragma pack directive ?

> then the simplest
> and easiest way to improve and simplify the code, and improve the speed
> of several parts, is to fix the file format.

that point is unmodificable. I strictly need full color and
no compression. The alpha channel is maybe a bit overkill
(255 level of transparency is more than needed ... but
taking less is useless or impossible, and 1 byte is natural
for the otherwise 3,sth bytes long struct)

> Make sure the members are
> aligned (add explicit dummy members if necessary)

since when reading/writing files suffer from this ? Files
are BYTE oriented internally (and the OS does caching
independently and transparently, basing on disk cluster
sizes but this is not sth I should care of)

as for memory : I instance the BMP array invoking
ptrBMP = new clsMyStructColor [TotSize];
and I expect the structs are aligned correctly as NEW knows
its task.

> - then forget you ever
> heard of "pragma pack".
>
>>
>>
>>> You haven't compared
>>> the time it takes to pack or unpack the format to the time it takes to
>>
>> chosen not to pack/unpack anything actually, as I'm hoping the RAM /
>> disk layout are the very same raw binary.
>> I was asking help in making sure this was (not if it was useful or not)
>
> No one can tell you if it is the same without knowing the file format.

I just said it : the stupidest format of all

16 bit (width) - 16 bit (height) - N x 32 bit (R,G,B,alpha)
where N = width x height = number of "pixels"

>
> There is nothing wrong with having the ram and disk layout the same -

oh happy to know ! :)

> often that is a very convenient and efficient choice. But if the disk
> layout is not directly usable in ram, then it often is /not/ a
> convenient and efficient choice - and if it /is/ usable directly in ram,
> you don't need or want "pack".

I was asking about the specific case

>
> However you look at it, "pack" is usually (but not always) the wrong
> choice. You should only consider it once you have eliminated better
> alternatives.

for a starting, I tried to use a 32 bit total size, hoping
it would have been packed regardless of any "hints" (pragma).

>
> As for asking for help here, you are getting the help you need even if
> it is not exactly what you asked for.

not a problem, every advice is appreciated

> It's as though you said "I need a
> new toothbrush. Can you tell me how to get to the toy shop?" It is
> conceivable that a toy shop might have toothbrushes, but it is very
> unlikely to be a good choice. Would you not rather have people say "You
> don't want to go to a toy shop. You want to go to a chemist - this is
> how you get there" ?
>
>>
>>> read or write it from the disk.  You haven't compared the time the
>>> processing takes for a "packed" struct to how it would run with an
>>> unpacked struct.
>>
>> forgive me : how to read an all packed file in an array of UNpacked data
>> of different layout, in a single read operation ? I can't figure out this
>>
>
> First, you take a step back. You look to see if the packed file really
> is "packed",

it is, as both Gambas supports BYTE format (And sizeof())
and BYTE-oriented streams. No overspace is generated. Then I
have to worry about the C++ back end, to comply with this fact.

I think the designer of Gambas relies very strongly upon C
libraries and mind-shape, apart from strings (where he had
to welcome BASIC users who want strings the basic way).

> in the sense of having unaligned data that would have
> padding in a normal struct. I have seen no answer to this (unless it is
> later in this post), despite asking this /crucial/ question.

the files are packed.

>
> If the format is already good, you are done. One memcpy, or read from
> the file directly to a suitable buffer, or memmap to the file directly
> (no reading or writing needed).

memmap is a problem in that I don't know how exactly works
and whether or not I'll be able to interoperate with memory
files opened in gambas

I am also considering to use some memory files supported
directly by the OS (here on linux files created on /shm
virtual folder are designed for such FAST temporaries to be
shared, but with portable stream syntax).
Using /shm would imply no specialized load/store code, just
the normal file access => delegating the ram usage to the OS.

>
> If the format is bad, the next step is to consider changing the format.

just observation.
So, the format is wrong ?

> This would be good for all parts of the system. When the format is
> changed, you are done. Again, I have had no answer to whether this is
> possible.
>
> If the format is bad, and cannot be fixed, then you have to write reader
> and writer functions that unpack and pack the data into the right
> formats. Reader and writer functions are common for all sorts of

well, I'd be considering these for small size, complex
structured input, not for a 4-12 MB size of simple pixels.
It would be a blood bathe

> reasons - you already have something to handle starting with the file
> header and then allocating memory depending on the size. It is very
> normal to have a function that reads in the data format and uses it to
> populate an internal format.

in fact, minimally, I do this. First create the container,
with just a couple of members, then depending on their
value, size the buffer and allocate it, then load the file
in it.

> Perhaps your file format uses 8-bit
> unsigned data for colour channels, but your internal format uses
> floating point because that suits your processing needs.

On certain routines I use floating point, but only
temporarily, as I never STORE floating point pixels. I don't
need a huge precision, as operations are not supposed to be
applied multiple times on the same picture, creating
rounding additive errors.

> Perhaps the
> file format uses run-length encoding for smaller files, and you want to
> expand it to a raw array. There are all sorts of things that go on in
> typical reader functions. Making them a "single operation", whatever
> that might mean,

sth encoded in a single call to fread / fwrite, at least for
the pixel data block. The header is distinct

> is rarely relevant - code that uses these functions has
> a "single operation" of calling the reader function.

well, fread/fwrite may maybe internally operate differently
than expected, but in this case I would not be able to do
better I guess :)

>
>
>>>
>>> (Yes, I realise I have made a lot of claims here about what you have and
>>> haven't done.  From your posts, I think they are almost certainly true.
>>>   If I am wrong there, let me know.)
>>
>> no problem, most are right.
>> Some are irrelevant imho
>
> Perhaps - but not necessarily the ones you think are irrelevant. IMHO,
> of course.
>
> (And to be clear - I did not right this to be rude, or insulting, or
> demoralising, and I hope it did not come across that way.

no problem, really

> I am writing
> to push you to think about things that are important here, but which I
> feel you are skipping. The aim is that you learn more, and that you can
> write better code that is clear, correct and efficient without wasting
> effort on poorer techniques.)
>
>>
>>>
>>>
>>> If you want to tell me that using "pack" makes your development process
>>> simpler and the code easier, that's fine
>>
>> I don't know. Just searching a way to do binary read/write in one single
>> shot directly to/from RAM without any conversions.
>
> Doing this all "in one single shot" should not be an aim. Doing it in a

I wish to both get fast disk I/O but also limit the
try/catch trapping. I have the habit to catch or check errno
on every disk access. A single huge read/write makes the
code thinner than splitting the accesses in many place and
having to multiplicate the error condition testing.

> way that makes the code simple and clear, easy to develop, easy to read
> and easily correct should be your aim. Quite possibly your real goal
> would be easiest achieved by using a standard format (like PNG) and
> existing libraries for gambas and C++. Call the one function - simple
> and clear.
>
>> And have a RAM layout simple enough not to require bitwise masking to
>> locate fields (the drawback of a pure "bit fields" solution)
>>
>
> Why would you want to bring bitfields into this?
>
>>> - I'll believe that, and it is
>>> a perfectly good reason for using it.  But if you try to tell me you are
>>> doing it for run-time speed reasons, then you are wrong.  Measure first,
>>> find out if it really is too slow, find your /real/ bottlenecks, /then/
>>> start looking for ways to improve the throughput.
>>
>> I don't have any running stuff by now, as I'm still in the design time.
>> But I could not redesign the BASIC disk and data structure later, so i
>> MUST make some premature optimization now as it affects the very base of
>> the pillar.
>>
>
> Fair enough. So get it /right/ by designing the right data structure
> /now/, rather than picking a bad structure from the start.

please give your judgement about the struct I have described
before, which drawbacks seems to have or don't have.

I have stripped everything non necessary as this raw format
is only a temporary. When I actually store files, I export
them as PNG of JPG or GIF by Gambas code.
So my internal format was really minimal.

> Make sure everything is
> aligned sensibly. Add padding members manually and explicitly to keep
> it clear - then there is never any doubt. (These may later be re-used
> for extra information in later file format versions.) Make sure
> separate blocks, such as the header and the data section, each are
> aligned to the main large alignment for efficiency.
>
>>
>> So I ask you, when I call NEW operator for an array of "pixel" structs,
>> is it naturally aligned or not ?
>
> Yes, as long as you don't faff about with "packed" stuff. Then you
> don't know.
>
>>
>>
>>> and non-standard
>>> formats, if that is what is best for the job.  But don't make a stupid
>>> format.
>>
>> I called it stupid in place of the simplest as possible.
>> There are no "tags" included as this is only an "internal" (and
>> personal) format for intermediate raw data.
>> It does not seem to be more sophisticated than this.
>
> Then it is inexcusable to have a format that needs "packed". That just

oh I completely disagree with this point ! The format has to
be packed as the C++ library receives the file from an other
source of data that offers them in that form, so it's mandatory.
And also the caller expects to get the transformed data back
in the same packed form.

> means totally pointless complications, inefficiencies, and non-portable
> code.

nope, I have constraints to fullfil. Maybe inefficient,
maybe non portable (who cares ?), but surely not pointless,
as otherwise data exchange would not work at all

>
> If you have learned nothing else from this conversation, hopefully you
> have learned that much.
>
>>
>>> If the fields in the struct are not properly ("naturally")
>>> aligned, change the format so that they are.  That will help the gambas
>>> side, help the C++ side, and mean you don't have to faff around with
>>> "pack" extensions /or/ with extra packing and unpacking stages.
>>>
>>>
>>
>>
>

[James Kuyper]

On 11/22/19 1:35 PM, Soviet_Mario wrote:
> Il 22/11/19 15:04, David Brown ha scritto:

...

> I have abandoned the unsigned Bit_Field : 8 solution as soon
> as you (a lot of people) stated that u_int_8 MUST have

uint8_t

> exactly 8 bits.

It must have exactly 8 bits, if it exists. uint8_t is optional. And on
any C99+ implementation that fails to support uint8_t, and on any C
platform where unsigned char doesn't have 8 bits, you can't do what you
want to do anyway.

> In that case either pragma is USELESS (and hence does not
> produce any reasonable effect, neither slowness), either is
> NECESSARY (and so the slowing effect is a unavoidable price
> that I MUST pay either)

Correct - it is both useless and unnecessary - in this context. Don't
extrapolate from this case to the general case. It is very specifically
the fact that you are trying to pack something that cannot possibly be
bigger than a byte, and therefore cannot have an alignment requirement
bigger than 1, that renders the pragma useless and unnecessary.

>> If you don't know what you are doing
>
> I'm rather weak in higher level C++ constructs, and I know
> that, but not particularly so in low level (some couple of
> tens of years ago I used to stuff C code with inline "asm"
> statements or even __emit() calls).

You don't appear to understand why the fact that you're trying to use an
8-bit type guarantees that #pragma pack will be useless and unnecessary
- so there's some low-level stuff you don't understand yet.

>> And when the more experienced
>> people in a group like this all say "don't use packed" and "use
>> uint8_t",
>
> I didn't read or desume the two things should be mutually
> exclusive.
>
> Anyway, there are three scenarios
>
> 1) added pragma (to the four u_int_8 members in a struct) is
> USELESS, as the struct was just packed natively (and hence
> does not produce any reasonable effect, neither slowness),
> either
> 2) is NECESSARY, as the struct would not have been natively
> packed otherwise (and so the slowing effect is a unavoidable
> price that I MUST pay either)
> either
> 3) it is NOT SUFFICIENT (in this case, luckily unlikely,
> nothing would have worked fine at all)

On implementations where CHAR_BIT==8, scenario #1 applies. On
implementations where CHAR_BIT>8, scenarios #1 and #3 both apply.

...
> u_int_32 AllChannels;

uint_32

>> in the sense of having unaligned data that would have
>> padding in a normal struct. I have seen no answer to this (unless it is
>> later in this post), despite asking this /crucial/ question.
>
> the files are packed.

OK - which fields are positioned differently from the way they would be
positioned if not packed?

[Soviet_Mario]

Il 22/11/19 20:11, James Kuyper ha scritto:

> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>> Il 22/11/19 15:04, David Brown ha scritto:
> ...
>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>> as you (a lot of people) stated that u_int_8 MUST have
>
> uint8_t

here the macro or typedef is

u_int8_t actually

>
>> exactly 8 bits.
>
> It must have exactly 8 bits, if it exists. uint8_t is optional. And on
> any C99+ implementation that fails to support uint8_t, and on any C
> platform where unsigned char doesn't have 8 bits, you can't do what you
> want to do anyway.

ok, that's a fact

>
>> In that case either pragma is USELESS (and hence does not
>> produce any reasonable effect, neither slowness), either is
>> NECESSARY (and so the slowing effect is a unavoidable price
>> that I MUST pay either)
>
> Correct - it is both useless and unnecessary - in this context. Don't
> extrapolate from this case to the general case. It is very specifically
> the fact that you are trying to pack something that cannot possibly be
> bigger than a byte, and therefore cannot have an alignment requirement
> bigger than 1, that renders the pragma useless and unnecessary.

when I say "in that case" I am not extrapolating, I am
contextualizing to the specific case

maybe QT headers decorate differently

here it is u_int32_t

>
>
>>> in the sense of having unaligned data that would have
>>> padding in a normal struct. I have seen no answer to this (unless it is
>>> later in this post), despite asking this /crucial/ question.
>>
>> the files are packed.
>
> OK - which fields are positioned differently from the way they would be
> positioned if not packed?

as I said, I do not have a working complete program yet, and
I hope to get to a scenario where this wrong situation is
avoided at all

[James Kuyper]

On 11/22/19 6:25 PM, Soviet_Mario wrote:
> Il 22/11/19 20:11, James Kuyper ha scritto:
>> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>>> Il 22/11/19 15:04, David Brown ha scritto:
>> ...
>>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>>> as you (a lot of people) stated that u_int_8 MUST have
>>
>> uint8_t
>
> here the macro or typedef is
>
> u_int8_t actually

Everyone who's been talking with you about this subject has been talking
about std::uint8_t from <cstdint>. If you're talking about something
else, you need to identify what it is and where is it is declared.

In gcc, and in many other compilers as well, there's a command line
option (-E in gcc) that produces output after pre-processing, but before
any other of the translation phases, but with #line directives inserted
to make sure that if you compile the output from -E, any resulting error
messages will refer to the correct file.
Compile one of the modules that uses u_int8_t using -E. Search the
output for the first occurrence of u_int8_t, which should be a typedef
declaration. What file is referred to by the last #line directive
preceding that declaration?

...
>>> u_int_32 AllChannels;
>>
>> uint_32
>
> maybe QT headers decorate differently

A search of the QT online reference manual for u_int8_t turns up nothing:
<https://doc.qt.io/qt-5/search-results.html?q=%22u_int8_t%22>

Nor do I get anything from u_int_32:
<https://doc.qt.io/qt-5/search-results.html?q=%22u_int_32%22>

Whatever is the source of this typedef, I don't think it's QT.

>>>> in the sense of having unaligned data that would have
>>>> padding in a normal struct. I have seen no answer to this (unless it is
>>>> later in this post), despite asking this /crucial/ question.
>>>
>>> the files are packed.
>>
>> OK - which fields are positioned differently from the way they would be
>> positioned if not packed?
>
> as I said, I do not have a working complete program yet, and
> I hope to get to a scenario where this wrong situation is
> avoided at all

I think you can avoid it perfectly without doing anything at all about
it, because it can't occur. I recommend not wasting any more time
worrying about it until you can come up with a plausible reason for
doing so.

[Keith Thompson]

Soviet_Mario <Sovie...@CCCP.MIR> writes:
[...]

> let's say it explicitely
>
> if I have
> union
> {
> struct
> {
> u_int_8 Red, Green, Blue, Alpha;
> };
> u_int_32 AllChannels;
> };
>
> without PRAGMA, do I have any explicit or implicit warranty
> that my union be 32 bit long ? If so, PRAGMA is useless. If
> not, it seems it might be useful.

Fixing the spelling of uintN_t (n=8,32) and adding a tag to the union so
you can use it to define objects:

union u {
struct {
uint8_t Red, Green, Blue, Alpha;
};
uint32_t AllChannels;
};

(You have an anonymous struct, which is standard but wasn't added until
C11, so older compilers might not support it, or might not support it
by default.)

You're guaranteed that the Red and AllChannels members start at the same
location, the very beginning of the union. You're guaranteed that Red,
Green, Blue, and Alpha are at increasing offsets in the order you
defined them.

It's very very likely that the struct and the union will both be
exactly 32 bits. A compiler is not prohibited from adding padding
between members or after the last one, but there's unlikely to be
any good reason to do so in this case. The most likely way this
could fail is if a compiler insisted on making the structure 64 bits.

You can easily add a compile-time or run-time assertion that
sizeof (struct u) == 4 && CHAR_BIT == 8
so your program won't compile, or at least won't run, in the
unlikely event that it's used with an implementation that doesn't
meet your requirements.

There is *no* guarantee about which of the four uint8_t members
correspond to which part of AllChannels. Google "endianness" if
you're not familiar with this. It's very common for Red to overlap
either the high-order byte or the low-order byte of AllChannels.
The other 22 permutations are rare or nonexistent, but Google
"PDP-11 middle-endian" for an odd case.

If you're doing bitwise operations on the entire 32-bit word without
caring about the order of the bytes, then that might not be an issue.

Attempting to use #pragma pack is very likely to do nothing *at best*.

[Soviet_Mario]

Il 23/11/19 05:56, James Kuyper ha scritto:

> On 11/22/19 6:25 PM, Soviet_Mario wrote:
>> Il 22/11/19 20:11, James Kuyper ha scritto:
>>> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>>>> Il 22/11/19 15:04, David Brown ha scritto:
>>> ...
>>>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>>>> as you (a lot of people) stated that u_int_8 MUST have
>>>
>>> uint8_t
>>
>> here the macro or typedef is
>>
>> u_int8_t actually
>
> Everyone who's been talking with you about this subject has been talking
> about std::uint8_t from <cstdint>. If you're talking about something
> else, you need to identify what it is and where is it is declared.

I'm using QT who seems to like to replicate anything its own
way even when avoiding specific Q-prefixes.

so, HERE, including
#include <sys/types.h>

I find this type (everyone might guess it is equivalent to
the other), u_int8_t, which finally stems from a typedef to
UNSIGNED CHAR.

>
> In gcc, and in many other compilers as well, there's a command line
> option (-E in gcc) that produces output after pre-processing, but before
> any other of the translation phases, but with #line directives inserted
> to make sure that if you compile the output from -E, any resulting error
> messages will refer to the correct file.

I'm not so confident with QT environment to manually set
options to the compiler as yet, I just use the ide-forms to
set or unset some warning levels (I keep enabled almost all
even if the outcome is very verbose)

> Compile one of the modules that uses u_int8_t using -E. Search the
> output for the first occurrence of u_int8_t, which should be a typedef
> declaration.

it is an unsigned char, I saw it DIRECTLY opening the
included header file (sys/types.h)

this type is also replicated in another header
<stdint.h> actually, ending up always in unsigned char

> What file is referred to by the last #line directive
> preceding that declaration?
>
> ...
>>>> u_int_32 AllChannels;
>>>
>>> uint_32
>>
>> maybe QT headers decorate differently
>
> A search of the QT online reference manual for u_int8_t turns up nothing:
> <https://doc.qt.io/qt-5/search-results.html?q=%22u_int8_t%22>
>
> Nor do I get anything from u_int_32:
> <https://doc.qt.io/qt-5/search-results.html?q=%22u_int_32%22>

sorry, but not an issue.

>
> Whatever is the source of this typedef, I don't think it's QT.

omg ... are you saying I am just lieing for the fun of it ???
what the heck am I supposed to know about which kind of
headers has included this install ?
I am using the FREE version from a .DEB package (which in
turn launched an updated and installed tons of "kits" by
itself").
I just opened the headers and never dreamed about editing
them. Just did a search.

the header file reports
/* Copyright (C) 1991-2016 Free Software Foundation, Inc.
This file is part of the GNU C Library.

The GNU C Library is free software; you can redistribute
it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any
later version.

The GNU C Library is distributed in the hope that it
will be useful,
but WITHOUT ANY WARRANTY; without even the implied
warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser
General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */

/*
* POSIX Standard: 2.6 Primitive System Data Types
<sys/types.h>
*/


and the macros and typedefs for such sized ints seem to
apply for GCC >= 2.7 (but I might misunderstand)

#if !__GNUC_PREREQ (2, 7)

>
>>>>> in the sense of having unaligned data that would have
>>>>> padding in a normal struct. I have seen no answer to this (unless it is
>>>>> later in this post), despite asking this /crucial/ question.
>>>>
>>>> the files are packed.
>>>
>>> OK - which fields are positioned differently from the way they would be
>>> positioned if not packed?
>>
>> as I said, I do not have a working complete program yet, and
>> I hope to get to a scenario where this wrong situation is
>> avoided at all
>
> I think you can avoid it perfectly without doing anything at all about
> it, because it can't occur. I recommend not wasting any more time
> worrying about it until you can come up with a plausible reason for
> doing so.
>

[Soviet_Mario]

Il 23/11/19 07:04, Keith Thompson ha scritto:

> Soviet_Mario <Sovie...@CCCP.MIR> writes:
> [...]
>> let's say it explicitely
>>
>> if I have
>> union
>> {
>> struct
>> {
>> u_int_8 Red, Green, Blue, Alpha;
>> };
>> u_int_32 AllChannels;
>> };
>>
>> without PRAGMA, do I have any explicit or implicit warranty
>> that my union be 32 bit long ? If so, PRAGMA is useless. If
>> not, it seems it might be useful.
>
> Fixing the spelling of uintN_t (n=8,32) and adding a tag to the union so
> you can use it to define objects:

again ...
Keith, I did not edit the header manually, and sized ints
HERE are spelled as I wrote.

...omissis...

#ifdef __USE_MISC
/* Old compatibility names for C types. */
typedef unsigned long int ulong;
typedef unsigned short int ushort;
typedef unsigned int uint;
#endif

/* These size-specific names are used by some of the inet
code. */

#if !__GNUC_PREREQ (2, 7)

/* These types are defined by the ISO C99 header
<inttypes.h>. */
# ifndef __int8_t_defined
# define __int8_t_defined
typedef char int8_t;
typedef short int int16_t;
typedef int int32_t;
# if __WORDSIZE == 64
typedef long int int64_t;
# else
__extension__ typedef long long int int64_t;
# endif
# endif

/* But these were defined by ISO C without the first `_'. */
typedef unsigned char u_int8_t;
typedef unsigned short int u_int16_t;
typedef unsigned int u_int32_t;
# if __WORDSIZE == 64
typedef unsigned long int u_int64_t;
# else
__extension__ typedef unsigned long long int u_int64_t;
# endif

typedef int register_t;

#else

/* For GCC 2.7 and later, we can use specific type-size
attributes. */
# define __intN_t(N, MODE) \
typedef int int##N##_t __attribute__ ((__mode__ (MODE)))
# define __u_intN_t(N, MODE) \
typedef unsigned int u_int##N##_t __attribute__
((__mode__ (MODE)))

# ifndef __int8_t_defined
# define __int8_t_defined
__intN_t (8, __QI__);
__intN_t (16, __HI__);
__intN_t (32, __SI__);
__intN_t (64, __DI__);
# endif

__u_intN_t (8, __QI__);
__u_intN_t (16, __HI__);
__u_intN_t (32, __SI__);
__u_intN_t (64, __DI__);

typedef int register_t __attribute__ ((__mode__ (__word__)));

...omissis...


as you can see the types are spelled as I wrote them, and a
search for uintXX_t does not find anything


is this that big problem ?
I guess the variations stands for the same concept

>
> union u {
> struct {
> uint8_t Red, Green, Blue, Alpha;
> };
> uint32_t AllChannels;
> };
>
> (You have an anonymous struct, which is standard but wasn't added until
> C11, so older compilers might not support it, or might not support it
> by default.)

it gives me warnings indeed, but seems to compile nevertheless

>
> You're guaranteed that the Red and AllChannels members start at the same
> location, the very beginning of the union. You're guaranteed that Red,
> Green, Blue, and Alpha are at increasing offsets in the order you
> defined them.
>
> It's very very likely that the struct and the union will both be
> exactly 32 bits. A compiler is not prohibited from adding padding
> between members or after the last one,

and that was the reason why I had thought of trying to give
it a strong hint not to pad by mean of a #pragma pack directive

> but there's unlikely to be
> any good reason to do so in this case. The most likely way this
> could fail is if a compiler insisted on making the structure 64 bits.

so either it packs natively, either it listens to my
explicit hint, either it ignore the #pragma and does not pack.

Does the #pragma directive increase, decrease, not affect
the likelyhood of packing ? That was the reasoning behind my
approach

I thought : I give you an hint : TRY to pack. If you really
cannot, there is nothing else I could do

>
> You can easily add a compile-time or run-time assertion that
> sizeof (struct u) == 4 && CHAR_BIT == 8

that's a very good idea !

> so your program won't compile, or at least won't run, in the
> unlikely event that it's used with an implementation that doesn't
> meet your requirements.
>
> There is *no* guarantee about which of the four uint8_t members
> correspond to which part of AllChannels.

and I don't need such warranty : I use AllChannels only for
mass-copying, assignments, copy constructor. When I need
access to separate channels I use named members.

> Google "endianness" if
> you're not familiar with this.

I just know the problem but I am not familiar in the sense I
had never had to write pieces of code depending on this aspect

> It's very common for Red to overlap
> either the high-order byte or the low-order byte of AllChannels.

Yes, but I did no assumption about the order of bytes in the
"long" int

> The other 22 permutations are rare or nonexistent, but Google
> "PDP-11 middle-endian" for an odd case.
>
> If you're doing bitwise operations

no, I chose the struct right to avoid that.
My only possible and very unlikely source of problems could
be if Gambas and C++ has a mean to use a different bit
ordering within a byte.
I did not even give a thought, it seemed to me more an
hardware related feature so transparent to both gambas and C++

> on the entire 32-bit word without
> caring about the order of the bytes, then that might not be an issue.
>
> Attempting to use #pragma pack is very likely to do nothing *at best*.

well ... until now, I am thinking it is likely to do nothing
*at worst* as no one has explained which drawbacks it were
supposed to be in the particular case (I'm not referring to
other more general examples with longer integers
hypothetically spanning non multiple addresses : here I have
1 byte sizes, and the only big member starts at 0 offset)


>
> [...]

[Paavo Helde]

On 23.11.2019 12:12, Soviet_Mario wrote:
> Il 23/11/19 05:56, James Kuyper ha scritto:
>> On 11/22/19 6:25 PM, Soviet_Mario wrote:
>>> Il 22/11/19 20:11, James Kuyper ha scritto:
>>>> On 11/22/19 1:35 PM, Soviet_Mario wrote:
>>>>> Il 22/11/19 15:04, David Brown ha scritto:
>>>> ...
>>>>> I have abandoned the unsigned Bit_Field : 8 solution as soon
>>>>> as you (a lot of people) stated that u_int_8 MUST have
>>>>
>>>> uint8_t
>>>
>>> here the macro or typedef is
>>>
>>> u_int8_t actually
>>
>> Everyone who's been talking with you about this subject has been talking
>> about std::uint8_t from <cstdint>. If you're talking about something
>> else, you need to identify what it is and where is it is declared.
>
> I'm using QT who seems to like to replicate anything its own way even
> when avoiding specific Q-prefixes.
>
> so, HERE, including
> #include <sys/types.h>

<sys/types.h> belongs to The Open Group which is "the certifying body
for the UNIX trademark". It has nothing to do with QT and neither with
standard C++. Also, it appears the u_int8_t typedef is not documented,
meaning that it is an implementation detail which you can't rely on. For
example, in Windows there is a <sys/types.h> header, but it does not
define u_int8_t.

So why cannot you listen to everybody here who are trying to give you
good advice, and use <cstdint> and std::uint8_t?

[Öö Tiib]

On such platform the compilers usually generate more code for
accessing packed structs ... something that they already do for
bit-fields. But yes, that is also factor, code may grow.

[Öö Tiib]

I agree that accessing packed versions is always slower.
Accessing code may be bigger and slower and difference can be
significant or insignificant depending on platform and situation.

However here is slight controversy. Same data can be laid out
so it is more efficient for storage and communications but
different layout can be more efficient for processing.
On lot of cases it does not matter and best layout is what
is for people most easy to reason about.
However where it matters there it can be worth to transform
data between the layouts or it can be worth to use one
or other (or even some third) layout for it all time since
transformations have some cost too. Sometimes only profiling
of fully functional product can show if it affects overall
performance and how.

[Keith Thompson]

Soviet_Mario <Sovie...@CCCP.MIR> writes:
> Il 23/11/19 07:04, Keith Thompson ha scritto:
>> Soviet_Mario <Sovie...@CCCP.MIR> writes:
>> [...]
>>> let's say it explicitely
>>>
>>> if I have
>>> union
>>> {
>>> struct
>>> {
>>> u_int_8 Red, Green, Blue, Alpha;
>>> };
>>> u_int_32 AllChannels;
>>> };
>>>
>>> without PRAGMA, do I have any explicit or implicit warranty
>>> that my union be 32 bit long ? If so, PRAGMA is useless. If
>>> not, it seems it might be useful.
>>
>> Fixing the spelling of uintN_t (n=8,32) and adding a tag to the union so
>> you can use it to define objects:
>
> again ...
> Keith, I did not edit the header manually, and sized ints
> HERE are spelled as I wrote.

[SNIP]

> as you can see the types are spelled as I wrote them, and a
> search for uintXX_t does not find anything

I didn't know where the names u_int_8 and u_int_32 came from,
and assumed they were misspellings of uint8_t and uint32_t.

On my own system, #include <sys/types.h> introduces the non-standard
type names u_int{8,32}_t. It doesn't introduce u_int_{8,32}, and I
don't see that name in the code you quoted (other than in your union
definition) or anywhere under /usr/include on my system.

On the other hand, the types uint8_t and uint32_t are defined by the C
and C++ standards (starting with C99 and C++11, respectively).

Nobody is suggesting that you're lying or manually editing headers. All
we're saying is that uint8_t and uint32_t are the *standard* types that
meet your requirements, and there is no good reason to use non-standard
types with similar names.

Do you have some reason to prefer to use u_int_8 rather than uint8_t?

> is this that big problem ?
> I guess the variations stands for the same concept

Yes. Names other than uint8_t are non-standard and will cause your code
not to compile in many environments.

>> union u {
>> struct {
>> uint8_t Red, Green, Blue, Alpha;
>> };
>> uint32_t AllChannels;
>> };
>>
>> (You have an anonymous struct, which is standard but wasn't added until
>> C11, so older compilers might not support it, or might not support it
>> by default.)
>
> it gives me warnings indeed, but seems to compile nevertheless

What warnings? If you compile with "gcc -std=c11" or with a
sufficiently new version of gcc, there shouldn't be any.

Always pay attention to warnings.

[...]

>> It's very very likely that the struct and the union will both be
>> exactly 32 bits. A compiler is not prohibited from adding padding
>> between members or after the last one,
>
> and that was the reason why I had thought of trying to give
> it a strong hint not to pad by mean of a #pragma pack directive

If it works without a non-standard #pragma pack, why would you add it?
You risk your code failing to compile, or compiling incorrectly, with an
implementation that doesn't support #pragma pack or that does something
unexpected.

>> but there's unlikely to be
>> any good reason to do so in this case. The most likely way this
>> could fail is if a compiler insisted on making the structure 64 bits.
>
> so either it packs natively, either it listens to my
> explicit hint, either it ignore the #pragma and does not pack.
>
> Does the #pragma directive increase, decrease, not affect
> the likelyhood of packing ? That was the reasoning behind my
> approach
>
> I thought : I give you an hint : TRY to pack. If you really
> cannot, there is nothing else I could do

There's no guarantee that all compilers will understand your "hint",
and nothing in the standard that says anything at all about what
"#pragma pack" does.

[...]

>> There is *no* guarantee about which of the four uint8_t members
>> correspond to which part of AllChannels.
>
> and I don't need such warranty : I use AllChannels only for
> mass-copying, assignments, copy constructor. When I need
> access to separate channels I use named members.

What operations do you need on AllChannels that you can't perforom on
the struct? You can assign structs.

[...]

>> Attempting to use #pragma pack is very likely to do nothing *at best*.
>
> well ... until now, I am thinking it is likely to do nothing
> *at worst* as no one has explained which drawbacks it were
> supposed to be in the particular case (I'm not referring to
> other more general examples with longer integers
> hypothetically spanning non multiple addresses : here I have
> 1 byte sizes, and the only big member starts at 0 offset)

The drawback of #pragma pack is that it's non-standard. Given the
concern you've expressed about portability, I would think that would
be important to you. And given that it will almost certainly do
nothing, I don't know why you're so insistent on using it.

[Soviet_Mario]

yes I miss-spelled forgetting the final _t (but the pieces
of headers pasted mentioned it)

>
> On the other hand, the types uint8_t and uint32_t are defined by the C
> and C++ standards (starting with C99 and C++11, respectively).
>
> Nobody is suggesting that you're lying or manually editing headers. All
> we're saying is that uint8_t and uint32_t are the *standard* types that
> meet your requirements, and there is no good reason to use non-standard
> types with similar names.
>
> Do you have some reason to prefer to use u_int_8 rather than uint8_t?

no no reason at all. I just noticed this subtle difference
after having rewritten always :)
But It will take short to do a mass-replace and change the
included header

>
>> is this that big problem ?
>> I guess the variations stands for the same concept
>
> Yes. Names other than uint8_t are non-standard and will cause your code
> not to compile in many environments.

well I must have mis-expressed again when I said I was
looking for a portable way, I actually meant I needed a way
that allowed proper data exchange from front-end gambas side
and C++ engine.

I'll never deploy the program anywhere else. So as long as
it compiles here, fine enough

>
>
>>> union u {
>>> struct {
>>> uint8_t Red, Green, Blue, Alpha;
>>> };
>>> uint32_t AllChannels;
>>> };
>>>
>>> (You have an anonymous struct, which is standard but wasn't added until
>>> C11, so older compilers might not support it, or might not support it
>>> by default.)
>>
>> it gives me warnings indeed, but seems to compile nevertheless
>
> What warnings?

that anonymous structs (or unions) are non standard and
supported only with GCC later than some version.

I'm using the "native" (= preinstalled) GCC here on MX (QT
itself did not ship with any own compiler in the free
version). I have installed the later build-essentials from
"stable" repository. I'm not very willing to update from
testing or backport.
Maybe if I found some flatpak or appimage version, more
isolated.

> If you compile with "gcc -std=c11" or with a
> sufficiently new version of gcc, there shouldn't be any.

I'm looking to assess the GCC version here (on Debian its a
bit older still).

back
here I have GCC 4:6.3.0-4
(and supporting packages seem 6.3.0-18)
anche il CPP riporta la stessa signature.

I found a 9.4-2 MX version and I'm adding it too.
I don't know if QT ide will automatically detect it (and I'm
sure I won't be able to configure it, otherwise)

>
> Always pay attention to warnings.

It was reading the "padding" warning that all this odessy
began :)

>
> [...]
>
>>> It's very very likely that the struct and the union will both be
>>> exactly 32 bits. A compiler is not prohibited from adding padding
>>> between members or after the last one,
>>
>> and that was the reason why I had thought of trying to give
>> it a strong hint not to pad by mean of a #pragma pack directive
>
> If it works without a non-standard #pragma pack, why would you add it?
> You risk your code failing to compile, or compiling incorrectly, with an
> implementation that doesn't support #pragma pack or that does something
> unexpected.
>
>>> but there's unlikely to be
>>> any good reason to do so in this case. The most likely way this
>>> could fail is if a compiler insisted on making the structure 64 bits.
>>
>> so either it packs natively, either it listens to my
>> explicit hint, either it ignore the #pragma and does not pack.
>>
>> Does the #pragma directive increase, decrease, not affect
>> the likelyhood of packing ? That was the reasoning behind my
>> approach
>>
>> I thought : I give you an hint : TRY to pack. If you really
>> cannot, there is nothing else I could do
>
> There's no guarantee that all compilers will understand your "hint",

I just use GCC, not others

> and nothing in the standard that says anything at all about what
> "#pragma pack" does.
>
> [...]
>
>>> There is *no* guarantee about which of the four uint8_t members
>>> correspond to which part of AllChannels.
>>
>> and I don't need such warranty : I use AllChannels only for
>> mass-copying, assignments, copy constructor. When I need
>> access to separate channels I use named members.
>
> What operations do you need on AllChannels that you can't perforom on
> the struct? You can assign structs.

uhm ... memberwise or all-at-once ?
I thought struct assignment (normally I wrote operator =)
should pass through memberwise copying.
So the aim was to copy everything in one assignment only
instead of four of them.
But if struct can be assigned smartly enough then the union
is not needed.

Also I duplicated pixel addressing. One slower way through
operator () (int row, int column) and a massive one
dimensional linear addressing for mass processing of all pixels.

>
> [...]
>
>>> Attempting to use #pragma pack is very likely to do nothing *at best*.
>>
>> well ... until now, I am thinking it is likely to do nothing
>> *at worst* as no one has explained which drawbacks it were
>> supposed to be in the particular case (I'm not referring to
>> other more general examples with longer integers
>> hypothetically spanning non multiple addresses : here I have
>> 1 byte sizes, and the only big member starts at 0 offset)
>
> The drawback of #pragma pack is that it's non-standard. Given the
> concern you've expressed about portability,

sorry I realize I had been unwittingly misleading. I was
just thinking of interoperate gambas code and library code.
As long as it works on my two couple of machines that would
be enough. I have Linux MX on both, BTW, among other things

> I would think that would
> be important to you. And given that it will almost certainly do
> nothing, I don't know why you're so insistent on using it.
>

oh, sorry at last for my often unclear English. I tend to
sow misunderstandings everywhere due to this :\

[Keith Thompson]

To be very clear, if you need an unsigned integer type of at least 8
bits, there is almost no good reason not to use uint8_t. There is no
good reason to use some system-specific (or even undocumented)
equivalent.

"Almost" means you might have to consider older implementations that
don't support <stdint.h> / <cstdint>. Any version of g++ you're likely
to encounter should support C++11 or later -- older versions might
require an explicit option. Even older versions are likely to support
<stdint.h> or <cstdint> as an extension.

>> On the other hand, the types uint8_t and uint32_t are defined by the C
>> and C++ standards (starting with C99 and C++11, respectively).
>>
>> Nobody is suggesting that you're lying or manually editing headers. All
>> we're saying is that uint8_t and uint32_t are the *standard* types that
>> meet your requirements, and there is no good reason to use non-standard
>> types with similar names.
>>
>> Do you have some reason to prefer to use u_int_8 rather than uint8_t?
>
> no no reason at all. I just noticed this subtle difference
> after having rewritten always :)
> But It will take short to do a mass-replace and change the
> included header
>
>>
>>> is this that big problem ?
>>> I guess the variations stands for the same concept
>>
>> Yes. Names other than uint8_t are non-standard and will cause your code
>> not to compile in many environments.
>
>
> well I must have mis-expressed again when I said I was
> looking for a portable way, I actually meant I needed a way
> that allowed proper data exchange from front-end gambas side
> and C++ engine.
>
> I'll never deploy the program anywhere else. So as long as
> it compiles here, fine enough

If you use uint8_t it will compile.

>>
>>
>>>> union u {
>>>> struct {
>>>> uint8_t Red, Green, Blue, Alpha;
>>>> };
>>>> uint32_t AllChannels;
>>>> };
>>>>
>>>> (You have an anonymous struct, which is standard but wasn't added until
>>>> C11, so older compilers might not support it, or might not support it
>>>> by default.)
>>>
>>> it gives me warnings indeed, but seems to compile nevertheless
>>
>> What warnings?
>
> that anonymous structs (or unions) are non standard and
> supported only with GCC later than some version.

It's always better to copy-and-paste diagnostic messages rather than
trying to summarize them.

BTW, I made a serious mistake here. C supports anonymous structs as
of C11. C++ does not support them at all. g++ does support them as
an extension. With "g++ -std=c17 -pedantic", I get "warning: ISO C++
prohibits anonymous structs [-Wpedantic]". My apologies for that.

If you're only using gcc, you can use anonymous structs in C++,
but your code will be non-portable.

[...]

>>>> There is *no* guarantee about which of the four uint8_t members
>>>> correspond to which part of AllChannels.
>>>
>>> and I don't need such warranty : I use AllChannels only for
>>> mass-copying, assignments, copy constructor. When I need
>>> access to separate channels I use named members.
>>
>> What operations do you need on AllChannels that you can't perforom on
>> the struct? You can assign structs.
>
> uhm ... memberwise or all-at-once ?
> I thought struct assignment (normally I wrote operator =)
> should pass through memberwise copying.
> So the aim was to copy everything in one assignment only
> instead of four of them.
> But if struct can be assigned smartly enough then the union
> is not needed.

The semantics of struct assignment (assuming you haven't overloaded
operator=) are equivalent to the semantics of copying memberwise,
but in the case of a struct containing 4 uint8_t members, no sane
compiler will do anything other than a 32-bit copy. That's assuming
the struct has an alignment that permits a 32-bit copy.

If copying the entire structure is the only operation you care about,
keep it simple: just define the structure and use ordinary structure
assignment.

[...]

[Soviet_Mario]

well strictly speaking I need a type of EXACTLY 8 bits, not more

well, I did not copy the warning when the compiler produced it.
I would have to re-edit the source (or restore the older
version) to just replicate the output ... maybe messing sth :\

>
> BTW, I made a serious mistake here. C supports anonymous structs as
> of C11. C++ does not support them at all. g++ does support them as
> an extension. With "g++ -std=c17 -pedantic", I get "warning: ISO C++
> prohibits anonymous structs [-Wpedantic]". My apologies for that.

no problem !!! this is not an helpdesk, I get the good
offered for free with good intention. That's more than enough !

>
> If you're only using gcc, you can use anonymous structs in C++,
> but your code will be non-portable.

yes only GCC.
BTW : as I feared, QT (even after a system reboot ...
actually power went off tonight, here it poured dogs and
cats ! 160 mm of rain in 24 hours, a disaster) did not
automatically detect the new version from MX repo. it just
still see the original version.
I'll see to this later If i find how to configure it (and
locate it first)

>
> [...]
>
>>>>> There is *no* guarantee about which of the four uint8_t members
>>>>> correspond to which part of AllChannels.
>>>>
>>>> and I don't need such warranty : I use AllChannels only for
>>>> mass-copying, assignments, copy constructor. When I need
>>>> access to separate channels I use named members.
>>>
>>> What operations do you need on AllChannels that you can't perforom on
>>> the struct? You can assign structs.
>>
>> uhm ... memberwise or all-at-once ?
>> I thought struct assignment (normally I wrote operator =)
>> should pass through memberwise copying.
>> So the aim was to copy everything in one assignment only
>> instead of four of them.
>> But if struct can be assigned smartly enough then the union
>> is not needed.
>
> The semantics of struct assignment (assuming you haven't overloaded
> operator=) are equivalent to the semantics of copying memberwise,
> but in the case of a struct containing 4 uint8_t members, no sane
> compiler will do anything other than a 32-bit copy. That's assuming
> the struct has an alignment that permits a 32-bit copy.

I would think that the existence of such monolythic member
should ensure the alignment.
I should look the asm generated, but I can't be bothered :) :)

>
> If copying the entire structure is the only operation you care about,
> keep it simple: just define the structure and use ordinary structure
> assignment.
>
> [...]
>


--

[David Brown]

On 22/11/2019 19:35, Soviet_Mario wrote:
> Il 22/11/19 15:04, David Brown ha scritto:
>> On 22/11/2019 13:14, Soviet_Mario wrote:
>>> Il 22/11/19 12:23, David Brown ha scritto:

<snipping quite a bit to cut down on the size of the post>

> put is alas irregular verb : I meant its PAST form actually.
> I have abandoned the unsigned Bit_Field : 8 solution as soon as you (a
> lot of people) stated that u_int_8 MUST have exactly 8 bits.
>
> In that case either pragma is USELESS (and hence does not produce any
> reasonable effect, neither slowness), either is NECESSARY (and so the
> slowing effect is a unavoidable price that I MUST pay either)

A "pack" attribute is unlike to have any effect when you have only
uint8_t types in a struct or array. There is a hypothetical possibility
of it affecting padding or alignments on the struct as a whole, but that
will definitely not apply for any platform "normal" enough to support
QT. But have no positive or useful effects does not guarantee that you
will get no negative effects in terms of efficiency or what the compiler
allows for the code, and certainly you will still have the negative
effects of non-portable code and confusing source code.

>
>> "Packed" is /not/ a
>> "precaution", nor is using a struct with a bitfield instead of uint8_t.
>
> the second part has been abandoned early

OK.

>
>>   If you don't know what you are doing
>
> I'm rather weak in higher level C++ constructs, and I know that, but not
> particularly so in low level (some couple of tens of years ago I used to
> stuff C code with inline "asm" statements or even __emit() calls).
>
>> , fine - if I understand you
>> correctly, you are experienced with gambas
>
> no, I am rather new to gambas actually.
> I was used to VB (Visual Studio, but not even recently). Going to Gambas
> has been actually a pain (not saying it is bad, absolutely not : it's me
> that knows much less than .net framework). On QT I am even more newby.
>
>>  programming but new to C++.
>
> well, I am more of outdated than new, I had used C++ (and C) a lot in
> 1995-2000 roughly, then less and less afterwards.
> But strangely enough, I find it more easy to think in C++ terms when I
> met problems in basic.

OK.

>
>> So /ask/ (and please read the answers).  Don't guess, or grab the first
>> thing that you find from a google search.
>
> but I DID follow the answers :)

I saw a lot of posts indicating that you either didn't read the answers,
didn't understand them, didn't agree with them, or at least did not rate
them as reliable sources of information. There have been so many posts
in this thread that it is hard to have an overview, but I believe you
made several "I'm keeping pragma pack as a precaution" posts after
having been told how it is worse than useless. But maybe I have a false
impression here, based on the order of reading posts or details of what
were written.

>
>> And when the more experienced
>> people in a group like this all say "don't use packed" and "use
>> uint8_t",
>
> I didn't read or desume the two things should be mutually exclusive.

I can see how that might not have been clear at the start.

>
> Anyway, there are three scenarios
>
> 1) added pragma (to the four u_int_8 members in a struct) is USELESS, as
> the struct was just packed natively (and hence does not produce any
> reasonable effect, neither slowness),

Yes, it is useless here.

> either
> 2) is NECESSARY, as the struct would not have been natively packed
> otherwise (and so the slowing effect is a unavoidable price that I MUST
> pay either)

When you have understood what "packed" does, and how compilers handle
structs, you will realise that this will not be the case. (Any
compiler/target combinations sufficiently odd that uint8_t structs are
not tightly packed anyway, are unlikely to support "pragma pack", almost
guaranteed not to support QT, and are definitely not platforms you will
be using anyway.)

> either
> 3) it is NOT SUFFICIENT (in this case, luckily unlikely, nothing would
> have worked fine at all)

Correct.

But you have missed the fact that there are potential disadvantages in
using "pack" even when it offers no advantages.

>
> as for ALIGNMENT problem, I should not meet any I guess.
> The packed (natively or forcefully) struct is 32 bit wide, so allocating
> an array should allow nice addressing of any item even on a 64 bit OS
> And no item would cross critical addressing boundary

The struct will be 32 bit in size, but the alignment will be one byte.
Whether you consider that an advantage or disadvantage compared to four
byte alignment depends on how it will be used.

>
>
>> then that is what you should do.  Ask /why/, and try to learn
>> - and once you do understand "pack", when it can be useful, when it is
>> useless, when it is worse than useless,
>
> If you have some further patience, tell me if the 3 scenarios fit the
> present situation (not a more general one, but just this)

Hopefully my reply above covers this.

>
>> what it does, and what it costs
>> - /then/ you can decide that because you know the full situation better
>> than we do, you still want to use it against general advice.
>
> I had given all details sufficient : the data layout is what I said, so
> no obscure aspects
>
>>
>> "Packed" is not a precaution -
>
> I call it a precaution as it reduces the chances of a non packed data
> layout.

It does not do that - the chances of the uint8_t data not being packed
are 0, and "packed" does not reduce those chances. Therefore it is not
a "precaution".

> I was not saying it would be successful necessarily in every
> scenario (even sizes sequence is relevant, and I know this, but my
> members were all 1 byte long)
>
>> it is a last choice solution when you
>> have nothing better and know you can't write the code as well without
>> it.  (That might mean you can't write the code as quickly or clearly
>> without it, which is fair enough - /if/ you are sure it is necessary.)
>
> let's say it explicitely
>
> if I have
> union
> {
>     struct
>     {
>         u_int_8 Red, Green, Blue, Alpha;
>     };
>     u_int_32  AllChannels;
> };
>
> without PRAGMA, do I have any explicit or implicit warranty that my
> union be 32 bit long ? If so, PRAGMA is useless. If not, it seems it
> might be useful.

If you keep using non-standard types, then we can't be entirely sure. I
know what the standard types "uint8_t" and "uint32_t" are, but I have no
definition for "u_int_8" and "u_int_32". I /assume/ they are defined
like the standard types, but you might have made a mistake. I strongly
recommend you don't use home-made types like this when there are
perfectly good standard types available.

The C and C++ standards give implementations a lot of freedom about
packing, padding and alignments in structs and unions. But practical
implementations do not do anything weird unless they have good reason
for it. When you are talking about mainframes, supercomputers, DSPs, or
particularly odd niche processors, you need to be careful and check the
details. For ordinary computers, and code that would never be used on
anything else, there is no problem. (For a guide, anything targetting a
POSIX or Windows system, or any embedded system supported by gcc or
clang, there will be no problems.) On such "ordinary" targets, all
basic types are 8-bit, 16-bit, 32-bit, 64-bit or 128-bit, and the
alignment of those types is at most equal to their size. (It can be
smaller, however, if the "bitness" of the cpu is smaller than that
size.) No padding is added except to make sure each struct member is
aligned, and that the struct itself is aligned by the biggest alignment
of any member.

> what damages could PRAGMA PACK produce ?

I've told you already.

> If the union would be just packed natively, I guess none.

No.

It /might/ be the case that the only disadvantages of "pragma pack" are
lack of portability to other tools on the same or similar platforms, and
confusion for the reader as to why you have added this useless pragma.

But it is also possible that it will restrict what you can do with the
struct (like taking the address of the members), it can lead to errors
that are not diagnosed (like letting you take the address of the
members, but those addresses not being aligned and this leading to
subtle problems when the compiler assumes they /are/ aligned), it can
lead to inefficiencies (like compilers generating byte-by-byte access to
larger fields on some targets) or compilers being pessimistic about
optimisations.

> If it wasn't, at best it would pack it (and maybe slow it, but this is
> irrelevant, I need precise disk layout).

If the union was not packed natively, you'd be working on a very odd
system and "pragma pack" would not apply anyway.

> At worsk it would be
> insufficient to actually pack it, so my problem is unsolved.
> In this last worst case, are there any other means of getting the thing
> to work ?
> What are them ?
>

If you are dealing with a system where you can't be sure your simple
arrays and structs are packed sensibly (at most padded for "natural"
alignment), you can't handle it this way anyway - you are going to have
to write code that manually unpacks the raw stream of byte data.

<snip>

>> And if your file format contains non-aligned members,
>
> why making general guess when I reported precisely the layout ?
> I said :
> 16 bit (width) - 16 bit (height) - N x 32 bit (R,G,B,alpha)
> where N = width x height = number of "pixels"

That does not tell us anything about alignment. Are your fields
properly aligned or not?

>
> every pixel is multiple of 32 address on file (but this seems irrelevant
> to me).

I presume you mean multiple of 32 bits, i.e., 4 bytes. And yes, this is
/highly/ relevant. It means your data is well aligned, and you can use
a union with a uint32_t for your pixels (if you want). Are you also
sure that your 16-bit values (height and width) are 16-bit aligned? As
long as all your elements are naturally aligned, then the struct is
properly aligned and you can use a plain, simple struct with no "packed"
complications.

This is /critical/ to being able to support your structure simply,
cleanly, efficiently and portably. That is why I have repeatedly asked
you if your format is properly aligned.

> In the ram layout,
> the BMP descriptor has width and height as "local" members, and a
> pointer to an array of struct aligned in memory by the NEW [] operator
> who is aware of alignment requirements.
>
> Where are the possible problems ?
> What problems are affected by a pragma pack directive ?
>
>
>> then the simplest
>> and easiest way to improve and simplify the code, and improve the speed
>> of several parts, is to fix the file format.
>
> that point is unmodificable. I strictly need full color and no
> compression. The alpha channel is maybe a bit overkill (255 level of
> transparency is more than needed ... but taking less is useless or
> impossible, and 1 byte is natural for the otherwise 3,sth bytes long
> struct)

I haven't suggested using smaller types for your data - I assume that
these are appropriate types.

>
>>  Make sure the members are
>> aligned (add explicit dummy members if necessary)
>
> since when reading/writing files suffer from this ? Files are BYTE
> oriented internally (and the OS does caching independently and
> transparently, basing on disk cluster sizes but this is not sth I should
> care of)

It does not affect the file reading and writing - it affects the struct
in memory that is accessed by the code. C and C++ only define access to
correctly aligned objects and subobjects. Access to unaligned data is
implementation dependent, may require extra extensions (like pragma
pack), may be subject to various restrictions, may be inefficient
(either extra code, or inefficiencies in the cpu processing), and may
cause problems if the data is used with third-party libraries or code.

That is a good plan. When you do that, you avoid any need for "pack"
extensions.

It appears that you have designed a /sensible/ file format. It is
/simple/, but certainly not "stupid". It has taken a while to get you
to give enough details to establish this, but it seems to have been a
sensible format all along.

What you have got wrong is your determination to distrust your own
sensible decisions in the file layout and add worse than useless extra
"pack" junk. Forget "pack" - forget you ever heard of it. It is not
part of the C++ language, it is rarely useful, and it is certainly not
useful here. Just use your file format in the clear, sensible layout
you have designed.

>
>>
>> As for asking for help here, you are getting the help you need even if
>> it is not exactly what you asked for.
>
> not a problem, every advice is appreciated
>
>> It's as though you said "I need a
>> new toothbrush.  Can you tell me how to get to the toy shop?"  It is
>> conceivable that a toy shop might have toothbrushes, but it is very
>> unlikely to be a good choice.  Would you not rather have people say "You
>> don't want to go to a toy shop.  You want to go to a chemist - this is
>> how you get there" ?
>>
>>>
>>>> read or write it from the disk.  You haven't compared the time the
>>>> processing takes for a "packed" struct to how it would run with an
>>>> unpacked struct.
>>>
>>> forgive me : how to read an all packed file in an array of UNpacked data
>>> of different layout, in a single read operation ? I can't figure out
>>> this
>>>
>>
>> First, you take a step back.  You look to see if the packed file really
>> is "packed",
>
> it is, as both Gambas supports BYTE format (And sizeof()) and
> BYTE-oriented streams. No overspace is generated. Then I have to worry
> about the C++ back end, to comply with this fact.

My knowledge of Gambas is very limited, but according to the
documentation I found its structs are /not/ packed:

<http://gambaswiki.org/wiki/lang/structdecl>

Padding /is/ generated - if needed to get the natural alignment of the
fields. You can be entirely confident that on any platform supported by
Gambas, the padding, alignment and layout of a Gambas struct and a C++
struct will be the same.

>
> I think the designer of Gambas relies very strongly upon C libraries and
> mind-shape, apart from strings (where he had to welcome BASIC users who
> want strings the basic way).

Of course. And while the details of C struct layouts is implementation
dependent, it is invariably the simplest and most obvious but efficient
layout for the target.

>
>> in the sense of having unaligned data that would have
>> padding in a normal struct.  I have seen no answer to this (unless it is
>> later in this post), despite asking this /crucial/ question.
>
> the files are packed.

The information you have given so far in this post suggests they are
properly aligned. They are without padding, but that is because of the
field sizes used and their arrangement, not because you have
specifically asked for non-standard packing.

>
>>
>> If the format is already good, you are done.  One memcpy, or read from
>> the file directly to a suitable buffer, or memmap to the file directly
>> (no reading or writing needed).
>
> memmap is a problem in that I don't know how exactly works and whether
> or not I'll be able to interoperate with memory files opened in gambas
>
> I am also considering to use some memory files supported directly by the
> OS (here on linux files created on /shm virtual folder are designed for
> such FAST temporaries to be shared, but with portable stream syntax).
> Using /shm would imply no specialized load/store code, just the normal
> file access => delegating the ram usage to the OS.

If your files are in a ram file system (/shm, tmpfs, etc.) then memmap
would not be faster anyway.

>
>>
>> If the format is bad, the next step is to consider changing the format.
>
> just observation.
> So, the format is wrong ?

/You/ tell /me/. Only /you/ know the full format. But from what you
have said here, the format is probably fine.

>
>>   This would be good for all parts of the system.  When the format is
>> changed, you are done.  Again, I have had no answer to whether this is
>> possible.
>>
>> If the format is bad, and cannot be fixed, then you have to write reader
>> and writer functions that unpack and pack the data into the right
>> formats.  Reader and writer functions are common for all sorts of
>
> well, I'd be considering these for small size, complex structured input,
> not for a 4-12 MB size of simple pixels. It would be a blood bathe

You are basing that on speculation with no evidence.

I am not at all saying that it is a bad idea to use a direct mapping
from file format to struct - indeed, it is a sensible idea whenever you
can define the format.

As far as I understand your project, you are using Gambas for a gui but
for some kind of image processing, you are saving the pictures from
Gambas as files, then starting an external C++ program that reads these
files, processes them in some way, and saves them again, and then the
Gambas program will read them in again.

I have no idea if this is a good method - either from the viewpoint of
development ease, or run-time efficiency. I can think of many
alternative ways to structure this task, which may or may not be better
- without knowing details of the task, or why you might prefer a
particular setup.

But it does seem to me that you are making your decisions based on gut
feeling rather than research or measurement. That does not mean your
decisions are wrong, but it might mean you should not be so confident in
them.

It sounds like it is probably fine - but as I have said before, you
haven't given all the important details clearly enough.

I want you to understand what is important when making such structures -
that everything is aligned properly (the best being "natural"
alignment), and I strongly recommend adding any padding manually as
unused "padding" fields. I want you to understand /why/ this is
important. And once you understand that, you will realise that "pragma
pack" and similar extensions are useless here and not only are you able
to avoid them, but you /should/ avoid them.

No, it is not. Not remotely. It would be a sad state for C and C++ if
they required compiler-specific extensions to deal with data from
external sources.

I think perhaps you are mixing up the concepts of "a structure where the
fields are adjacent with no padding" and "uses pragma packed". I am
recommending using structs fit the former, without ever using the later.

Or perhaps you think compilers add random padding and re-arrange structs
in odd and unpredictable ways. The details here are usually specified
by the platform's ABI, precisely so that programs can exchange data
safely (by files, pipes, streams, shared memory, shared libraries, etc.).

> And also the caller expects to get the transformed data back in the same
> packed form.
>
>> means totally pointless complications, inefficiencies, and non-portable
>> code.
>
> nope, I have constraints to fullfil. Maybe inefficient, maybe non
> portable (who cares ?), but surely not pointless, as otherwise data
> exchange would not work at all

"pragma pack" is pointless here.

[Soviet_Mario]

On 24/11/2019 20:53, David Brown wrote:
> <snip>
>

>>> And if your file format contains non-aligned members,
>>
>> why making general guess when I reported precisely the
>> layout ?
>> I said :
>> 16 bit (width) - 16 bit (height) - N x 32 bit (R,G,B,alpha)
>> where N = width x height = number of "pixels"
>
> That does not tell us anything about alignment.  Are your
> fields properly aligned or not?

these fields start at the offset 0 of the file.
And they are loaded (fwrite) not in a generic void * buffer
but in an array of typed structs of the kind we are saying.
So I think they are ...

>
>>
>> every pixel is multiple of 32 address on file (but this
>> seems irrelevant to me).
>
> I presume you mean multiple of 32 bits, i.e., 4 bytes.  And
> yes, this is /highly/ relevant.  It means your data is well
> aligned, and you can use a union with a uint32_t for your
> pixels (if you want).  Are you also sure that your 16-bit
> values (height and width) are 16-bit aligned?

they are in the FILE only, but are freaded separately.
The struct itself does not contain these fields.

> As long as
> all your elements are naturally aligned, then the struct is
> properly aligned and you can use a plain, simple struct with
> no "packed" complications.

okay.

we agree that this will not happen here
CUT

>>>
>>> However you look at it, "pack" is usually (but not
>>> always) the wrong
>>> choice.  You should only consider it once you have
>>> eliminated better
>>> alternatives.
>>
>> for a starting, I tried to use a 32 bit total size, hoping
>> it would have been packed regardless of any "hints" (pragma).
>
> That is a good plan.  When you do that, you avoid any need
> for "pack" extensions.
>
> It appears that you have designed a /sensible/ file format.
> It is /simple/, but certainly not "stupid".  It has taken a
> while to get you to give enough details to establish this,
> but it seems to have been a sensible format all along.
>
> What you have got wrong is your determination to distrust
> your own sensible decisions in the file layout and add worse
> than useless extra "pack" junk.  Forget "pack" - forget you
> ever heard of it.  It is not part of the C++ language, it is
> rarely useful, and it is certainly not useful here.  Just
> use your file format in the clear, sensible layout you have
> designed.

LOL ... okay ! Surrendered :)

>>> First, you take a step back.  You look to see if the
>>> packed file really
>>> is "packed",
>>
>> it is, as both Gambas supports BYTE format (And sizeof())
>> and BYTE-oriented streams. No overspace is generated. Then
>> I have to worry about the C++ back end, to comply with
>> this fact.
>
> My knowledge of Gambas is very limited, but according to the
> documentation I found its structs are /not/ packed:

no, they arent, but the internal format it uses for 2D
graphics is. Also my very simple structure layout (4 bytes)
was easy enough for even gambas to produce a file with the
expected size (and consistent content, as I realoaded it).

>
> <http://gambaswiki.org/wiki/lang/structdecl>
>
> Padding /is/ generated - if needed to get the natural
> alignment of the fields.  You can be entirely confident that
> on any platform supported by Gambas, the padding, alignment
> and layout of a Gambas struct and a C++ struct will be the
> same.

I had the sensation gambas is little more than a wrapper
around C, with some syntactic sugar.

>
>>
>> I think the designer of Gambas relies very strongly upon C
>> libraries and mind-shape, apart from strings (where he had
>> to welcome BASIC users who want strings the basic way).
>
> Of course.  And while the details of C struct layouts is
> implementation dependent, it is invariably the simplest and
> most obvious but efficient layout for the target.

in fact it was working OK in gambas.
I would have stayed there, the performance were very good,
but I was starting not to be able to manage the code.
C++ helps so much to organize the structure and to think
correctly.
I am also changing to huge switch () case: to arrays of
pointers to functions for single bit manipulation variants.

Uh ... btw, I go Off Topic.

I was declaring some 3 lines long function as inlined
But, as I started to get their address and populate an array
of pointers, I removed the inline suggestion.

just for, lol :) precaution.
What's the effect of inline linkage when one takes the
address of a function (and stores it) ? I assumed, without
reading documentation supporting this, It would just ignore
inline suggestion.
Or is it undefinite behaviour ?

>>>
>>> If the format is already good, you are done.  One memcpy,
>>> or read from
>>> the file directly to a suitable buffer, or memmap to the
>>> file directly
>>> (no reading or writing needed).
>>
>> memmap is a problem in that I don't know how exactly works
>> and whether or not I'll be able to interoperate with
>> memory files opened in gambas
>>
>> I am also considering to use some memory files supported
>> directly by the OS (here on linux files created on /shm
>> virtual folder are designed for such FAST temporaries to
>> be shared, but with portable stream syntax).
>> Using /shm would imply no specialized load/store code,
>> just the normal file access => delegating the ram usage to
>> the OS.
>
> If your files are in a ram file system (/shm, tmpfs, etc.)
> then memmap would not be faster anyway.

I am reading now (The GNU C library reference manual)
that there are functions seeming to directly and
specifically support the /shm shared virtual folder

shm_open, shm_unlink =>
both support NAMED memory streams
and they seem very fit for sharing big pictures, as
exchanging ascii names will be simpler and more readable;
at need even "hardwired" in constants : there won't be many
temporaries at a time, and recycled over and over again in
case, so not a true limitations. Once the results are
passed, they will be saved in regular files (.PNG or .JPG).

also memfd_create seem to have this same advantage (use
names instead of file descriptors).


But this will be the homework for when I am finished the
other parts.

>
>>
>> well, I'd be considering these for small size, complex
>> structured input, not for a 4-12 MB size of simple pixels.
>> It would be a blood bathe
>
> You are basing that on speculation with no evidence.
>
> I am not at all saying that it is a bad idea to use a direct
> mapping from file format to struct - indeed, it is a
> sensible idea whenever you can define the format.
>
> As far as I understand your project, you are using Gambas
> for a gui but for some kind of image processing, you are
> saving the pictures from Gambas as files,

originally I was doing in gambas, saving / loading included
(as it exports in standard formats, allows for desktop
grabbing and so), but also then I converted colors to my
modified format for greater control and effects that I did
not find the way I wanted them.

Then it grow and I started to no longer be able to "read the
code", find things, know the impact of modifications.

the fact is that in C++ layers are fare more isolated and
when code is written well, modifications often are just
local and do not alter the way other layers use the underlying.
In fact Im slowing regaining control of what happens :)

but the GUI with QT is out of my skills. I just roughly
understand its tipical sockets/signal lazy binding.

> then starting an
> external C++ program that reads these files, processes them
> in some way, and saves them again, and then the Gambas
> program will read them in again.

possibly using /shm files, not to slow down too much

>
> I have no idea if this is a good method -

no, in general is not. But I'm not a "pro" neither in gambas
nor in C++, so I am trying to mix things to use the easier
part of each in its own natural context.
The GUI in gambas is really simple to manage, compared to
QT. Almost as easy as Visual Studio.

But I was losing control over gambas code, it does not
enforce nor allows for tidyness and sealed objects, its
project management is a nightmare (one file => one class) to
me, so I simply spent a lot of time re-reading my own code.
I was obliget to return to C++ to sweep the dirt :)

> either from the
> viewpoint of development ease, or run-time efficiency.  I
> can think of many alternative ways to structure this task,
> which may or may not be better - without knowing details of
> the task, or why you might prefer a particular setup.

the bottleneck is by far my own skills with this two tools.
With the old Visual Basic 2012 or C# I would have finished
it all in days.
But I left windows, so I have to use other tools I am not
much familiar with
I cannot stand windows, but I must admit Visual Studio was
really powerful, complete, cozy and all, and above all
worked always out of the bag, preconfigured consistently. I
really miss it ! QT creator is hard to configure :\

>
> But it does seem to me that you are making your decisions
> based on gut feeling rather than research or measurement.

absolutely true. But I AM the bottleneck :) of the all chain.

> That does not mean your decisions are wrong, but it might
> mean you should not be so confident in them.

oh I am not confident in the solution : I simply crashed in
using gambas alone for both the gui and more complex
elaborations (struggling with the computations), and also
using QT alone for both (struggling with GUI designer).

Then I decided to try to split the interface from the
engine, at the price of data exchange (possibly with memory
mapped files). I did not have much else left to do.

Maybe I'll crash again, and resign to reinstall a virtual
machine with Visual Studio inside ! LOL.
No ... I'm not so willing to resign actually.

> enough.
>
> I want you to understand what is important when making such
> structures - that everything is aligned properly (the best
> being "natural" alignment), and I strongly recommend adding
> any padding manually as unused "padding" fields.  I want you
> to understand /why/ this is important.  And once you
> understand that, you will realise that "pragma pack" and
> similar extensions are useless here and not only are you
> able to avoid them, but you /should/ avoid them.

all true, but as I said : the file is packed. And I don't
want accessors like iomanip and that stuff.
I need to binary write in one block


CUT

>>
>> oh I completely disagree with this point ! The format has
>> to be packed as the C++ library receives the file from an
>> other source of data that offers them in that form, so
>> it's mandatory.
>
> No, it is not.  Not remotely.  It would be a sad state for C
> and C++ if they required compiler-specific extensions to
> deal with data from external sources.
>
> I think perhaps you are mixing up the concepts of "a
> structure where the fields are adjacent with no padding" and
> "uses pragma packed".  I am recommending using structs fit
> the former, without ever using the later.

okay. Some day I'll test if it loads well or not.

>
> Or perhaps you think compilers add random padding and
> re-arrange structs in odd and unpredictable ways.

:) well, not randomly. I think they just try to keep int_32,
float at addresses multiple of 4, and int_64, double
multiple of 8 and so.

I'm not sure of the degree of freedom they have in
"reordering" members though.
Sometimes in constructors I get some warning about the
members will be initializated in a different order than
declared.
But I did it not too much care, as they did not depend on
each other and so actual order was not important.

> The
> details here are usually specified by the platform's ABI,
> precisely so that programs can exchange data safely (by
> files, pipes, streams, shared memory, shared libraries, etc.).
>
>> And also the caller expects to get the transformed data
>> back in the same packed form.
>>
>>> means totally pointless complications, inefficiencies,
>>> and non-portable
>>> code.
>>
>> nope, I have constraints to fullfil. Maybe inefficient,
>> maybe non portable (who cares ?), but surely not
>> pointless, as otherwise data exchange would not work at all
>
> "pragma pack" is pointless here.
>
>>
>>>
>>> If you have learned nothing else from this conversation,
>>> hopefully you
>>> have learned that much.
>>>
>>>>

ok ... discussion closed, apart from some OT issues

[Keith Thompson]

Soviet_Mario <Sovie...@CCCP.MIR> writes:
> On 24/11/2019 02:06, Keith Thompson wrote:

[...]

>> To be very clear, if you need an unsigned integer type of at least 8
>
> well strictly speaking I need a type of EXACTLY 8 bits, not more
>
>> bits, there is almost no good reason not to use uint8_t. There is no
>> good reason to use some system-specific (or even undocumented)
>> equivalent.

Yes, I meant exactly. I have no idea why I typed "at least".

[Öö Tiib]

On Sunday, 24 November 2019 22:53:53 UTC+2, Soviet_Mario wrote:
>
> I was declaring some 3 lines long function as inlined
> But, as I started to get their address and populate an array
> of pointers, I removed the inline suggestion.
>
> just for, lol :) precaution.
> What's the effect of inline linkage when one takes the
> address of a function (and stores it) ? I assumed, without
> reading documentation supporting this, It would just ignore
> inline suggestion.
> Or is it undefinite behaviour ?

There are no inline linkage. Functions declared inline (or constexpr
that is implicitly inline) have external linkage by default.
The sole "benefit" we are guaranteed to get from inline is that we
may include same header that defines the inline function in multiple
translation units without violating ODR. Same benefit we get with
function templates.

So removal of inline from non-template function when it is included
from multiple translation units leads to ODR violation and undefined
behavior.

[Soviet_Mario]

uhm ... I must admit I did not understand most.

> There are no inline linkage.

in general or in this case ?

> Functions declared inline (or constexpr
> that is implicitly inline) have external linkage by default.
> The sole "benefit" we are guaranteed to get from inline
is that we
> may include same header that defines the inline function
in multiple
> translation units without violating ODR.

I'm not sure : then Inline seems to grant the possibility of
"redefinition" without redefining (as it is not considered a
real function) ? Right ?

I tend to embed some very short function in headers instead
of a "pure" interface description and put ALL code in the
.CPP unit. Is it a bad practice ?

In case it is not a so bad habit :
* I ought to add inline in all functions in the headers ?
* It is a "must" only if I wish to include the same headers
multiple times in different .CPP units ? (to avoid the
redeclaration) ?
* in headers I actually put "safeguard" like

#ifndef THIS_HEADER
#define THIS_HEADER
.....
.....
.....
#endif

structure.
* Shouldn't this protect against the multiple useless
inclusion ?
Are there scenarios where a multiple REAL inclusion is
useful ? I can't figure out any ...

> Same benefit we get with
> function templates.

sorry this point is obscure to me, I am very rusty with
templates :\

[Öö Tiib]

On Monday, 25 November 2019 13:45:01 UTC+2, Soviet_Mario wrote:
> On 24/11/2019 22:46, Öö Tiib wrote:
> > On Sunday, 24 November 2019 22:53:53 UTC+2, Soviet_Mario wrote:
> >>
> >> I was declaring some 3 lines long function as inlined
> >> But, as I started to get their address and populate an array
> >> of pointers, I removed the inline suggestion.
> >>
> >> just for, lol :) precaution.
> >> What's the effect of inline linkage when one takes the
> >> address of a function (and stores it) ? I assumed, without
> >> reading documentation supporting this, It would just ignore
> >> inline suggestion.
> >> Or is it undefinite behaviour ?
> >
> > There are no inline linkage. Functions declared inline (or constexpr
> > that is implicitly inline) have external linkage by default.
> > The sole "benefit" we are guaranteed to get from inline is that we
> > may include same header that defines the inline function in multiple
> > translation units without violating ODR. Same benefit we get with
> > function templates.
> >
> > So removal of inline from non-template function when it is included
> > from multiple translation units leads to ODR violation and undefined
> > behavior.
> >
>
>
> uhm ... I must admit I did not understand most.
>
> > There are no inline linkage.
>
> in general or in this case ?

In general. Implementation may inline or not inline functions, member
functions or lambdas with whatever linkage as long as the
observable behavior of program does not change and compilers and
linkers use that freedom a lot. There may be attributes that force
implementations to behave in certain way but inline keyword does
have that power.

> > Functions declared inline (or constexpr
> > that is implicitly inline) have external linkage by default.
> > The sole "benefit" we are guaranteed to get from inline
> is that we
> > may include same header that defines the inline function
> in multiple
> > translation units without violating ODR.
>
> I'm not sure : then Inline seems to grant the possibility of
> "redefinition" without redefining (as it is not considered a
> real function) ? Right ?

No, there may be no tricks used, the copies in different
translation units must be same.

> I tend to embed some very short function in headers instead
> of a "pure" interface description and put ALL code in the
> .CPP unit. Is it a bad practice ?

It is good practice for readers of your code. It keeps your interface
terse but not uselessly bleak. However it does not prevent linkers
from picking something defined in .cpp and inlining it.

>
> In case it is not a so bad habit :
> * I ought to add inline in all functions in the headers ?
> * It is a "must" only if I wish to include the same headers
> multiple times in different .CPP units ? (to avoid the
> redeclaration) ?

There are lots of nuances. Function members defined within
class are implicitly inline, constexpr functions and member functions
are implicitly inline, members of templates and template functions
do not violate ODR, non-member static functions do not violate ODR,
and there can be more.

> * in headers I actually put "safeguard" like
>
> #ifndef THIS_HEADER
> #define THIS_HEADER
> .....
> .....
> .....
> #endif
>
> structure.
> * Shouldn't this protect against the multiple useless
> inclusion ?

That is protecting against including header multiple times to same
translation unit (.cpp file). But header is typically meant to be
included to multiple different translation units.

> Are there scenarios where a multiple REAL inclusion is
> useful ? I can't figure out any ...

Oh, some good old preprocessor metaprogramming tricks involved
not only multiple inclusions but outright recursive inclusions.

In modern C++ most of those are unneeded since templates and
constexpr provide decent alternatives and upcoming in C++20
Ranges, Coroutines, Concepts, and Modules will likely
alter C++ as lot as C++11 did.

However we have decades of history and desire to be more modern
is not rational reason for to refactor (and so possibly break)
already working codes.

> > Same benefit we get with
> > function templates.
>
> sorry this point is obscure to me, I am very rusty with
> templates :\

Most templates are simply compile-time configurable classes and
functions. These can be configured ("instantiated") in different
ways in same program and that is very handy actually.
Template functions are usually in header file, often non-inline
but that does not cause any ODR violations.

[James Kuyper]

On 11/23/19 5:12 AM, Soviet_Mario wrote:
> Il 23/11/19 05:56, James Kuyper ha scritto:
>> On 11/22/19 6:25 PM, Soviet_Mario wrote:
>>> Il 22/11/19 20:11, James Kuyper ha scritto:

...

>>>> uint8_t
>>>
>>> here the macro or typedef is
>>>
>>> u_int8_t actually
>>
>> Everyone who's been talking with you about this subject has been talking
>> about std::uint8_t from <cstdint>. If you're talking about something
>> else, you need to identify what it is and where is it is declared.
>
> I'm using QT who seems to like to replicate anything its own
> way even when avoiding specific Q-prefixes.
>
> so, HERE, including
> #include <sys/types.h>
>
> I find this type (everyone might guess it is equivalent to
> the other), u_int8_t, which finally stems from a typedef to
> UNSIGNED CHAR.

Please do not change the capitalization of the characters in C++ code.
As I keep telling you, C++ is a case-sensitive language. If UNSIGNED
CHAR even exists, in principle, it's not necessarily related in any way
to unsigned char. In practice, there's likely to be a connection, but if
anyone did bother creating, for instance, macros with those names, the
connections is unlikely to be a simple one. In such as case, they
probably defined those macros precisely because there are circumstances
where they do NOT expand to "unsigned char".

Such names as u_int8_t are, as far as C++ itself is concerned, reserved
for use by user code. However, POSIX reserves identifiers ending in _t
to itself to represent types, and sys/types.h is not a C++ standard
header, so I that makes it legal for that header to declare such
identifiers. However, I can find no mention of them in the Single UNIX
standard, so I've no idea why they actually are declared. There's
certainly no good reason for using them rather than uint8_t for this
purpose.

>> Whatever is the source of this typedef, I don't think it's QT.
>
> omg ... are you saying I am just lieing for the fun of it ???

No, I'm just assuming that you're confused. Using u_int8_t rather than
uint8_t, just because such an identifier exists, tends to justify such
assumptions.

[James Kuyper]

On 11/24/19 3:53 PM, Soviet_Mario wrote:
> On 24/11/2019 20:53, David Brown wrote:

...
> What's the effect of inline linkage ...

C++ supports only three kinds of linkage: internal, external, or no
linkage (6.5p2). "inline" has it's own special rules, which interact
with the linkage rules, but it doesn't define a distinct kind of linkage.

> ... when one takes the
> address of a function (and stores it)? I assumed, without> reading documentation supporting this, It would just ignore

> inline suggestion.
> Or is it undefinite behaviour ?

The term is "undefined behavior". Note, in particular, the American
spelling - US standards organizations played a disproportionate role in
the development of both the C and C++ standards.

Taking the address of an inlined function is not prohibited by any rule
of C++.

Any function may be inlined at the point of call (unless it has features
that prevent inlining), or it can have an actual function created for it
that gets called - that's true whether or not it's declared inline. If
you take the address of an inline function, and there's any possibility
that the function might be called through that pointer, that would be
one thing that might require an actual function to be created - without
preventing inlining of other calls to that function. Note also that if
the compiler can determine that the pointer is guaranteed to point at
this particular function at the time it is dereferenced, even that call
could be inlined.

The standard does say one thing about inline functions, linkage, and
pointers that might be of interest to you:'
"If a function or variable with external linkage is declared inline in
one translation unit, it shall be declared inline in all translation
units in which it appears; no diagnostic is required. An inline function
or variable with external linkage shall have the same address in all
translation units." (10.1.6p6)

That makes it quite clear that there's nothing wrong with taking the
address of such a function.

...

>> Or perhaps you think compilers add random padding and
>> re-arrange structs in odd and unpredictable ways.

...

> I'm not sure of the degree of freedom they have in
> "reordering" members though.

"Non-static data members of a (non-union) class with the same access
control (Clause 14) are allocated so that later members have higher
addresses within a class object." (12.2p17).

Therefore, implementations have the freedom to reorder class members
only if you explicitly give them that freedom by using access control
specifiers.

> Sometimes in constructors I get some warning about the
> members will be initializated in a different order than
> declared.

That could happen through the use of access control specifiers.

[David Brown]

On 25/11/2019 19:18, James Kuyper wrote:
> On 11/24/19 3:53 PM, Soviet_Mario wrote:
>> On 24/11/2019 20:53, David Brown wrote:
> ...

> ...
>>> Or perhaps you think compilers add random padding and
>>> re-arrange structs in odd and unpredictable ways.
> ...
>> I'm not sure of the degree of freedom they have in
>> "reordering" members though.
>
> "Non-static data members of a (non-union) class with the same access
> control (Clause 14) are allocated so that later members have higher
> addresses within a class object." (12.2p17).
>
> Therefore, implementations have the freedom to reorder class members
> only if you explicitly give them that freedom by using access control
> specifiers.

Just to add a touch of confusion here - implementations can reorder
class or struct members any way they want, as long as it is invisible to
the code. gcc had an optimisation that did this for a while,
re-arranging structs for better packing or more efficient access in
code. But it could only do that for very "local" structs - nothing that
was passed around much, or where the layout could conceivably have been
visible or dictated by external requirements. It was removed before
long, however, because it was rarely possible for the compiler to make
such re-arrangements safely and the code was complicated to maintain.
(It is just another example of the "as if" rule - compilers can do what
they want with the code, as long as it works "as if" it had followed all
the rules slavishly.) Needless to say, this re-arranging would not have
occurred for the kind of code under discussion in this thread, since the
order matters here.

>
>> Sometimes in constructors I get some warning about the
>> members will be initializated in a different order than
>> declared.
>
> That could happen through the use of access control specifiers.
>

I don't think that is it. The "-Wreorder" warning in gcc, enabled by
"-Wall", gives you a warning if the order of initialisers does not match
the order of the members:

struct A {
int i;
int j;
A(): j (0), i (1) { }
};

That is because the compiler will actually initialise "i" first, then
"j" (I think the standards say this ordering is unspecified), and it
warns you that the order does not match what you wrote in the code.

[Paavo Helde]

The standard specifies the order of initialization very strictly:

"non-static data members are initialized in the order they were declared
in the class definition ([...] regardless of the order of the
mem-initializers)"

and even gives the reason for this order:

"[ Note: The declaration order is mandated to ensure that base and
member subobjects are destroyed in the reverse order of initialization.
—end note ]"

[David Brown]

Thanks for that correction.

[James Kuyper]

Yes, that was a mistake on my part. I assumed that his comment was
relevant to the previous discussion, and therefore was referring to a
warning about the order of allocation differing from the order of
declaration. There's a big difference between "allocation" and
"initialization", and I should have noticed it.

[Tim Rentsch]

James Kuyper <james...@alumni.caltech.edu> writes:

> On 11/22/19 10:49 AM, James Kuyper wrote:
>
>> On 11/22/19 4:10 AM, David Brown wrote:
>>
>>> On 22/11/2019 09:37, Keith Thompson wrote:
>>
>> ...
>>
>>>> Or an array wrapped in a struct to allow assignment. (An array of 4
>>>> uint8_t elements is guaranteed to be 32 bits, but a struct containing
>>>> such an array is not -- but it's a nearly safe assumption that there
>>>> will be no padding between the elements or at the end.)
>>>
>>> While it is true in theory that a struct can have extra padding at the
>>> end (beyond what is needed for an array of the struct to have proper
>>> alignment for all members), are there any compilers that have done so in
>>> practice?
>>
>> I can't name names, but my understanding is that there have been (and
>> might still be) implementations where all struct types have the same
>> alignment requirement as the most strictly aligned type supported by
>> that implementation. If that alignment requirement is greater than the
>> requirements of any of the member types, that will sometimes require
>> padding at the end of the struct.
>
> I said that wrong. Unless all members of the struct have the maximum
> alignment requirement, it's possible that such an implementation would
> have to put padding at the end.

Don't you mean, unless the last member of a struct has the
maximum alignment requirement, it's possible such an
implementation would have to put padding at the end.

[Tim Rentsch]

Soviet_Mario <Sovie...@CCCP.MIR> writes:

> I am trying to create a statically linked library in the said IDE.
> I would not want to include many QT features for now
> so, what is the STANDARD C++ way to get
> unsigned 8 bits integer type ?
>
> I fear that unsigned char is system dependent (and even if there are
> "wider" explicitely supported types, no plain char type seems to
> warranty to be 8 bits wide).
>
> The solution of creating an unnamed structure with a "bit field like
> struct {unsigned Pixel : 8} Mask;
> gives me a warning like : 3 byte PADDING used to align the struct
> Mask.
>
> Now I don't know whether this padding would exist also in ARRAY
> dynamic allocation (I'm not speaking for just the first address of the
> array, but for EVERY SINGLE ITEM in it, wasting as much as 3 times the
> space required !)
>
> How to get rid of such alignment feature ?
>
> in Visual Studio there were align / pad directives, but here on QT I
> dunno anything.
>
> I don't care a lot about fastness of access, but need a true compact
> byte array (QT has its own, but I will need to communicate with a
> Gambas program using the static library, so I'd be not happy to
> interface QT internals with Gambas, I'd prefer a standard C++
> types-based solution).
>
> long ago CHAR was 8 bit wide, but now it seems to be platform
> dependent, and more often "a word" wide.
>
> QT has its explicitely sized integers (even if no 8-bit sized)

I would offer an alternate set of suggestions.

First, pick a name for the type you will use in your code. For
example, in this case, you might pick 'GamByte' as the type
name you will be using.

Second, see if the library being used (Gambas) supplies a name
for its type in one of its headers. If it does, define your
type name in terms of that. These names might even be the same,
except they might be declared in different namespaces.

Third, if the library being used does not supply a name for the
type it is using, but just spells out the concrete type in its
function declarations, etc, define your type name as that, eg,

using GamByte = unsigned char; // for use with Gambas

where 'unsigned char' is meant to be the type given in Gambas
header files, assuming there is one.

Fourth, if the library being used does NOT have the type in
question appear in any of its header files, use 'unsigned char'.
If it's important to guarantee that the type is exactly eight
bits, use a static assertion that CHAR_BIT == 8. The reason
is that 'unsigned char' is guaranteed to work, if any type
is, and if for any reason it doesn't work then the error message
produced by the static assertion will be more easily understood
than, eg, a syntax error trying to use a non-existent uint8_t.

Fifth, if you never need individual 8-bit bytes in isolation, but
only in aggregates (arrays or array-like), you might consider
defining the aggregate type as a struct inside a template, which
externally could look like an array of 8-bit types even though
internally it might be, eg, 32-bit unsigned words used to hold
four 8-bit values.