Re: [comp.lang.c.moderated] valid "struct hack" in C89 (= flexible array member in C99)
Ersek, Laszlo
> Paraphrasing [...] the C99 rationale [...] in "6.7.2.1 Structure and union
> specifiers" [...]
>
> --v--
>
> struct s
> {
> int n_items;
> /* possibly other fields */
> int items[1];
> };
>
> [...] there was no way to implement the "struct hack" in C89 [...]
>
> --^--
After perusing the C FAQ 2.6 [0] and following the reference to the C89
rationale [1], I am led to believe that the above statement of the C99
rationale on C89 and the C89 rationale "3.5.4.2 Array declarators" are in
direct contradiction.
Are the struct hacks present in my C89 code nonportable, questionable,
perhaps undefined? According to the C FAQ 2.6 and the C89 rationale, the
answer is no. According to the C99 rationale's above statement on C89, the
answer is yes.
Or is that "C89" string in the citation at top a typo, and the C99
Committee actually meant "there's no way we allow C89 to bequeath the
struct hack to C99"? That would mean the struct hack is undefined in C99
*only*.
Thanks,
lacos
[0] http://www.c-faq.com/struct/structhack.html
[1] http://www.lysator.liu.se/c/rat/c5.html#3-5-4-2
--
comp.lang.c.moderated - moderation address: cl...@plethora.net -- you must
have an appropriate newsgroups line in your header for your mail to be seen,
or the newsgroup name in square brackets in the subject line. Sorry.
Ersek, Laszlo
I have just realized that the "struct hack" is undefined behavior in C89.
Paraphrasing what the C99 rationale [0] sayeth in "6.7.2.1 Structure and union
specifiers", starting with line 24:
--v--
/A new feature of C99/: There is a common idiom known as the "struct hack" for
creating a structure containing a variable-size array:
struct s
{
int n_items;
/* possibly other fields */
int items[1];
};
struct s *p;
p = malloc(sizeof(struct s) + (n - 1) * sizeof(int));
The validity of this construct has always been questionable. In the response to
one Defect Report, the Committee decided that it was undefined behavior because
the array p->items contains only one item, irrespective of whether the space
exists. [...]
The Committee felt that, although there was no way to implement the "struct
hack" in C89 [...]
--^--
a) Can somebody please provide historical context to the Defect Report in
question?
b) I readily accept that accessing p->items[1] and onwards is undefined, and
that creating pointers &p->items[2] and onwards is also undefined, but I
question the sentence >>there was no way to implement the "struct hack" in
C89<<.
For example, if we'd like to have a flexible array with element type "char
unsigned", I cannot see where the following code violates the C89 standard, ie.
introduces undefined or even unspecified or implementation-defined behavior. I
trust I'm missing something so please enlighten me.
--v--
struct s
{
size_t n_items;
char unsigned first; /* not an array! */
};
struct s *p;
size_t n;
p = malloc(sizeof *p + (n - 1u));
(&p->first)[n - 1u] = 0u;
--^--
I believe this char unsigned access is an "object representation" access (even
though C89 doesn't seem to define that concept yet), and it happens in a
contiguous, valid area. (Any (char unsigned *) pointer should be valid and
dereferrable that points to any byte of a malloc()'d region, and also the
pointer one byte past the region should be valid (but not dereferrable). How
could memcpy() work otherwise?)
Furthermore, if the element type has greater alignment requirements, like in
--v--
struct s
{
size_t n_items;
long double first; /* not an array! */
};
struct s *p;
size_t n;
p = malloc(sizeof *p + (n - 1u) * sizeof p->first);
(&p->first)[n - 1u] = 0.0L;
--^--
then I still reckon that the member named "first" ensures correct alignment for
all memory accesses in range [0 .. (n-1)].
To summarize my opinion / question, the undefined nature of the original
"struct hack" is plausible since we make a promise about the valid array
subscripts to the implementation and later violate that promise. In the
modified code examples we make no such promise, only access allocated memory
via pointers that are correctly aligned for the element type.
A short reformulation of the long double assignment could be:
*(&p->first + (n - 1u)) = 0.0L;
An overly verbose reformulation of the same assignment could be:
*(long double *)(void *)(
(char unsigned *)(void *)p
+ offsetof(struct s, first)
+ (n - 1u) * sizeof(long double)
) = 0.0L;
So which part of all this is wacky?
Thank you very much,
Laszlo Ersek
http://lacos.hu/
[0] http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf
James Kuyper
> On Wed, 16 Dec 2009, Ersek, Laszlo wrote:
>
>> Paraphrasing [...] the C99 rationale [...] in "6.7.2.1 Structure and
>> union
>> specifiers" [...]
>>
>> --v--
>>
>> struct s
>> {
>> int n_items;
>> /* possibly other fields */
>> int items[1];
>> };
>>
>> [...] there was no way to implement the "struct hack" in C89 [...]
>>
>> --^--
>
> After perusing the C FAQ 2.6 [0] and following the reference to the C89
> rationale [1], I am led to believe that the above statement of the C99
> rationale on C89 and the C89 rationale "3.5.4.2 Array declarators" are
> in direct contradiction.
>
> Are the struct hacks present in my C89 code nonportable, questionable,
> perhaps undefined? According to the C FAQ 2.6 and the C89 rationale, the
> answer is no. According to the C99 rationale's above statement on C89,
> the answer is yes.
>
> Or is that "C89" string in the citation at top a typo, and the C99
> Committee actually meant "there's no way we allow C89 to bequeath the
> struct hack to C99"? That would mean the struct hack is undefined in C99
> *only*.
I have no copy of the C89 standard, so I can't verify any statement
below which refers to that standard.
My understanding is that code which used of the struct hack technically
had undefined behavior in C89, but that it was essentially 100%
portable. The actual behavior on virtually every actual C89
implementation was exactly what the user needed it to be, in order to
make the struct hack useful. The specified section of the C89 rationale
provides an example of the struct hack without mentioning that issue,
but I think that reflects popular usage more accurately than it does the
specifications of the standard. I gather that a DR was filed on the
issue, and the committee reluctantly agreed that it is undefined
behavior; but I can't locate a citation of the specific DR.
It is still undefined behavior in C99, and I believe the relevant
wording has been made less ambiguous. It is also no longer necessary in
C99, now that flexible array members have been introduced. I would
expect that this provides compilers with slightly less incentive than
they had in C89 to allow the struct hack to work as intended. However,
since the struct hack is so popular, I imagine that most C99 compilers
will continue to allow it to work, if only for compatibility with legacy
code.
Clive D.W. Feather
> After perusing the C FAQ 2.6 [0] and following the reference to the C89
> rationale [1], I am led to believe that the above statement of the C99
> rationale on C89 and the C89 rationale "3.5.4.2 Array declarators" are in
> direct contradiction.
That is so.
> Are the struct hacks present in my C89 code nonportable, questionable,
> perhaps undefined? According to the C FAQ 2.6 and the C89 rationale, the
> answer is no. According to the C99 rationale's above statement on C89, the
> answer is yes.
Also according to WG14, when answering DR051.
What it comes to is that the Rationale is *NOT* definitive and can contain
errors. This is one of them.
Note that the response to DR051 also contains an error: the approach using
"HUGE_ARR" is invalid because the malloced space is smaller than the space
required for a "struct A". See DR073 and DR178.
<http://www.open-std.org/Jtc1/sc22/wg14/www/docs/dr_073.html>
<http://www.open-std.org/Jtc1/sc22/wg14/www/docs/dr_178.html>
There is also further material on this topic in DR072.
<http://www.open-std.org/Jtc1/sc22/wg14/www/docs/dr_072.html>
--
Clive D.W. Feather | If you lie to the compiler,
Email: cl...@davros.org | it will get its revenge.
Web: http://www.davros.org | - Henry Spencer
Mobile: +44 7973 377646
Clive D.W. Feather
> I have just realized that the "struct hack" is undefined behavior in C89.
>
> Paraphrasing what the C99 rationale [0] sayeth in "6.7.2.1 Structure and
> union specifiers", starting with line 24:
> a) Can somebody please provide historical context to the Defect Report in
> question?
The DR was DR051, available at:
<http://www.open-std.org/Jtc1/sc22/wg14/www/docs/dr_051.html>
The proposal that led to the C99 feature was:
<http://www.davros.org/c/wg14n791.txt>
> b) I readily accept that accessing p->items[1] and onwards is undefined,
> and that creating pointers &p->items[2] and onwards is also undefined, but
> I question the sentence >>there was no way to implement the "struct hack"
> in C89<<.
>
> For example, if we'd like to have a flexible array with element type "char
> unsigned", I cannot see where the following code violates the C89 standard,
> ie. introduces undefined or even unspecified or implementation-defined
> behavior. I trust I'm missing something so please enlighten me.
[...]
> char unsigned first; /* not an array! */
[...]
> p = malloc(sizeof *p + (n - 1u));
> (&p->first)[n - 1u] = 0u;
You're missing the wording in 6.3.6:
For the purposes of these operators, a pointer to a nonarray object
behaves the same as a pointer to the first element of an array of
length one with the type of the object as its element type.
If this wasn't the case, then:
int x;
*(&x + 42) = 0;
would be defined behaviour!
> (Any (char unsigned *) pointer should be valid
> and dereferrable that points to any byte of a malloc()'d region, and also
> the pointer one byte past the region should be valid (but not
> dereferrable). How could memcpy() work otherwise?)
In the case of memcpy() with third parameter n, the two pointers point to
arrays of n unsigned chars. This is significant in dealing with (for
example) the question of whether the two pieces of memory overlap.
What it comes down to is that arrays and pointers aren't the same. If the
compiler can see an array declaration in place, it is *permitted* (but not
required) to take the array size into account when compiling the code and
generate code that goes wrong when used outside the bounds of the array.
When using a pointer - unless it can deduce what that pointer points to -
it must assume that the pointer points to the largest possible object in
the system, even if that requires less efficient code. Equally, if it *can*
deduce what is being pointed to, it is entitled to take the array bounds
into account.
(This is also the rationale behind providing "int [static 5]" in parameter
declarations, as someone asked the other day.)
On many architectures this makes no difference. But, for example, if the
computer has opcodes that allow cheap indexing in the range 0 to 255, then
the compiler can use it for arrays of up to 256 bytes. When it does so, a
reference of the form "x [258]" (assuming the element type has size 1)
would be silently converted to "x [2]".