Why does offsetof need a standard-layout class?

[FrankHB1989]

The current standard specifies that if the type argument of the macro offsetof(type, member-designator) is not a standard-layout class, the results are undefined.
This restriction is looser than it in C++03(which needs a POD structure or POD union), but I think it is still too strict. If a class has to be non-standard-layout just because it has a different member ordering with standard-layout one, the offsets of non-static data member can still be determined statically. Thus it is reasonable to allow offsetof accept such types(perhaps also fit for polymorphic classes without virtual bases).

[Ville Voutilainen]

On 16 November 2012 11:46, FrankHB1989 <frank...@gmail.com> wrote:
> The current standard specifies that if the type argument of the macro
> offsetof(type, member-designator) is not a standard-layout class, the
> results are undefined.
> This restriction is looser than it in C++03(which needs a POD structure or
> POD union), but I think it is still too strict. If a class has to be
> non-standard-layout just because it has a different member ordering with
> standard-layout one, the offsets of non-static data member can still be
> determined statically. Thus it is reasonable to allow offsetof accept such

Can it? I don't think we guarantee contiguous layout unless the class has
standard layout.

[FrankHB1989]

在 2012年11月16日星期五UTC+8下午5时48分43秒，Ville Voutilainen写道：

Can't it? At least we can restrict some of the offsets can be determined during compiling(i.e. as a constant expression), whether contiguous layout is further guaranteed or not.

[Ville Voutilainen]

On 16 November 2012 11:57, FrankHB1989 <frank...@gmail.com> wrote:
>> > standard-layout one, the offsets of non-static data member can still be
>> > determined statically. Thus it is reasonable to allow offsetof accept
>> > such
>> Can it? I don't think we guarantee contiguous layout unless the class has
>> standard layout.
> Can't it? At least we can restrict some of the offsets can be determined
> during compiling(i.e. as a constant expression), whether contiguous layout
> is further guaranteed or not.

I suppose so, but I guess that requires more implementation magic, right?
Seems like an idea worth discussing in the committee.

[FrankHB1989]

在 2012年11月16日星期五UTC+8下午6时01分00秒，Ville Voutilainen写道：

Likely. But if there is no support from the standard, users who want to use optimization technique dependent on compile-time determined offsets have to play more magic, and/or leave the code non-conforming.
 

[Jean-Marc Bourguet]

Le vendredi 16 novembre 2012 10:48:43 UTC+1, Ville Voutilainen a écrit :

Can it? I don't think we guarantee contiguous layout unless the class has
standard layout.

I'm quite sure that a most derived object must be contiguous (what would be the meaning of sizeof), and I'm also sure that virtual base position (and thus the offset of their member) can vary depending on the most derived object's type and thus a subobject can be non contiguous.

What I'm not sure are the constraints in the absence of virtual bases. I'd not be surprised that formally there is no more constraints than for virtual bases but that in practice this flexibility is not used by compiler  However I'd need convincing examples that reducing the flexibility is worthwhile.

Yours,

--
Jean-Marc

[Ville Voutilainen]

Well, yes, contiguous, but with what kind of padding, and we don't guarantee
member order if there are multiple different access specifiers for members. That
bit, which was maybe-theoretically-exploited by some historical implementation,
would need implementation support for the correct implementation of offsetof.

[Alberto Barbati]

Il giorno 16/nov/2012, alle ore 11:17, FrankHB1989 <frank...@gmail.com> ha scritto:

> Likely. But if there is no support from the standard, users who want to use optimization technique dependent on compile-time determined offsets have to play more magic, and/or leave the code non-conforming.

Do such optimization techniques really exist? I'm really curious to have them explained. Could you please elaborate?

Ganesh

[FrankHB1989]

在 2012年11月17日星期六UTC+8上午1时54分08秒，Alberto Ganesh Barbati写道：

Here is my original requirement, for working on a GUI framework for some embedded devices.

Assuming there are widget classes X, Y and Z, all are subclasses of an non-virtual abstract base class called IWidget.

class IWidget
{
    // ...

    // AnyIterator is a generic iterator class using type erasure. It can be either null(default-initialized) or constructed from an iterator of any type.

    // This member is something like std::map::equal_range.
    virtual pair<AnyIterator, AnyIterator> GetChildren() = 0;
};

Now I'm going to create another superclass of IWidget called Container, which should implement the pure virtual function GetChildren.

An ideal implementation looks like:

class Container : public IWidget
{
public:
    X x; // a private child widget
    Y y; // ditto

public:
    Z z; // a public child widget

    // some code for iteration support
    static const std::size_t offsets[];
    // Out of class definition I would write: std::size_t Container::offsets = {offsetof(z), offset(x), offset(y)};
    // Obviously using 'static constexpr std::size_t offsets = {offsetof(z), offset(x), offset(y)};' in the class definition is better, but it's ill-formed as per current standard. See N3308 for details.

    // It can be other type such as std::list<IWidget>::iterator for other container class.
    // In reality it should be a class template to be reused across different containers.
    class Iterator
    {
        Container* cont;
        std::size_t idx;

        // Pointer to member object is not fot because conversions between them are not free.

    public:
        Iterator(Container&, std::size_t);

        /*constexpr*/ IWidget& operator*()
        {
            *(IWidget*)((char*)cont + Container::offsets[idx]);
        }

        Iterator& operator++()
        {
            assert(idx != sizeof(Container::offsets) / sizeof(std::size_t));

            ++idx;
            return *this;
        }
        //...
    };

public:
    Iterator begin()
    {
        return Iterator(*this, 0);
    }

    Iterator end()
    {
        return Iterator(*this, sizeof(Container::offsets) / sizeof(std::size_t)));
    }

    pair<AnyIterator, AnyIterator> GetChildren() override
    {
        return std::make_pair(begin(), end());
    }

    //...
};

If N3308 (hopefully) resolves the problem on use of constexpr, a lot of operations for the class Container::Iterator can be certainly optimized away.
But it is impossible to use this implementation and keep the code standard-conforming, due to the use of non-standard-layout type as offset arguments.

Of course I can use pointers like X* x instead of directly using the widget type objects as data members to avoid this problem, but it is also troublesome, as http://www.ultimatepp.org/www$uppweb$overview$en-us.html section "Who_owns_widgets" demonstrated. There should be no real need to call 'new X'.
Adding redundant pointers as non-static data members is bad for performance. It also needs more code. (Try thinking about there would be 30 or more, not 3, widgets in the container...Hmm, nasty initialization.)


 

[FrankHB1989]

在 2012年11月17日星期六UTC+8下午2时02分07秒，FrankHB1989写道：

    // Out of class definition I would write: std::size_t Container::offsets = {offsetof(z), offset(x), offset(y)};
    // Obviously using 'static constexpr std::size_t offsets = {offsetof(z), offset(x), offset(y)};' in the class definition is better, but it's ill-formed as per current standard. See N3308 for details.

Sorry for typos. 'offsetof(z), offset(x), offset(y)' should be 'offsetof(Container, z), offsetof(Container, x), offsetof(Container, y)', respectively.

[FrankHB1989]

在 2012年11月17日星期六UTC+8下午2时02分07秒，FrankHB1989写道：

    // Out of class definition I would write: std::size_t Container::offsets = {offsetof(z), offset(x), offset(y)};
    // Obviously using 'static constexpr std::size_t offsets = {offsetof(z), offset(x), offset(y)};' in the class definition is better, but it's ill-formed as per current standard. See N3308 for details.

[Nicol Bolas]

Why it is that you want this is more or less irrelevant. `offsetof` is currently a very simple macro. What you're wanting can't be a simple macro, because it needs to do very different things for non-standard-layout types. It would have to be a language feature (since you can't pass types to functions and identifiers to functions).

At which point, you're now talking about something very close to compile-time reflection and iteration over members of a class. Which is best handled by providing actual compile-time reflection, rather than a mere `offsetof` macro.

[FrankHB1989]

在 2012年11月17日星期六UTC+8下午2时27分09秒，Nicol Bolas写道：

Why it is that you want this is more or less irrelevant. `offsetof` is currently a very simple macro. What you're wanting can't be a simple macro, because it needs to do very different things for non-standard-layout types. It would have to be a language feature (since you can't pass types to functions and identifiers to functions).

At which point, you're now talking about something very close to compile-time reflection and iteration over members of a class. Which is best handled by providing actual compile-time reflection, rather than a mere `offsetof` macro.

You are right. The real problem is beyond the title, and is not only concerned with offsetof itself. Something like __builtin_offset with implementation magic must be provided for this purpose.
But this time I don't need a full set of compile-time reflection features. I think there exists a superset of standard layout classes could be the type argument of the simple macro offsetof, and hence the results could be not undefined. If polymorhpic classes(without virtual bases) is a subset of this intended superset, my original problem would be solved. Even if not, its existence is still reasonable. What do you think about the difficulty to clarify it? Is there any implementation using run-time determined member offset within all non-standard-layout classes?

[Nicol Bolas]

It's not about compile-time vs. runtime determination of the offset. It's about how the offset is determined. The computation for non-standard layout classes may not work for standard layout ones. Thus, you would have to detect whether the type passed to the macro is standard layout and use the proper offset. That's not possible with a macro, so it would have to be a language feature.

So what is this "superset" you speak of? How would you define this classification, and based on what would you do so?

[FrankHB1989]

I found another example hacking offsetof:
http://locklessinc.com/articles/flexible_lists_in_cpp/

[FrankHB1989]

在 2012年11月17日星期六UTC+8下午4时24分23秒，Nicol Bolas写道：

Being standard-layout is a so strict restriction.
Consider a class has to be non-standard-layout just only because it doesn't have the same member access control.
Is the way to determine its member offsets should be different with the way for a standard-layout class? And how the different computation method for it can be in actual implementations?

[FrankHB1989]

在 2012年11月17日星期六UTC+8下午4时43分33秒，FrankHB1989写道：

Is the way to determine its member offsets should be different with the way for a standard-layout class?

Sorry for editing and stupid grammar mistake...

Is the way to determine its member offsets should be different with the way for a standard-layout class? -> Should the way to determine its member offsets be different with the way for a standard-layout class?

[Alberto Barbati]

Did you consider using pointer-to-members instead of integral offsets?

Ganesh

[Nicol Bolas]

Who's to say whether it "should" or not? The Standards committee could have enforced layout rules for such types, but they explicitly did not. This allows implementations the freedom to re-order the members of a non-standard layout struct as it sees fit. It can group all the public and private members together if it wants to.

The way I see it, if you're doing encapsulation, there's no reason to make any variable public. If you need something to be publicly modifiable and accessible, just use accessors.

[FrankHB1989]

在 2012年11月19日星期一UTC+8下午3时23分19秒，Alberto Ganesh Barbati写道：

Yes, I did. But I found that a pointer-to-member type such as X&(Container::*) cannot be static_cast to IWidget&(Container::*) in this scenario. I didn't find any text in the standard that guarantees I can reinterpret_cast a value of X&(Container::*) to IWidget&(Container*::) and use operators->* to get expected IWidget& results, either.

[FrankHB1989]

在 2012年11月19日星期一UTC+8下午4时15分12秒，Nicol Bolas写道：

The standard would use "shall" in most cases, and might use "should" seldom. Others can say "should" in a suggestion which is believed to lead to expected results. But here the use of "should" is in a question. I thought it is questionable because I found no sufficient and explicit reasons imply loosing the restriction is forbidden or impossible. The committee did not enforced layout rules in the formal standard, but they could change them in future, if the change can be proved to be necessary/beneficial enough and there are few difficulties or negative effects. I suggested a change of adding a new permissive category of class types and I think it is probably acceptable, though I'm not confident if it is the proper(namely "necessary/beneficial enough") way.
Encapsulation or accessibility is not the key point. Using accessors here would likely bloat the framework code a lot and make it far less flexible.

[Nicol Bolas]

It should be noted that the place for discussing proposals (which this is, since it isn't in the standard) is elsewhere.

That being said, the standard-layout/trivial rules follow existing practice. Many compilers didn't have to change much to support them (outside of the machinery needed to provide is_standard_layout and so forth). The change you're proposing would be an actual change to compilers and how they lay out objects in memory.
 

Encapsulation or accessibility is not the key point. Using accessors here would likely bloat the framework code a lot and make it far less flexible.

If accessibility was not a key point, then why do you need to make some of the members public and some not? And I've never seen encapsulation or accessors make something less flexible; I'm curious as to how that would work.

[FrankHB1989]

It should be noted that the place for discussing proposals (which this is, since it isn't in the standard) is elsewhere.

That being said, the standard-layout/trivial rules follow existing practice. Many compilers didn't have to change much to support them (outside of the machinery needed to provide is_standard_layout and so forth). The change you're proposing would be an actual change to compilers and how they lay out objects in memory.

I know the place for discussing proposals. But it isn't a proposal, but just an idea. I can't work out a proposal now, because I am not 100% sure it's feasible, as you said, to make them "follow existing practice".
Perhaps you misunderstood my idea. I strongly suspect there is some work can be done conceptually, like changing POD to standard-layout/trivial, with almost no actual change to compilers. The only things to implement should be: a) adding an interface(something like std::is_standard_layout) to check if a type is fit for the proposed category; b) changing implementation of offsetof if necessary, and confirm there are no redundant diagnostic messages. I'm glad to see how compilers have to make actual change of object layout. 

If accessibility was not a key point, then why do you need to make some of the members public and some not? And I've never seen encapsulation or accessors make something less flexible; I'm curious as to how that would work.

Here different access control of data members are used to show the enclosing class is not standard-layout. Even if all the members are standard-layout types, no base or virtual member function exists, it is still the problem. The example is more closed to the actual code to illustrate the original problem and to try getting some other ways unrelated to object layout can solve it.
Excessive encapsulating doesn't make sense. It exposes no extra functionality but often force the client code to be cumbersome. Same amount of code using excessive encapsulated interface would do less work than proper one. If there are both simple public getter and setter for a data member, and no need to work with source code tools depend on accessors, I'd rather like make the data member itself to be public, not to add accessors. (Though I do use accessors if necessary.)
And here accessors don't help at all: I need to get/access/traverse children via a virtual function in the abstract base class, but different derived classes can hold different children and accessors.

[Nicol Bolas]

On Tuesday, November 20, 2012 4:24:26 AM UTC-8, FrankHB1989 wrote:

It should be noted that the place for discussing proposals (which this is, since it isn't in the standard) is elsewhere.

That being said, the standard-layout/trivial rules follow existing practice. Many compilers didn't have to change much to support them (outside of the machinery needed to provide is_standard_layout and so forth). The change you're proposing would be an actual change to compilers and how they lay out objects in memory.

I know the place for discussing proposals. But it isn't a proposal, but just an idea. I can't work out a proposal now, because I am not 100% sure it's feasible, as you said, to make them "follow existing practice".

It's also the place for working out ideas that would become proposals.

This place is for discussing the standard as it currently is. Like asking if an initializer list is a type.
 

Perhaps you misunderstood my idea. I strongly suspect there is some work can be done conceptually, like changing POD to standard-layout/trivial, with almost no actual change to compilers. The only things to implement should be: a) adding an interface(something like std::is_standard_layout) to check if a type is fit for the proposed category; b) changing implementation of offsetof if necessary, and confirm there are no redundant diagnostic messages. I'm glad to see how compilers have to make actual change of object layout.

I understand that you suspect that, but why do you suspect that? Do you have real knowledge of multiple compilers to know that this is true?

If accessibility was not a key point, then why do you need to make some of the members public and some not? And I've never seen encapsulation or accessors make something less flexible; I'm curious as to how that would work.

Here different access control of data members are used to show the enclosing class is not standard-layout. Even if all the members are standard-layout types, no base or virtual member function exists, it is still the problem. The example is more closed to the actual code to illustrate the original problem and to try getting some other ways unrelated to object layout can solve it.

The point of having a division between "standard layout" and "not standard layout" is to give compilers the freedom to rearrange members where they see fit on objects where that could matter. There are three cases that dissallow standard layout currently:

Different access control for different data members. This allows compilers the freedom to put all of the public members first and all of the private ones later. Or vice-versa.

Base and Derived classes both having data members. This allows compilers the freedom to put the base class before the derived class, or to put the derived class before the base class. And I know for a fact that there are some compilers that do it one way and some that do it another. I've been involved in code that has to work with both while allowing for a form of serialization. You're not going to get the committee to force some compilers to put their base classes first or something.

Virtual member functions and/or virtual base classes. This allows compilers the freedom to implement virtual functions/bases as it sees fit. You're not going to get the committee to force compilers to adopt the vtable as the only way to do virtual dispatch, nor are you going to get them .

I focus on standard-layout because it's the only way that you can have a consistent enough layout scheme for offsetof to work. If you can't expand the defintion of standard-layout to cover these cases, then you're not going to be able to get an offsetof macro to work.

The most you might get the committee to go along with is allowing the access controls to be standard layout. And that's only a maybe.
 

Excessive encapsulating doesn't make sense. It exposes no extra functionality but often force the client code to be cumbersome. Same amount of code using excessive encapsulated interface would do less work than proper one. If there are both simple public getter and setter for a data member, and no need to work with source code tools depend on accessors, I'd rather like make the data member itself to be public, not to add accessors. (Though I do use accessors if necessary.)

This is more of a code style discussion, but this kind of coding runs into a problem the moment you want to *change* how data is stored. You modify the type of a member or split it into two members or whatever. That now means that you have to change everyone who uses it, with no real fall-back or backwards compatibility.

Religiously using accessors allows you to use the same interface but with different back-end data. This is a good 30% of the reason for why encapsulation is good. As to how "cumbersome" it makes the client, that is a matter of opinion.
 

And here accessors don't help at all: I need to get/access/traverse children via a virtual function in the abstract base class, but different derived classes can hold different children and accessors.

You're not getting the committee to allow virtual classes to be standard-layout. The distinction was made specifically to allow compilers the freedom to do what they feel is best for derived classes and virtual dispatch.

[Olaf van der Spek]

Op dinsdag 20 november 2012 17:52:57 UTC+1 schreef Nicol Bolas het volgende:

Excessive encapsulating doesn't make sense. It exposes no extra functionality but often force the client code to be cumbersome. Same amount of code using excessive encapsulated interface would do less work than proper one. If there are both simple public getter and setter for a data member, and no need to work with source code tools depend on accessors, I'd rather like make the data member itself to be public, not to add accessors. (Though I do use accessors if necessary.)

This is more of a code style discussion, but this kind of coding runs into a problem the moment you want to *change* how data is stored. You modify the type of a member or split it into two members or whatever. That now means that you have to change everyone who uses it, with no real fall-back or backwards compatibility.

Religiously using accessors allows you to use the same interface but with different back-end data. This is a good 30% of the reason for why encapsulation is good. As to how "cumbersome" it makes the client, that is a matter of opinion.

Wouldn't properties solve this? 

[Nicol Bolas]

Properties are just syntactic sugar for accessors. That's not to say that I'm against their inclusion, but I would be against some kind of library kludge (as some have suggested) instead of as a proper language feature.

Properties would solve this to the extent that accessors are "cumbersome" to the client. So for me, it's more of a lateral move.

[FrankHB1989]

It's also the place for working out ideas that would become proposals.

This place is for discussing the standard as it currently is. Like asking if an initializer list is a type.

 

The question is about "why the restriction should be as it currently is, and can't be looser". The idea is a complement of the latter clause.
But if what you said is totally true, the idea need not to be a proposal...

I understand that you suspect that, but why do you suspect that? Do you have real knowledge of multiple compilers to know that this is true?

 
I do know reordering the members of non-standard-layout classes is "mandated" freedom for implementations currently, but I haven't seen or heard a compiler with sophisticated layout strategies which force the offsets of member unknown or not existent meaningfully at compile-time for a class without virtual bases. Could you please show me some examples?
 

Base and Derived classes both having data members. This allows compilers the freedom to put the base class before the derived class, or to put the derived class before the base class. And I know for a fact that there are some compilers that do it one way and some that do it another. I've been involved in code that has to work with both while allowing for a form of serialization. You're not going to get the committee to force some compilers to put their base classes first or something.

Could you please tell me what compilers you used then?
And for the same class without virtual bases, same member designator and same compiler, is it possible that the result of offsetof to be variable among different programs or different runs of the same program?
 

Virtual member functions and/or virtual base classes. This allows compilers the freedom to implement virtual functions/bases as it sees fit. You're not going to get the committee to force compilers to adopt the vtable as the only way to do virtual dispatch, nor are you going to get them .

I focus on standard-layout because it's the only way that you can have a consistent enough layout scheme for offsetof to work. If you can't expand the defintion of standard-layout to cover these cases, then you're not going to be able to get an offsetof macro to work.

I'd expected "even if not" for virtual stuff. Why it's the only way that make offsetof work? Could you tell me what is the difficulty to keep it consistent enough?
 

The most you might get the committee to go along with is allowing the access controls to be standard layout. And that's only a maybe.

I don't think so. Maybe we can also use different wording on offsetof, to allow some results for non-standard-layout classes not undefined, at least implementation-defined.
 

This is more of a code style discussion, but this kind of coding runs into a problem the moment you want to *change* how data is stored. You modify the type of a member or split it into two members or whatever. That now means that you have to change everyone who uses it, with no real fall-back or backwards compatibility.

 

Religiously using accessors allows you to use the same interface but with different back-end data. This is a good 30% of the reason for why encapsulation is good. As to how "cumbersome" it makes the client, that is a matter of opinion.

This is a design issue. I know the consistency on accessing all of the data members is somewhat nice, probably friendly to refactoring and so on, but once I'm confident to keep the interface invariant, I'd like to use it on my own risk.
 

You're not getting the committee to allow virtual classes to be standard-layout. The distinction was made specifically to allow compilers the freedom to do what they feel is best for derived classes and virtual dispatch.

 It's just bad for applying a workaround containing undefined behavior that suffered by more than only one specific implementation.

[Gordon Woodhull]

Another use case is intrusive containers like Boost.Intrusive, which store the bookkeeping data of the container inside the items, and then use offsetof and similar techniques to get object pointers from member pointers. (Think of it as the inverse of pointer-to-member.)

Currently that library has code that exploits UB, very reliably for a broad range of compilers AFAIK. I presume it's automatically gotten more standard compliant with the advent of "standard layout".

[Bb Hh]

Note one more thing. I realized CWG597: "it is not clear why it is forbidden to bind a reference to a non-virtual base class of an out-of-lifetime object, as that is just an address offset calculation. " Is the calculation "trivial" enough?