class black and white, separate interface from content

[mobiphil]

Hi there, 

maybe somebody can join and help with this following brainstorming, may sound naive, but would love to hear opinions. 

Two new keywords: (white and black), or (iface and impl) or anything else, I will go for the moment with two rather neutral keywords black and white. 

Black is for blackbox, white is for whitebox. Black is for what you hide, white is what you show. Black is really private, white is really public. No more pimpl, guaranteed fast compilation times, less recompilation and the list does not stop here.

// content of A.h

class white A {

   /* here come all public virtuals and non virtuals */

   /* we could also allow public data members */

};

// content of A.cpp

class black A {

   /* here come all data members  */

   /* here come all the private methods */

};

/* how would I solve */

A * = new A();

what do we need for allocation? The size of the data members from white and black? Compiler can either generate a classsize() function or with -fPIC trick, put the size into data section, so that it can be available at runtime. Advantage of a classize() function is that like this, classes can have an easier life in shared libraries (plugin pattern for instance). 

What about allocating on the stack? Do the same in alloca style. 

What about subclassing? Well here are two problems: the size and the offsets. 

Subclass size: For the subclass the classize will be it's size plus the data it defines additionally. 

Offsets: we either force all the public data members into the blackbox (so only private data members, thus defined in black), and outside world does not have to know about offsets, everything will be accessed only through getters/setters. Or one can use the normal offset calculation like for normal classes, one can make the convention that "white" members are at the beginning of the object. The problem would be a bit tricky with subclasses. But also here the compiler can calculate offests as: offset in subclass + size of baseclass.

thanks for your feedback

[Douglas Boffey]

While the concept sounds good on the surface, there seems an awful lot
of baggage associated with it.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> Visit this group at
> http://groups.google.com/a/isocpp.org/group/std-proposals/.
>

[Douglas Boffey]

Have you had a look at the work done by the Modules SG?

[mobiphil]

While the concept sounds good on the surface, there seems an awful lot
of baggage associated with it.

I know... but that is the objective of this proposed brainstorming, to name all the elements... What I forgot to mention is that obviously using the keywords thus splitting the class would be obviously optional 

[mobiphil]

Have you had a look at the work done by the Modules SG?

Yes, but they are almost orthogonally different, though their effect may overlap. I am hacking right now the modules implementation of clang.

[Jean-Marc Bourguet]

If you go that road, I suggest looking at Modula 3 and its "partial revelations", ie the possibility for a component to provides different interfaces to different customers.

Yours,

-- 

Jean-Marc

[mobiphil]

If you go that road, I suggest looking at Modula 3 and its "partial revelations", ie the possibility for a component to provides different interfaces to different customers.

was not aware about this, skimmed the article (link below), will read it more carefully at a later time, but it seems that it is a generalization of what I just proposed.

What is ill in C++ is that one tried to push everything into one class declaration. I was thinking also about including in the same proposal the fact that non virtual functions declarations could be easily removed from the class declaration and allow anywhere just Class::method() {}.. kind of declarations/definitions. This would allow moving of different details of a class into different header. Obviously This does not mean that one wants to create a class that can do everything.

  The "black" I proposed, could be split into several blacks, the class black idiom could be regarded as even superfluous, and just allow "int Class::member ;" declaration anywhere. The linker could gather all the parts of the classes and reason about it. But this later detail is a bit extreme, I agree.

http://www.cs.tut.fi/lintula/manual/modula3/modula-3/html/partial-rev/index.html

[Tom Honermann]

On 01/22/2015 06:19 PM, mobiphil wrote:
> maybe somebody can join and help with this following brainstorming, may
> sound naive, but would love to hear opinions.

Are you familiar with Objective-C? If not, I recommend researching
Objective-C interfaces, implementations, categories, and class
extensions to help flesh out your idea.

What would sizeof(A) produce in a translation unit that observes only
the interface (white) declaration? Would this require a dynamic sizeof
similar to that required for C99 variable length arrays? (I think the
need for a dynamic sizeof is one of the issues that have prevented
variable length arrays being added to C++).

Would class data members of these split class types be allowed when only
the interface (white) declaration is present in a translation unit? If
so, what would be the result of sizeof(X) for such a class?

Tom.

[mobiphil]

Thanks for joining 

> maybe somebody can join and help with this following brainstorming, may
> sound naive, but would love to hear opinions.

Are you familiar with Objective-C?  If not, I recommend researching
Objective-C interfaces, implementations, categories, and class
extensions to help flesh out your idea.

Have some very basic notion. I would not spend too much time for further study if there is zero acceptance for such ideas. 

What would sizeof(A) produce in a translation unit that observes only
the interface (white) declaration?

 Would this require a dynamic sizeof
similar to that required for C99 variable length arrays?  (I think the
need for a dynamic sizeof is one of the issues that have prevented
variable length arrays being added to C++).

Compilation error for static sizeof. If really needed for runtime, then replace with a pointer to some global text that would be solved by the linker (like fpic). Or go for the function Class::sizeof() that would be automatically generated by the compiler.

 

Would class data members of these split class types be allowed when only
the interface (white) declaration is present in a translation unit?  If
so, what would be the result of sizeof(X) for such a class?

static sizeof(x), does not make sense, we both agree. I am not supposed to want know the color, shape etc. of an object that does not exist.

If you do not provide the blackbox, you will have linker error, the same way you would have for missing vtables. This linker error would be "implemented" either by missing Class::size() function or Class::size global variable.

 

[Thiago Macieira]

On Friday 23 January 2015 08:05:25 mobiphil wrote:
> static sizeof(x), does not make sense, we both agree. I am not supposed to
> want know the color, shape etc. of an object that does not exist.
> If you do not provide the blackbox, you will have linker error, the same
> way you would have for missing vtables. This linker error would be
> "implemented" either by missing Class::size() function or Class::size
> global variable.

Without a static sizeof, you can't:

- use operator new
- use make_shared or make_unique
- declare the object on the stack
- use it as a member of another object

The only way you can create such an object is to call an allocator function.

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
PGP/GPG: 0x6EF45358; fingerprint:
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358

[mobiphil]

I thought I touch all the subjects, but I will comment once more

 

Without a static sizeof, you can't:

 - use operator new

Not if you do not separate the concepts. When does new happen? At runtime. Do I know the size of the object at runtime yes. Proposed here either a function that the compiler would generate, or a global that would hold the size of the object deduced based on the size of "black" (+ size of white if I want to allow data in the white)

 

 - use make_shared or make_unique

Well, did not look into this one, but not all objects or classes fit into all patterns. Non copiables do not fit into patterns that need copiable etc...

 

 - declare the object on the stack

Yes you could, as mentioned earlier, do alloca based on the dynamic size mentioned above 

 - use it as a member of another object 

Yes you could. The size and offset needed though would be calculated at runtime with similare mechasism like -fpic

[Thiago Macieira]

On Friday 23 January 2015 10:42:28 mobiphil wrote:
> > - use it as a member of another object
>
> Yes you could. The size and offset needed though would be calculated at
> runtime with similare mechasism like -fpic

More like virtual inheritance instead of position-independent code.

[mobiphil]

> Yes you could. The size and offset needed though would be calculated at
> runtime with similare mechasism like -fpic

More like virtual inheritance instead of position-independent code.

What do you mean?

What I mean with "similar mechanism like -fPIC": offsets and size will be known at runtime by pointing to a global location that will be resolved by the linker

[Thiago Macieira]

Also possible, but just not the way I envisioned it. Loading it from a very
different page of memory may be slower than keeping it local and just
referencing each dynamic item by a pointer, like a virtual base.

[mobiphil]

Also possible, but just not the way I envisioned it. Loading it from a very
different page of memory may be slower than keeping it local and just
referencing each dynamic item by a pointer, like a virtual base.

I had this in mind. I am trying to think both in terms of dynamic and static linking. Dynamic linking is probably bringing more complex, I was rather thinking in terms of it. Without going into details dynamic linking with current compilers/linkers can be mainly done in two ways: either intrusive or non intrusive. The linker may go to the code and patch function addresses, or use the function wrappers. (sorry if I do not use the right terminology). All these are implementation details that have their advantages/disadvantages. As most of shared libraries are implemented with -fpic, for the sake of simplicity I mentioned this one. I am perfectly aware of the expense. Patching references at dynamic linking time would be another solution. Patching all the references has the disadvantage that you may patch 99% of references that are not used, slowing down thus dynamic linking. If it is static linking the linker could patch it at linking time.

Referencing each dynamic item by a pointer : you are proposing that dynamic items shall be allocated separately and keep a pointer like in the case of opaque pointers? Do not think this is a solution. These pointers equally can fall outside of the page and have same cache issues you are touched or?  For me it is important that the object itself stays compact. The question is what is the best solution to access members of the "white". The most straightforward would be for the sake of simplicity to forbid data members in the white, and really keep it like an interface.

[Thiago Macieira]

On Friday 23 January 2015 16:04:55 mobiphil wrote:
> Referencing each dynamic item by a pointer : you are proposing that dynamic
> items shall be allocated separately and keep a pointer like in the case of
> opaque pointers?

No, I am not proposing that. First of all, I am not proposing anything. I
personally don't think your proposal will fly...

But if it were to be implemented, I'd expect it to be done like a class with
virtual bases: the allocator allocates a block of memory with sufficient size
for all of the dynamic objects inside, which it calculates at runtime by
adding up the sizes of each one, and initialises a helper pointer to those
objects. This extra pointer is an implementation detail and is hidden from the
user, just like a virtual base's pointer.

The other solution is to create a "dynamic table" of the class, which like the
virtual table, is associated one per class. This table would contain the size
of the full class and the offsets of each of the dynamic-sized members. The
problem here is initialising this table, since it can only be done at runtime,
it has to be done on program load.

[Douglas Boffey]

The other option would be for the compiler to produce an interface
file containing the necessary details.

[Douglas Boffey]

... and references that when compiling the dependent TU.

On 1/24/15, Thiago Macieira <thi...@macieira.org> wrote:

[mobiphil]

> Referencing each dynamic item by a pointer : you are proposing that dynamic
> items shall be allocated separately and keep a pointer like in the case of
> opaque pointers?

No, I am not proposing that. First of all, I am not proposing anything.

Well, suggesting, writing, proposing, I think you get it... 

 

personally don't think your proposal will fly...

That is a bit negative. Seeing how slowly C++ developed, and how rigid it is, that would not surprise me. Unfortunately most of industries failed to influence the evolution of C++. Such a feature would have saved for most of projects I worked lot of dollars. I see in your signature that you work in kde area, thus probably you deal lot with QT. I wonder if having such feature QT would have bothered with opaque pointers.

 

But if it were to be implemented, I'd expect it to be done like a class with
virtual bases: the allocator allocates a block of memory with sufficient size
for all of the dynamic objects inside,

sufficient you mean allocated can exceed much the size of all dynamic objects?

 

which it calculates at runtime by
adding up the sizes of each one, and initialises a helper pointer to those
objects.

Sorry: one pointer to several objects, is that correct? Who would store this pointer, how would it work?

 

The other solution is to create a "dynamic table" of the class, which like the
virtual table, is associated one per class. This table would contain the size
of the full class and the offsets of each of the dynamic-sized members. The
problem here is initialising this table, since it can only be done at runtime,
it has to be done on program load.

You get it... This is exactly what I did mean with the -fpic like mechanism, was bit lazy to describe. 

But you made a good point and will link it with your previous observation about having this information in a different memory page that will probably cause cache miss. Well unfortunately virtual tables suffer of the same problem. So we arrived to the point: this information could be part of the virtual pointer table.

Another solution would be (at least for the beta :) ) to forbid data members in the interface. The dynamic linker or the linker (I know, linkers do not have such tasks) could go only through the code that knows about the black, and patch it with right offset, as he would already know about the blacks of superclases.

[mobiphil]

The other option would be for the compiler to produce an interface
file containing the necessary details.

I am afraid that would still introduce compile time dependencies. Once you changed the content of the black, that would regenerate the interface thus force the dependees to recompile.

 

[Thiago Macieira]

On Saturday 24 January 2015 03:37:39 mobiphil wrote:
> > But if it were to be implemented, I'd expect it to be done like a class
> > with
> > virtual bases: the allocator allocates a block of memory with sufficient
> > size
> > for all of the dynamic objects inside,
>
> sufficient you mean allocated can exceed much the size of all dynamic
> objects?

Exceeds yes. By the padding amount and the extra pointers.

>
> > which it calculates at runtime by
> > adding up the sizes of each one, and initialises a helper pointer to those
> > objects.
>
> Sorry: one pointer to several objects, is that correct? Who would store
> this pointer, how would it work?

Let's say Foo and Bar are dynamic in the object below:

class MyObject : public virtual SomeOther
{
Foo foo;
int i;
int j;
void *d;
Bar bar;
virtual ~MyObject();
};

If it's implemented as I described, the real object's ABI is:

struct MyObject_real
{
MyObject_vtable *_vptr;
SomeOther *_vbase;
Foo *foo;
Bar *bar;

int i;
int j;
void *d;

SomeOther _vbase_space;
// plus space for Foo and Bar
};

The code that allocates MyObject needs to take sizeof(MyObject_real) and add
the dynamic sizes of Foo and Bar. The difference between that and the allocated
space for the virtual base is that the size of the former isn't known at
compile time.

If you do object.foo.x, it's implemented as real(object).foo->x.

> > The other solution is to create a "dynamic table" of the class, which like
> > the
> > virtual table, is associated one per class. This table would contain the
> > size
> > of the full class and the offsets of each of the dynamic-sized members.
> > The
> > problem here is initialising this table, since it can only be done at
> > runtime,
> > it has to be done on program load.
>
> You get it... This is exactly what I did mean with the -fpic like
> mechanism, was bit lazy to describe.
> But you made a good point and will link it with your previous observation
> about having this information in a different memory page that will probably
> cause cache miss. Well unfortunately virtual tables suffer of the same
> problem. So we arrived to the point: this information could be part of the
> virtual pointer table.

This requires careful research by the ABI developers to see which solution is
more efficient. My educated guess is that mine above is more cache- and
instruction-efficient, but I have not (and have no intention to) actually
benchmarked this.

Your solution requires loading a ptrdiff_t from a known, global location in
memory, then adding it to your class's base pointer. With -fPIC, this becomes
a double indirection: you need to load a pointer from the GOT and then load
the ptrdiff_t from the address the GOT contained, only then add it.

And then there's the initialisation of this table. For a single dynamic-sized
object, the value can be emitted by the compiler at compile time. For an
object that contains other dynamic-sized objects, the compiler can't do it, so
it has to emit load-time code to calculate this and save in the "dynamic
table". I don't want to imagine the problems caused by getting the
initialisation order wrong.

[mobiphil]

struct MyObject_real 

{
        MyObject_vtable *_vptr;
        SomeOther *_vbase;
        Foo *foo;
        Bar *bar;

        int i;
        int j;
        void *d;

        SomeOther _vbase_space;
        // plus space for Foo and Bar
};

I see, but personally I would not be happy seeing the space wasted for pointers especially not 64 bit. I am thinking to make another proposal about 64bit pointer compression. Eventually offsets would do it. Why should I store the layout of each object in each object. And that is what I mean by "this information could be part of the virtual pointer table".  It is true that vtable is not present for all classes, but well, there will be another reason for the existence of vtable.

 

The code that allocates MyObject needs to take sizeof(MyObject_real) and add
the dynamic sizes of Foo and Bar. The difference between that and the allocated
space for the virtual base is that the size of the former isn't known at
compile time.

Regarding layout I think the base should come first. Isn't it implemented like that. Casting normally keeps the address, so base class data should be at lower addresses. 

Well, I think better not to mix -fpic anymore and mix dynamic and static linking for the moment. When I wrote the first message, did not think about virtual tables. Probably the cleanest is to say size and offset (layout)  information will be available at runtime in vtable or vtable alike structure.

[Thiago Macieira]

On Saturday 24 January 2015 10:00:55 mobiphil wrote:
> Regarding layout I think the base should come first. Isn't it implemented
> like that. Casting normally keeps the address, so base class data should be
> at lower addresses.

Not for virtual bases. The virtual base is kept after the non-virtuals.

[mobi phil]

so, this topic can be forgotten, nobody bites? ;)

[Thiago Macieira]

On Friday 06 February 2015 18:15:02 mobi phil wrote:
> so, this topic can be forgotten, nobody bites? ;)

You haven't written a proposal.

[mobi phil]

On Fri, Feb 6, 2015 at 11:22 PM, Thiago Macieira <thi...@macieira.org> wrote:

On Friday 06 February 2015 18:15:02 mobi phil wrote:
> so, this topic can be forgotten, nobody bites? ;)

You haven't written a proposal.

Sorry, do not understand this. How would you pretend there was no proposal while you reacted to it yourself....

[Thiago Macieira]

I meant that you did not write a paper and submit to the committee for
consideration.

I think your idea has merit. It makes sense to present it and talk to the
people doing Modules. Some aspects of it could be adopted there.

[mobi phil]

On Saturday 07 February 2015 00:57:53 mobi phil wrote:
> On Fri, Feb 6, 2015 at 11:22 PM, Thiago Macieira <thi...@macieira.org>
>
> wrote:
> > On Friday 06 February 2015 18:15:02 mobi phil wrote:
> > > so, this topic can be forgotten, nobody bites? ;)
> >
> > You haven't written a proposal.
>
> Sorry, do not understand this. How would you pretend there was no proposal
> while you reacted to it yourself....

I meant that you did not write a paper and submit to the committee for
consideration.

I think your idea has merit. It makes sense to present it and talk to the
people doing Modules. Some aspects of it could be adopted there.

While I do not know exactly the formal procedure, I do not think it is worth the effort to write a formal paper unless some of the honorable members of the honorable committee would show minimum interest to ever consider it. The reason of the brainstorming is to find together weaknesses of the model and solutions for them.

[gmis...@gmail.com]

I agree

[Ville Voutilainen]

On 7 February 2015 at 16:06, <gmis...@gmail.com> wrote:
>> While I do not know exactly the formal procedure, I do not think it is
>> worth the effort to write a formal paper unless some of the honorable
>> members of the honorable committee would show minimum interest to ever
>> consider it. The reason of the brainstorming is to find together weaknesses
>> of the model and solutions for them.
> I agree

As far as I understand it, the idea deals with being able to express
interface-implementation splits in a way that doesn't incur size/abstraction
penalty. Some aspects of that are handled by modules, some probably
aren't. Overall, the "in size" "physical design" aspect of having to
state the types of members in a class definition can indeed be problematic
in the sense that you have to commit to the types in a class definition
regardless of whether the types are ever used by clients. There are
work-arounds(*) for that, so the question is whether language support
for such a facility is truly necessary and/or worth the cost of it.

(*) The work-arounds aren't necessarily easy - having a data buffer
in a class and having pimpl-side implementations know the actual
type of objects stored in such a buffer require proper alignment
and proper lifetime handling, which can be non-trivial.

[mobi phil]

On 7 February 2015 at 16:06,  <gmis...@gmail.com> wrote:
>> While I do not know exactly the formal procedure, I do not think it is
>> worth the effort to write a formal paper unless some of the honorable
>> members of the honorable committee would show minimum interest to ever
>> consider it. The reason of the brainstorming is to find together weaknesses
>> of the model and solutions for them.
> I agree


As far as I understand it, the idea deals with being able to express
interface-implementation splits in a way that doesn't incur size/abstraction
penalty. Some aspects of that are handled by modules, some probably
aren't.

If you change the class definition, you need to recompile module + everything depends on module.
If you split the stuff into to, you need to recompile only if you change the real interface towards the rest of the world.

In my experience you spend 90% of time doing useless recompiles just because you need to ad a damm' private method. If it is private why do I need to recompile billions of lines of code in client code?

While this is not the only benefit the proposal would bring, it is the only one that would have some way to deal with modules. Unfortunately though the same benefit could not be achieved by modules.

 

Overall, the "in size" "physical design" aspect of having to
state the types of members in a class definition can indeed be problematic
in the sense that you have to commit to the types in a class definition
regardless of whether the types are ever used by clients. There are
work-arounds(*) for that, so the question is whether language support
for such a facility is truly necessary and/or worth the cost of it.

(*) The work-arounds aren't necessarily easy - having a data buffer
in a class and having pimpl-side implementations know the actual
type of objects stored in such a buffer require proper alignment
and proper lifetime handling, which can be non-trivial.

Of course there are work-arounds. Pimpl would be just an annoying hack that would make inheritance difficult. Alignment and similar problems should be the compilers problem. 

"there is a workaround", pattern is often used, as most of the cases there is a workaround. There is a workaround for any aspect for which C++ tries to prove that it is inferior to C, including object oriented programming, type safety, templates etc. You can implement dynamic type checking and you will be on the safe side. You can implement constructs that mimic templates at a very high cost, including some trivial text based templates etc. etc. For everything there is a workaround. Unfortunately most of the cases it turns out that work-arounds are ugly hacks with serious negative consequences. 

For me the equation is simple: 

-> how many people/projects are using pimple?

-> why are they using pimple?

-> what is overhead to implement pimple for each class?

-> how much this overhead compromises the readibility of the code?

-> how many people/projects avoided pimple, just because pimple is ugly and preferred to discard advantages concerning the binary interface 

[Vicente J. Botet Escriba]

Le 07/02/15 15:51, mobi phil a écrit :

On 7 February 2015 at 16:06,  <gmis...@gmail.com> wrote:
>> While I do not know exactly the formal procedure, I do not think it is
>> worth the effort to write a formal paper unless some of the honorable
>> members of the honorable committee would show minimum interest to ever
>> consider it. The reason of the brainstorming is to find together weaknesses
>> of the model and solutions for them.
> I agree


As far as I understand it, the idea deals with being able to express
interface-implementation splits in a way that doesn't incur size/abstraction
penalty. Some aspects of that are handled by modules, some probably
aren't.

If you change the class definition, you need to recompile module + everything depends on module.

I don't know well the module proposal, but I would expect that only everything that depends on the public part of the module.

If you split the stuff into to, you need to recompile only if you change the real interface towards the rest of the world.

I would expect modules would bring this.

In my experience you spend 90% of time doing useless recompiles just because you need to ad a damm' private method. If it is private why do I need to recompile billions of lines of code in client code?

Agreed, there is no need.

While this is not the only benefit the proposal would bring, it is the only one that would have some way to deal with modules. Unfortunately though the same benefit could not be achieved by modules.

Why? What is missing in modules so that we can achieve it?

 

Overall, the "in size" "physical design" aspect of having to
state the types of members in a class definition can indeed be problematic
in the sense that you have to commit to the types in a class definition
regardless of whether the types are ever used by clients. There are
work-arounds(*) for that, so the question is whether language support
for such a facility is truly necessary and/or worth the cost of it.

(*) The work-arounds aren't necessarily easy - having a data buffer
in a class and having pimpl-side implementations know the actual
type of objects stored in such a buffer require proper alignment
and proper lifetime handling, which can be non-trivial.

Of course there are work-arounds. Pimpl would be just an annoying hack that would make inheritance difficult. Alignment and similar problems should be the compilers problem. 

"there is a workaround", pattern is often used, as most of the cases there is a workaround. There is a workaround for any aspect for which C++ tries to prove that it is inferior to C,

Don't forget that C++ includes C, so that anything you can do with C you can do with C++.

including object oriented programming, type safety, templates etc. You can implement dynamic type checking and you will be on the safe side. You can implement constructs that mimic templates at a very high cost, including some trivial text based templates etc. etc. For everything there is a workaround. Unfortunately most of the cases it turns out that work-arounds are ugly hacks with serious negative consequences. 

For me the equation is simple: 

-> how many people/projects are using pimple?

-> why are they using pimple?

-> what is overhead to implement pimple for each class?

-> how much this overhead compromises the readibility of the code?

-> how many people/projects avoided pimple, just because pimple is ugly and preferred to discard advantages concerning the binary interface 

I don't think pimpl is the idiom needed to split the back and the white parts of a class. IMHO, it is more the backdoor idiom that is needed. The backdoor is the black part of your proposal.

C.hpp
-----
class C {
public:
    // public member functions declaration
protected:
    // protected member functions declaration
private:
    aligned_storage<N>storage;
    friend struct black;
};


C.cpp
-----

struct C::black {
  C& that_;
  black(C& that) : that_(that) {}
  struct private_ {... };
  static_assert(sizeof(private_) == N, "Public and private sizes don't match"); 
  private_& pdata() { return *reinterpret_cast<private_*>(that_.storage);}

  // Add here any private static or non static member
};

int C::f() {
  // use of the C::black(*this) for any private access, either for data or private functions
  // use of C::black:: for any internal type or static member
}

My questions are:
* how many people/projects having compile time issues (millions of lines) don't use this backdoor idiom?
* why?

I'm all for a future C++ in which we don't need this idiom, and I expect that the Module proposal would provide it.

Vicente

[mobi phil]

If you split the stuff into to, you need to recompile only if you change the real interface towards the rest of the world.

I would expect modules would bring this.

well, while I do not know in depth the modules proposal, I would rather bet, that it will not be able to do the impossible. I have doubts that it would be so clever, that will not force clients to recompile once you add the declaration of a private method. 

 

While this is not the only benefit the proposal would bring, it is the only one that would have some way to deal with modules. Unfortunately though the same benefit could not be achieved by modules.

Why? What is missing in modules so that we can achieve it?

see above: IMHO modules will not be able to deal with

-> do not recompile client code if you add private method

-> do not recompile if you change class internal data

-> recompile only if the interface of a class changes, in best case if the method definition that client code depends on changes.

 

Please note that this black-white proposal does not mean that all classes have to behave this "dynamic" way.

Of course there are work-arounds. Pimpl would be just an annoying hack that would make inheritance difficult. Alignment and similar problems should be the compilers problem. 

"there is a workaround", pattern is often used, as most of the cases there is a workaround. There is a workaround for any aspect for which C++ tries to prove that it is inferior to C,

Don't forget that C++ includes C, so that anything you can do with C you can do with C++.

Thanks for telling me that... really forgot :) ... well trying to be a bit sarcastic here ... Then please read it: "with the C core features"

while you remove the problem of pimple, you introduce sthg. else, which my proposal tries to solve. You are strongly depending on this N. Too loosen a bit this dependency you could make at the beginning this N enough big, but that would lead to waste of memory. 

My proposal covers the fact that this "N" is  practically replaced at runtime, with the real size.

[Vicente J. Botet Escriba]

Le 08/02/15 13:23, mobi phil a écrit :

If you split the stuff into to, you need to recompile only if you change the real interface towards the rest of the world.

I would expect modules would bring this.

well, while I do not know in depth the modules proposal, I would rather bet, that it will not be able to do the impossible. I have doubts that it would be so clever, that will not force clients to recompile once you add the declaration of a private method.

IMO, this should be possible and so it is a QOI. Only if the implementers consider that this wouldn't be possible or too complex we would need to adapt the proposal.

 

While this is not the only benefit the proposal would bring, it is the only one that would have some way to deal with modules. Unfortunately though the same benefit could not be achieved by modules.

Why? What is missing in modules so that we can achieve it?

see above: IMHO modules will not be able to deal with

-> do not recompile client code if you add private method

-> do not recompile if you change class internal data

-> recompile only if the interface of a class changes, in best case if the method definition that client code depends on changes.

IMHO, it should be easier to ensure that the module proposal is able to do it that adding your alternative approach.

 

Please note that this black-white proposal does not mean that all classes have to behave this "dynamic" way.

I don't see anything dynamic in the behavior you have defined.

<snip>

For me the equation is simple: 

-> how many people/projects are using pimple?

-> why are they using pimple?

-> what is overhead to implement pimple for each class?

-> how much this overhead compromises the readibility of the code?

-> how many people/projects avoided pimple, just because pimple is ugly and preferred to discard advantages concerning the binary interface 

I don't think pimpl is the idiom needed to split the back and the white parts of a class. IMHO, it is more the backdoor idiom that is needed. The backdoor is the black part of your proposal.

<snip>

I'm all for a future C++ in which we don't need this idiom, and I expect that the Module proposal would provide it.

while you remove the problem of pimple, you introduce sthg. else, which my proposal tries to solve. You are strongly depending on this N. Too loosen a bit this dependency you could make at the beginning this N enough big, but that would lead to waste of memory. 

There is not big or small, just are equals or not. This N, should be provided by the compiler from the module definition as well as all other public and protected functions.

My proposal covers the fact that this "N" is  practically replaced at runtime, with the real size.

We don't need it at run-time, but at compile time, and IMO, the Module proposal should be such that such implementation is possible.

After reading the module proposal, it seams that, they are not addressing this problem. I suggest you to join the Module ML and request for the requirements you are looking for (not the solutions). Maybe your white/black split could be the solution.

c++std-...@accu.org

Vicente

[mobi phil]

Le 08/02/15 13:23, mobi phil a écrit :

If you split the stuff into to, you need to recompile only if you change the real interface towards the rest of the world.

I would expect modules would bring this.

well, while I do not know in depth the modules proposal, I would rather bet, that it will not be able to do the impossible. I have doubts that it would be so clever, that will not force clients to recompile once you add the declaration of a private method.

IMO, this should be possible and so it is a QOI. Only if the implementers consider that this wouldn't be possible or too complex we would need to adapt the proposal.

I do not pretend that I know the proposal, I have some basic idea of how clang is doing it. Without going into the details I think there are two things that needs to be looked at separated:

-> if the header changes, the module that depends on it needs to be rebuilt.

-> the way at least clang approached it, you still need to work with makefile dependencies.

If you changed the header, the timestamp, checksum, or whatsoever way you would implement dependencies, the source file that depends on the header in question will have to be rebuilt. This must be true even in case where you add a private method, unless a special file is generated for dependencies that reflect only really "public" stuff from headers.

 

While this is not the only benefit the proposal would bring, it is the only one that would have some way to deal with modules. Unfortunately though the same benefit could not be achieved by modules.

Why? What is missing in modules so that we can achieve it?

see above: IMHO modules will not be able to deal with

-> do not recompile client code if you add private method

-> do not recompile if you change class internal data

-> recompile only if the interface of a class changes, in best case if the method definition that client code depends on changes.

IMHO, it should be easier to ensure that the module proposal is able to do it that adding your alternative approach.

While reducing dependencies is one of the goals of my approach/proposal I really doubt that handling private data changes in headers will be delt with modules. For simple reason that any source file that depends on the definition of the class (due to sizeof()) will need to recompile once the size of the class changes. In my proposal this dependency is postponed till runtime. 

 

Please note that this black-white proposal does not mean that all classes have to behave this "dynamic" way.

I don't see anything dynamic in the behavior you have defined.

Well, do not wonder, I have the impression that you did not understand the main idea. 

Did you carefully read the original post? Sorry, but I doubt

Let's rewrite the reasoning with different perspective

-> What do we want?

->>> to really hide private part of a class 

--->>>> private methods

--->>>> private data

-> What is the main problem with this?

--->>>> private methods: not really.. they are forced to be declared in the class declaration, without any reason, so no problem to remove

--->>>> private data: well... this is the problem, as "client" classes need to know the size of the class

---> solution: make any local/global/heap allocation depending on a "global" variable that will be known at dynamic linking

while this would open doors for other dynamic things, you already see a little "dynamic" here.

while you remove the problem of pimple, you introduce sthg. else, which my proposal tries to solve. You are strongly depending on this N. Too loosen a bit this dependency you could make at the beginning this N enough big, but that would lead to waste of memory. 

There is not big or small, just are equals or not. This N, should be provided by the compiler from the module definition as well as all other public and protected functions.

My proposal covers the fact that this "N" is  practically replaced at runtime, with the real size.

We don't need it at run-time, but at compile time, and IMO, the Module proposal should be such that such implementation is possible.

Well, while do not know who are the "we" I need it and I see tha major advantage. Whatsoever way you define N at compile time, you will force everybody who depends on N to rebuild once you change the size of the implementation which is exactly the thing I want to achieve among others. 

Another point of this "dynamic" classes is that you could use C++ classes from dynamic libraries without any pimpl overhead.

 

After reading the module proposal, it seams that, they are not addressing this problem. I suggest you to join the Module ML and request for the requirements you are looking for (not the solutions). Maybe your white/black split could be the solution.

c++std-...@accu.org

No problem to help there, but personally I fail to see that this could be done with modules...

[Thiago Macieira]

On Sunday 08 February 2015 18:14:26 mobi phil wrote:
> If you changed the header, the timestamp, checksum, or whatsoever way you
> would implement dependencies, the source file that depends on the header in
> question will have to be rebuilt. This must be true even in case where you
> add a private method, unless a special file is generated for dependencies
> that reflect only really "public" stuff from headers.

Unless, of course, that class has friends. In which case, even private changes
will require rebuilding.

> While reducing dependencies is one of the goals of my approach/proposal I
> really doubt that handling private data changes in headers will be delt
> with modules. For simple reason that any source file that depends on the
> definition of the class (due to sizeof()) will need to recompile once the
> size of the class changes. In my proposal this dependency is postponed till
> runtime.

Your idea, not proposal. You need to do a little more work to make it a
proposal, including writing a paper and presenting it in a committee meeting.
Or convincing someone to do it for you.

> > After reading the module proposal, it seams that, they are not addressing
> > this problem. I suggest you to join the Module ML and request for the
> > requirements you are looking for (not the solutions). Maybe your
> > white/black split could be the solution.
> >
> > c++std-...@accu.org
> >
> > No problem to help there, but personally I fail to see that this could be
>
> done with modules...

Are you saying that modules would make it impossible to implement your
suggestion?

Because if you aren't saying that, then you're missing the point. We're not
saying that modules is doing what you suggested. We are saying that you should
join the modules discussion so that your solution gets worked into the modules
proposal.

[mobi phil]

On Sunday 08 February 2015 18:14:26 mobi phil wrote:
> If you changed the header, the timestamp, checksum, or whatsoever way you
> would implement dependencies, the source file that depends on the header in
> question will have to be rebuilt. This must be true even in case where you
> add a private method, unless a special file is generated for dependencies
> that reflect only really "public" stuff from headers.

Unless, of course, that class has friends. In which case, even private changes
will require rebuilding.

Friendship as always should be sometimes reconsidered. If not really necessary for performance reason, just use getters/setters.

But if really necessary and most of the time friends like to be together, most of the time they are close to each other.

Such friendship would be declared in the black, and friends may/should know about blacks, but they should not propagate this dependency. 

inside A.h

white  A { /* white is the real iterface, could be called "class iface" 

}

inside A.cpp

black A {

friend black F;
}

inside F.cpp

#include "A.cpp"

#include "F.h" 

the black of A could also be factored into A.black.h, that would be included by both A.cpp and F.cpp

> While reducing dependencies is one of the goals of my approach/proposal I
> really doubt that handling private data changes in headers will be delt
> with modules. For simple reason that any source file that depends on the
> definition of the class (due to sizeof()) will need to recompile once the
> size of the class changes. In my proposal this dependency is postponed till
> runtime.

Your idea, not proposal. You need to do a little more work to make it a
proposal, including writing a paper and presenting it in a committee meeting.
Or convincing someone to do it for you.

Let's not get lost in words. It is a proposal, though it is not a formal one. It was not just an idea, I thought about at coffee time. Have it in my mind for a while and though about certain aspects. While I wanted to implement it in pure C with macros, gave up. Though could reformulate the original post, do not feel at the moment it would be worth to go into a deep analyzis if the added value is not recognized. 

 

> > After reading the module proposal, it seams that, they are not addressing
> > this problem. I suggest you to join the Module ML and request for the
> > requirements you are looking for (not the solutions). Maybe your
> > white/black split could be the solution.
> >
> > c++std-...@accu.org
> >
> > No problem to help there, but personally I fail to see that this could be
>
> done with modules...

Are you saying that modules would make it impossible to implement your 

suggestion?

Because if you aren't saying that, then you're missing the point. We're not
saying that modules is doing what you suggested. We are saying that you should
join the modules discussion so that your solution gets worked into the modules
proposal.

To avoid further confusion: I am not saying anything else than that I have strong doubts that this could be worked in some way into modules if you do not add the necessary language extension. Once you added those extensions, should be treated as a separate story as it involves mainly non-module related changes.

[Thiago Macieira]

On Sunday 08 February 2015 19:17:37 mobi phil wrote:
> > On Sunday 08 February 2015 18:14:26 mobi phil wrote:
> > Unless, of course, that class has friends. In which case, even private
> > changes
> > will require rebuilding.
>
> Friendship as always should be sometimes reconsidered. If not really
> necessary for performance reason, just use getters/setters.

That's advice. Unless forbid the practice, the advice can be ignored and you
have to deal with it.

It's irrelevant whether it's a bad practice or not.

> But if really necessary and most of the time friends like to be together,
> most of the time they are close to each other.
> Such friendship would be declared in the black, and friends may/should know
> about blacks, but they should not propagate this dependency.
>
> inside A.h
>
> white A { /* white is the real iterface, could be called "class iface"
>
> }
>
> inside A.cpp
>
> black A {
>
>
> friend black F;
> }
>
> inside F.cpp
> #include "A.cpp"
> #include "F.h"
>
> the black of A could also be factored into A.black.h, that would be
> included by both A.cpp and F.cpp

Sounds like advice. Therefore, it can be ignored.

> Let's not get lost in words. It is a proposal, though it is not a formal
> one. It was not just an idea, I thought about at coffee time. Have it in my
> mind for a while and though about certain aspects. While I wanted to
> implement it in pure C with macros, gave up. Though could reformulate the
> original post, do not feel at the moment it would be worth to go into a
> deep analyzis if the added value is not recognized.

What recognition are you expecting before you go further?

> > Are you saying that modules would make it impossible to implement your
> > suggestion?
> > Because if you aren't saying that, then you're missing the point. We're
not
> > saying that modules is doing what you suggested. We are saying that you
> > should
> > join the modules discussion so that your solution gets worked into the
> > modules
> > proposal.
>
> To avoid further confusion: I am not saying anything else than that I have
> strong doubts that this could be worked in some way into modules if you do
> not add the necessary language extension. Once you added those extensions,
> should be treated as a separate story as it involves mainly non-module
> related changes.

Understood.

I still recommend talking to the people doing modules because they may have
other points to bring up and they're the right people to help you with your
suggestion.

[mobi phil]

On Sunday 08 February 2015 19:17:37 mobi phil wrote:
> > On Sunday 08 February 2015 18:14:26 mobi phil wrote:
> > Unless, of course, that class has friends. In which case, even private
> > changes
> > will require rebuilding.
>
> Friendship as always should be sometimes reconsidered. If not really
> necessary for performance reason, just use getters/setters.

That's advice. Unless forbid the practice, the advice can be ignored and you
have to deal with it.

It's irrelevant whether it's a bad practice or not.

while this new construct has to have a different name than "class" or another keyword attached to it, it is straightforward to turn the above into a rule by forbidding friend keyword inside the body of with the white. 

 

Though could reformulate the
> original post, do not feel at the moment it would be worth to go into a
> deep analyzis if the added value is not recognized.

What recognition are you expecting before you go further?

Well, so far you as QT "guy" (I understand), you did not seem to show any sign that it would be valuable in a framework like QT (obviously without thinking about rewriting parts of it, to make this used). Obviously this does not mean that I do not see that value.

 

Understood.

I still recommend talking to the people doing modules because they may have
other points to bring up and they're the right people to help you with your
suggestion.

Will try to do so, just that it needs first go through those papers and not too much time for it.

[Thiago Macieira]

On Sunday 08 February 2015 19:47:37 mobi phil wrote:
> > What recognition are you expecting before you go further?
>
> Well, so far you as QT "guy" (I understand), you did not seem to show any
> sign that it would be valuable in a framework like QT (obviously without
> thinking about rewriting parts of it, to make this used). Obviously this
> does not mean that I do not see that value.

I have two qualms about the proposal: one, which we discussed, is the runtime
overhead which may be too great, greater than the use of d pointers (pimpl).
The other is that I can't use for a long time. Let me put it this way: if this
showed up for C++17, we might be able to start relying on it in Qt by 2025.

Other than that, the idea is interesting.

If you want more feedback, write a paper.

[mobi phil]

I have two qualms about the proposal: one, which we discussed, is the runtime
overhead which may be too great, greater than the use of d pointers (pimpl).

I do not think this would be the case. For accessing data members that are in the pimpl or back through getters/setters, the overhead would be equal, that is a function call. For data members that are in the "white", indeed there is an overhead compared to normal member access, but this overhead would not be greater than accessing "black" members. 

For both "black" and "white" members (well, both for real public ones from white and the one's that are accessed through getters/setters from black) one could think about another language extension I would call it runtime inline-ing. Given a  pair of setter/getter or "white" public data member, the compiler would generate placeholder code and the dynamic loader would patch it with the correct de-referencing of the member based on the "final" layout of the class. This mechanism will trade code space for this patching versus efficiency of patching/replacement.

Overhead for creating objects would be in worst case equal with pimpl.

 

The other is that I can't use for a long time. Let me put it this way: if this
showed up for C++17, we might be able to start relying on it in Qt by 2025.

Well, tend to agree, but this sad law of reality applies to any other proposals.

Other than that, the idea is interesting.

If you want more feedback, write a paper.

Will try to write then a first version of the paper. At the end of the day it does not need to pretend it would cover all the details about the impact on the future of the humanity :) 

 

[mobi phil]

is this the paper about modules?

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2073.pdf

did skim it, but:

-> not too much thing is clearly proposed as standard in terms of templates instantiation which in my opinion takes most of compilation times

-> does not say anything about recompilation dependencies when changing the data members thus size of classes...

will go later to their mailing list, to form an idea where they are

[Klaim - Joël Lamotte]

On Sun, Feb 8, 2015 at 10:00 PM, mobi phil <mo...@mobiphil.com> wrote:

is this the paper about modules?

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2073.pdf

There have been several proposals and updates since this one (see the year in the url).

The last module proposal is from microsoft (they have a private implementation apparently): http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4047.pdf

It's similar to the one implemented in clang, with a few differences. See http://clang.llvm.org/docs/Modules.html

[mobi phil]

thanks for the correct link. 

-> I was investigating clang modules implementation few weeks ago. Unfortunately did not gain anything in terms of compilation time improvements for some source code that was heavily depending on heavy boost templates. It does not support implicit template instantiations to be cache in the compiled modules, which for me is the most important thing. Would love to see instant compilation for a file that includes few headers that contain heavy templates. Maybe I failed to grasp how to do it and raised the question on clang dev-list, but did not get till today a clear answer. Would love to see somebody showing me real compilation time gain when building such. But this is not the topic of this thread.

-> went through the latest proposal:

->> The proposal luckily makes reference to implicit template instantiation caching. 

->> Do not like that they are convinced to throw away the preprocesor without creating something in place

->> related to my current proposal (see also the quote from the paper below), they touch the subject about "changes to private member" but the seem not to want to adventure by coming up with any solution.

from the proposal

"An occasionally vexing rule of standard C++ is that protection controls access,

not visibility. E.g. a private member of a class is visible to, but not accessible to

non-member entities. In particular, any change to a private member of a class is

likely to trigger re-processing of any translation unit that depends on that class’s

definition even if the change does not affect the validity of dependent units. It

is tempting to solve that problem with a module system. However, having two

distinct sets of rules (visibility and accessibility) for class members strikes us as

undesirable and potentially fertile source of confusion. Furthermore, we want to

support mass-migration of existing codes to modules without programmers having

to worry about class member name lookup rules: if you understand those rules

today, then you do not have to learn new rules when you move to modules and you

do not have to worry about how the classes you consume are provided (via modules

or non-modules).

That being said, we believe the visibility vs. accessibility issue is a problem that

should be solved by an orthogonal language construct, irrespectively of whether a

class is defined in a module interface declaration or in an ordinary translation unit.

[Vicente J. Botet Escriba]

Le 09/02/15 02:56, mobi phil a écrit :

> thanks for the correct link.
>
> -> I was investigating clang modules implementation few weeks ago.
> Unfortunately did not gain anything in terms of compilation time
> improvements for some source code that was heavily depending on heavy
> boost templates. It does not support implicit template instantiations
> to be cache in the compiled modules, which for me is the most
> important thing. Would love to see instant compilation for a file that
> includes few headers that contain heavy templates. Maybe I failed to
> grasp how to do it and raised the question on clang dev-list, but did
> not get till today a clear answer. Would love to see somebody showing
> me real compilation time gain when building such. But this is not the
> topic of this thread.
>

Agreed. However the goal is the same: reduce compile time.

Thanks for the quote. It seems I was wrong, and that the module proposal
would not take care of the visibility issues.

I believe that you have here the go from the honorable people you were
looking for, at least for the problem to be solved. I'm sure the same
people would be interested in discussing possible solutions.

I will add this quoting the proposal:
"Runtime Performance
Moving an existing code to a brave new module world, or writing new
codes with modules, should not in any way degrade its runtime
performance characteristics. In particular, we do not seek a module
system requiring a compiler to perform automatic “boxing” of object
representation (exposed in class private members) –in attempts to
reducing re-compilation– via opaque data pointers a la ` pImpl idiom."


Respect to the visibility issue, I would like to preserve Runtime
Performance also: Moving an existing code to a brave new visibility
aware world, or writing new codes with visibility concerns, should not
in any way degrade its runtime performance characteristics."

I'm not saying that I'm against any visibility solution that doesn't
preserve runtime performance, just that the performance aspect must be
considered.

Vicente

[mobi phil]

I will add this quoting the proposal:
"Runtime Performance
Moving an existing code to a brave new module world, or writing new codes with modules, should not in any way degrade its runtime performance characteristics. In particular, we do not seek a module system requiring a compiler to perform automatic “boxing” of object representation (exposed in class private members) –in attempts to reducing re-compilation– via opaque data pointers a la ` pImpl idiom."


Respect to the visibility issue, I would like to preserve Runtime Performance also: Moving an existing code to a brave new visibility aware world, or writing new codes with visibility concerns, should not in any way degrade its runtime performance characteristics."

I'm not saying that I'm against any visibility solution that doesn't preserve runtime performance, just that the performance aspect must be considered.

Well, while it would be annoying to have a general solution where private data would be automatically moved to some pimpl one has to have the freedom to do so. Some applications may suffer from this little performance penalty of the black/white approach, but as described earlier this would be the same overhead as a non-inlined pair of setter/getter. You are trading something against something.

While non-inlined setters/getters would be a performance penalty for both actual approach and black/white, maybe one could think of some link time or dynamic link time inlining solution, where a certain code binary code pattern could be replaced with the "real implementation".

 

[mobi phil]

FYI: some of you for sure know about llvm-lto. It could solve the overhead of getter/setters or with other words how offsets would be solved.

llvm/clang has llvm-lto and the gold plugin that can inline functions at linking time. The setters/getters defined in the black and containing the correct member offset information could be inlined at link time. It seems that there would be no runtime access overhead for data members defined in "black". 

I assume that this trick with lto(=inline) could be used also for pimpl related call overhead. 

[Chris Gary]

This is an attractive alternative to pointer-based PIMPL, though it would require sweeping modifications to be made across every compliant loader/toolset.

That said, I still like it.

However, it doesn't seem to address re-using the same interface for multiple implementations simultaneously (exposing "new" through the interface layer constrains it to a single implementation).

Another issue is version discrepancy: What metadata must be stored in external module containing all the "black" in order to ensure that the addition of new members to that layer do not cause undefined behavior when linked against an older "white" that doesn't know about them? There needs to be a way to verify a new "black" against an already compiled "white" during compilation, which seems to indicate the need for a large amount of metadata...

These concerns stem from a number of problems I've been facing lately:

Say we need to display the same dataset on a workstation monitor and through a projector.

The monitor is connected through an Nvidia adapter, and the projector through an old ATI.

We have to create one rendering context per adapter, as moving a window from the workstation display to the projector will crash or result in garbage (unless all rendering is done with GDI).

Both contexts have different feature sets resulting in different "black" layers, however the "white" layer looks the same for both.

The obvious solution is PIMPL, though there is a lot of work involved in maintaining the parallel sets of types required to realize this.
After a small change in the interface, resolving version discrepancies for every affected module slows development to a crawl.

The biggest time-waster is a new "black" failing its tests after compilation.

[Chris Gary]

Answering my first question: Just have the "black" layer return a pointer, which brings us back to PIMPL.

The real power of this proposal would be a standardized way to validate an implementation against a compiled interface.

On Monday, February 16, 2015 at 11:42:28 PM UTC-7, Chris Gary wrote:

This is an attractive alternative to pointer-based PIMPL, though it would require sweeping modifications to be made across every compliant loader/toolset.

That said, I still like it.

However, it doesn't seem to address re-using the same interface for multiple implementations simultaneously (exposing "new" through the interface layer constrains it to a single implementation).

*snip*

[mobi phil]

This is an attractive alternative to pointer-based PIMPL, though it would require sweeping modifications to be made across every compliant loader/toolset.

yes, but as usual, the classes without the additional keyword are supposed to behave as earlier.

 

That said, I still like it.

happy to hear that 

However, it doesn't seem to address re-using the same interface for multiple implementations simultaneously (exposing "new" through the interface layer constrains it to a single implementation).

Does not address multiple implementations as you could have multiple implementations exactly like for a usual cpp file that implements it's header. This can be seen bad or good. You may provide for linking one or other version, depending on your intentions.

 

Another issue is version discrepancy: What metadata must be stored in external module containing all the "black" in order to ensure that the addition of new members to that layer do not cause undefined behavior when linked against an older "white" that doesn't know about them? There needs to be a way to verify a new "black" against an already compiled "white" during compilation, which seems to indicate the need for a large amount of metadata...

Not sure if I completely understand your concern, but let me add some thoughts. Indeed I did not advise if there should be one black for the same white interface. Allowing different "cumulative" blacks in different files for the same white would allow much more flexibility at a very high cost (complexity). Thus from the beginning I would discard that option, so only one black for each white. 

Also keep in mind, like above even now you can compile "foo.cpp" against "foo.h" but "bar.cpp" against an older version of "foo.h". There may be serious object misalligment that may cause crash. Thus similar care must be taken for building, but build (dependency etc.) management discipline is out of scope. 

On the other side this black/white story would reduce such dangers, as class layout information would be moved to only one place and that is the black.

 

These concerns stem from a number of problems I've been facing lately:

Say we need to display the same dataset on a workstation monitor and through a projector.

The monitor is connected through an Nvidia adapter, and the projector through an old ATI.

We have to create one rendering context per adapter, as moving a window from the workstation display to the projector will crash or result in garbage (unless all rendering is done with GDI).

Both contexts have different feature sets resulting in different "black" layers, however the "white" layer looks the same for both.

Do not have too much knowledge here, but my general idea abut it was that such implementation details about rendering should be done in the graphics card driver, and the user space program should not be aware if it is displaying on one or other monitor. 
Nevertheless this would go a bit out of the scope of the proposal. 
Only related thing I can comment on is that at run-time there would be only one black. Design patters like polymorphism/strategy  would be implemented like before. 

 

The obvious solution is PIMPL, though there is a lot of work involved in maintaining the parallel sets of types required to realize this.

Take it like that this black/white is at the end of the day PIMPL without the pointer overhead and parallel set of types.

 

After a small change in the interface, resolving version discrepancies for every affected module slows development to a crawl.

Still not clear. If you change the interface, your dependency system should rebuild everything. One of the gains with with black/white is that you reduce this rebuild risk to minimum.

 

Conclusion: I am afraid did not understand clearly your concerns. Perhaps you did not understand completely the concept? 

[Chris Gary]

If you don't mind me horribly abusing the terminology here...

From what I gather: "white" is the interface and "black" is the implementation (duh). Instead of the pointer-to-PIMPL, this would work kinda like partial classes in C#: The loader fills out offset/size details at load time.

The display example is just one of several (Nvidia and ATI crap don't get along, so everything breaks, but that's beside the point).

Rebuilding isn't the problem: Its validating changes that have to be made on a per-implementation basis.

The problem, in general, is that there are numerous interfaces that the "core" application is meant to work with, and those interfaces are backed by implementations known only at runtime (plugins).

This is why I asked about re-using interfaces: All the "whites" would be in a central codebase (in some cases, compiled and distributed to end-users who write their own plugins), and all the "blacks" would be implemented on a per-plugin/component basis. Multiple plugins that back the same interface would need to be loaded simultaneously.

The problem with the distribution of the "white" binary is that users will need/want to add members to the "black" halves they are writing, creating part of the "version discrepancy" issue I was describing (this also happens when a new "white" binary is distributed that must work with all existing "blacks").

In the display example, we'd need two different graphics backends: Usually the small vs. large display or comparing OpenGL and D3D with synchronized resources (CAD shapes w/ optimized representations for each).

D3D is very, very different from OpenGL, so the "black" half of either would need to be updated by hand. Add to that OpenGL's abomination of an "extension" system, and you have a swampland of implementations to fill in.

If your proposal is supposed to work statically, then I've completely missed the boat (wouldn't surprise me, I've been looking for a way to simplify these messes for years)...

[mobi phil]

If you don't mind me horribly abusing the terminology here...

any abuse should be accepted as long as it helps to converge ;) 

 

First let's fix your terminology a bit to avoid confusion. You mentioned core. This may create confusion. Let's better call it internal world of the class (both black and white) and the external world (consumer) of the class. 

Let me also resume something which may be spread in the original post and subseqeunt comments. The external world, or core as you call. 

It is only the black that knows about the size and layout of the class (black). The outside world can have access to this layout through setters/getters.

Implementation detail cannot be ignored for a successful design of the abstraction. So far two approaches were identified how to solve the problem of data member offset. (the sizeof for new() is similar, let's ignore for the moment)

1. without overhead of pointer dereference or function call, but less flexible

2. with this overhead, either with pointers or function call, but more flexible.

for 1.) the setters/getters would be inlined at link time with setters/getters automatically generated by the compiler for black members.

for 2.) any setter/getter in the white, thus public data member would be solved at runtime as:

-> a non inlined method implemented in the black 

-> a pointer to a global offset table that contains the offset.

At the end the both implementation could be provided depending on the usage. 

In your example you seem to focus rather on 2.) which again:

-> has advantage that allows dynamic loading of blacks from shared libraries, or different versions of them etc.

-> has the disadvantage that any member reference will "cost" an extra pointer dereferencing that often may be expensive due to cache incoherence.

From what I gather: "white" is the interface and "black" is the implementation (duh). Instead of the pointer-to-PIMPL, this would work kinda like partial classes in C#: The loader fills out offset/size details at load time.

se above... depends which implementation you would chose. If setters/getters are inlined at link-time, the story at the end would be exactly like with normal classes. 

 

The display example is just one of several (Nvidia and ATI crap don't get along, so everything breaks, but that's beside the point).

ok, let's forget this example, probably was not very relevant or confusing. 

Rebuilding isn't the problem: Its validating changes that have to be made on a per-implementation basis.

The problem, in general, is that there are numerous interfaces that the "core" application is meant to work with, and those interfaces are backed by implementations known only at runtime (plugins).

Again, based on the implementation one would choose, it can open the doors for such mechanism like loading blacks as plugins. 

 

This is why I asked about re-using interfaces: All the "whites" would be in a central codebase (in some cases, compiled and distributed to end-users who write their own plugins), and all the "blacks" would be implemented on a per-plugin/component basis. Multiple plugins that back the same interface would need to be loaded simultaneously.

I think I understand what is your point. But you go one step further. One could the same time say you "abuse" the feature or you "bulid on top" of the feature. In my original proposal, again, there is one black to one white. What you say can be implemented if the offsets are implemented by pointers or setters/getters. At the end of the day a black will be a set of functions towards the external world, so you can see it as plugin. But this is out of the scope of the original proposal. Though it is a strong point to support it.

 

The problem with the distribution of the "white" binary is that users will need/want to add members to the "black" halves they are writing, creating part of the "version discrepancy" issue I was describing (this also happens when a new "white" binary is distributed that must work with all existing "blacks").

Att the end you may add anything to the black, even at runtime. What is important is that any object of the black or subclass once crceated with a certain version ( sizeof/new), should use the same implementation for referencing members.

 

In the display example, we'd need two different graphics backends: Usually the small vs. large display or comparing OpenGL and D3D with synchronized resources (CAD shapes w/ optimized representations for each).

D3D is very, very different from OpenGL, so the "black" half of either would need to be updated by hand. Add to that OpenGL's abomination of an "extension" system, and you have a swampland of implementations to fill in.

again, this example is bit two complex and unclear. But if you are refering to a software system where some OpenGL or D3D class layuot changes often during development cycles and lot of code is depending on it, then the proposed system would allow you to add/remove private methods and data without having to rebuild tons of files depending on the class layout.

 

 

If your proposal is supposed to work statically, then I've completely missed the boat (wouldn't surprise me, I've been looking for a way to simplify these messes for years)...

again, see above. Probably both implementations should be supported (static and dynamic)

it is on my TODO list to write a more formal version of the paper, where I will take care to give more clear examples

[Chris Gary]

Based on your explanation, I have some suggestions.

To avoid introducing new CSK, you can re-use "virtual" and "override" for this purpose.

For example:

    class white button
    {
    public:

        // Speculative data members
        // (roughly equivalent to properties)
        string title;
        visual_style &style;
    };

    class black button
    {
    public:

        string title;
        HWND hWnd;
        visual_style &style;
       
        // Etc...
    };

Change "white" to "virtual", and "black" to "override":

    // Speculative interface
    class virtual button
    {
    public:

        string title;
        visual_style &style;
       
        // Note: We can still have virtual methods.
    };

    // Concrete definition
    class override button
    {
    public:
   
        // At least "string title" and "visual_style &style"
        // must be declared. Anything else is free game.

        string title;
        HWND hWnd;
        visual_style &style;
    };

   
Accessing a member through a speculative interface would merely involve fetching an offset from a vtable. Much like static binding of a virtual method call, it is also possible to deduce a member offset statically if the concrete definition is available.

Inheriting from a virtual class makes the result virtual and requires that the override inherit from the same bases (possibly in the same order). It might be reasonable to automatically attach that inheritance to the override when its definition is encountered.

As the override might inherit from additional bases, this makes chasing down "repeated base" errors a bit more involved.

Unfortunately, using this model, it must be illegal to instantiate or apply sizeof() to a "virtual" class if an "override" is not visible in the same translation unit.

[Chris Gary]

I would also like to add that, unless there is a reasonable way to combine a "white" and "black" into the same definition, this actually forces the two to be separated.

[Matthew Woehlke]

On 2015-02-17 12:02, Chris Gary wrote:
> To avoid introducing new CSK, you can re-use "virtual" and "override" for
> this purpose.

Hmm... not loving it. Anything wrong with:

public class Foo /* white */ { ... };
private class Foo /* black */ { ... };

...?

On 2015-02-17 12:16, Chris Gary wrote:
> I would also like to add that, unless there is a reasonable way to combine

> a "white" and "black" into the same definition, this actually *forces* the
> two to be separated.

How would such a combination be different from a regular class?

--
Matthew

[mobi phil]

hi, as stated in the original post, personally do not care how the two constructs will be named, thus if majority of folks would love virtual/override do not care... However it is not really virtual and override in my original concept. I would not force all the data members to be virtual. I would allow part of data to be in white. The black would complement it. This in my concept the title from white will be different than the title from black, well, there should be no duplicates however. What is also open for "vote" how one would link the data member from black to the interface. One way would be sthg. you are mentioning, the other through manually specified getter/setters.

 

    class white button
    {
    public:

        // Speculative data members
        // (roughly equivalent to properties)

        virtual string title; /* note I added here virtual */                         //         // it would be equally possible string getTitle(); /* note I added here virtual */

        visual_style &style;
    };

    class black button
    {
    public:

        string title; /* the linker will make this public through the virtual from white */

        HWND hWnd;
        visual_style &style;
       
        // Etc...
    };

Accessing a member through a speculative interface would merely involve fetching an offset from a vtable. Much like static binding of a virtual method call, it is also possible to deduce a member offset statically if the concrete definition is available.

indeed, if you read through the thread vtable and other "synonims" were mentioned. I used global offset table, but all is about accessing an offset from a location that is known only at link time or load time versus the "efficient" hard coded way.

 

Inheriting from a virtual class makes the result virtual and requires that the override inherit from the same bases (possibly in the same order). It might be reasonable to automatically attach that inheritance to the override when its definition is encountered.

well, I am afraid this would make things rather complicated and lower the chance for acceptance. In the original concept the inheritance and lot of things would be like in the normal classes to avoid complications. One could translate the original proposal into: make invisible what is indeed private, but keep the rest of the rules unchanged

As the override might inherit from additional bases, this makes chasing down "repeated base" errors a bit more involved.

Hm... again... seems for me unnecessary complication unless you come with example and use case.

 

Unfortunately, using this model, it must be illegal to instantiate or apply sizeof() to a "virtual" class if an "override" is not visible in the same translation unit.

Indeed, this was mentioned in the original post if I am not wrong, and was debated. For dynamic allocation the allocator should call a sizeof() member function or will be a global variable calculated at link time.

 

--

---
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposal...@isocpp.org.
To post to this group, send email to std-pr...@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/std-proposals/.

--

rgrds,
mobi phil

being mobile, but including technology
http://mobiphil.com

[mobi phil]

sorry.. did you read the original post and the thread?  

[mobi phil]

probably not bad idea for keywoard.. though 

class public Foo {

}

and 

class private Foo {

}

the "public class Foo" would rather mean that the class is public, whereas it is about the part that is being declared is public/private.

[Matthew Woehlke]

On 2015-02-17 12:42, mobi phil wrote:

> Matthew Woehlke wrote:
>> On 2015-02-17 12:16, Chris Gary wrote:
>>> I would also like to add that, unless there is a reasonable way
>>> to combine a "white" and "black" into the same definition, this
>>> actually *forces* the two to be separated.
>>
>> How would such a combination be different from a regular class?
>
> sorry.. did you read the original post and the thread?

It's been a while, but I think I recall the general gist. I think one of
us is missing something, however. I get the black/white concept when you
separate the interface from the implementation (isn't that the point?).
What I *don't* get is what it means to use this feature when the two are
*not* separated. (In particular, as per my question, what would be the
use case for that and how would it differ from a "regular" class?)

--
Matthew

[Matthew Woehlke]

On 2015-02-17 12:41, mobi phil wrote:
> hi, as stated in the original post, personally do not care how the two
> constructs will be named, thus if majority of folks would love
> virtual/override do not care... However it is not really virtual and
> override in my original concept. I would not force all the data members to
> be virtual. I would allow part of data to be in white. The black would
> complement it.

Agreed, but note also I don't think we're suggesting that. (Overriding
(pardon the pun) those keywords for this is a bit abusive of what they
normally mean, and a source of potential confusion as you've just
demonstrated, which is one reason using public/private instead might be
better.)

--
Matthew

[mobi phil]

It's been a while, but I think I recall the general gist. I think one of
us is missing something, however. I get the black/white concept when you
separate the interface from the implementation (isn't that the point?).

exactly

 

What I *don't* get is what it means to use this feature when the two are
*not* separated.

sorry... really confused. If you do not separate the the feature is not present. They are hand in hand.. or I miss what you miss ;) 

(In particular, as per my question, what would be the
use case for that and how would it differ from a "regular" class?)

Take it like that: they don't want to differ from a regular classe. They want to solve few problems:

1. keep it simple (the interface)

2. avoid useless dependency between interface consumer and the interface, thus reduce recompilation in development process when you change layout of the class that is not affecting consumers

2. depending on the implementation, other "positive side effects" like loading classes from shared library, dynamic patching of the source code etc.

 

[Chris Gary]

On Tuesday, February 17, 2015 at 10:38:24 AM UTC-7, Matthew Woehlke wrote:

*snip*

On 2015-02-17 12:16, Chris Gary wrote:
> I would also like to add that, unless there is a reasonable way to combine
> a "white" and "black" into the same definition, this actually *forces* the
> two to be separated.

How would such a combination be different from a regular class?

--
Matthew

The idea would be that it would simultaneously define an interface and implementation, while allowing other implementations to be defined later. This would reduce repetition, as a separate "black" would not have to be defined right away.

Unless there is some way to add a "pure" specifier, the absence of which would allow a "white" definition to double as "black".

I still have a problem with needing to resolve sizeof() at link time: Using sizeof() is critical for templates that decide things like whether or not an instance will fit in a small aligned buffer versus needing heap allocation.

Obviously, alignment becomes a problem, as we might have some member with larger than alignof(double) requirement (e.g. AVX tuple). I don't think this can be dealt with efficiently at link time, as it would basically require all template "algorithms" to be recorded as bytecode somehow. That, or simply forbidding sizeof() or alignof() to be used on an interface in the context of a template, which makes the idea incompatible with half the language...

Concerning keywords: I don't really care what is chosen, just as long as they don't turn nested definitions and the like into keyword soup or become easily confused with other commonly-defined symbols (such as color enums). My policy about keywords is that the result needs to be reasonably readable as monochrome text...

[Chris Gary]

On Thursday, February 19, 2015 at 3:45:07 AM UTC-7, Chris Gary wrote:

That, or simply forbidding sizeof() or alignof() to be used on an interface in the context of a template, which makes the idea incompatible with half the language...

I exaggerated a bit there... Let me clarify: If a "white" can double as a "black" (or there is a compatible "black" in the TU), then sizeof() and alignof() are well-defined, but it absolutely must be deduced at compile time. The case where it can't be used in a template really isn't that different than applying sizeof() to an incomplete type (pure "white" is fundamentally incomplete)...

[mobi phil]

The idea would be that it would simultaneously define an interface and implementation, while allowing other implementations to be defined later. This would reduce repetition, as a separate "black" would not have to be defined right away.

Did not understand this entirely. One of the main objective is to keep class interfaces thus "whites" untouched during development cycles. Thus cannot see too much reason not to have separate black. can you give an example pls.

 

Unless there is some way to add a "pure" specifier, the absence of which would allow a "white" definition to double as "black".

again, I fail to see the point, but please develop, give example

 

I still have a problem with needing to resolve sizeof() at link time: Using sizeof() is critical for templates that decide things like whether or not an instance will fit in a small aligned buffer versus needing heap allocation.

Well, for sure it is not easy. Maybe such templates will be forbidden to use such constructs, unless adapted. 

 

Obviously, alignment becomes a problem, as we might have some member with larger than alignof(double) requirement (e.g. AVX tuple). I don't think this can be dealt with efficiently at link time, as it would basically require all template "algorithms" to be recorded as bytecode somehow. That, or simply forbidding sizeof() or alignof() to be used on an interface in the context of a template, which makes the idea incompatible with half the language...

Might be an issue, but do not understand it. I do not think allignment would be an issue in places where it was not an issue, or it will be an issue where it was an issue. In the class layout the alignment will be as before, with same freedom and restrictions. Do you mean at runtime when the members are referenced? Well... I need help there... did not study the issue.

 

Concerning keywords: I don't really care what is chosen, just as long as they don't turn nested definitions and the like into keyword soup or become easily confused with other commonly-defined symbols (such as color enums). My policy about keywords is that the result needs to be reasonably readable as monochrome text...

I am kind of convinced that so far the syntax should be

class private {... }

class public {...}

[mobi phil]

I exaggerated a bit there... Let me clarify: If a "white" can double as a "black" (or there is a compatible "black" in the TU), then sizeof() and alignof() are well-defined, but it absolutely must be deduced at compile time.

what do you mean compatible? Same layout, same size? Well then we missed the point of the full story. Again, I emphasize the reason of having chosen the terms black and white. Black box is black box... thus it contains the 

 

The case where it can't be used in a template really isn't that different than applying sizeof() to an incomplete type (pure "white" is fundamentally incomplete)...

still needs to be analyzed.. but I tend to think the same. If alloca is standard then class could be allocated on stack. I still fail to see how call by value could be solved if sizeof is not known.

Not sure if I mentioned in my original post, but this black/white story is mainly intended for "big classes" which will not be candidate for call by value.

[Chris Gary]

still needs to be analyzed.. but I tend to think the same. If alloca is standard then class could be allocated on stack. I still fail to see how call by value could be solved if sizeof is not known.

The compiler has to reduce a 'sizeof(button)' or 'alignof(button)' to a constant integer and fold any dependent constant expressions or templates dependent thereupon before sending the whole thing to code generation. It is not possible to know size or alignment if the precise content and layout of a 'button' is only speculative or unspecified.

Not sure if I mentioned in my original post, but this black/white story is mainly intended for "big classes" which will not be candidate for call by value.

Classes are treated as general categories, regardless of size. This proposal will need to work for classes of any size, even empty ones.

[mobi phil]

still needs to be analyzed.. but I tend to think the same. If alloca is standard then class could be allocated on stack. I still fail to see how call by value could be solved if sizeof is not known.

The compiler has to reduce a 'sizeof(button)' or 'alignof(button)' to a constant integer and fold any dependent constant expressions or templates dependent thereupon before sending the whole thing to code generation. It is not possible to know size or alignment if the precise content and layout of a 'button' is only speculative or unspecified.

I did not wake up on this idea few days ago. Had it in my mind for a while and it was clear for me from the beginning the two main issues: sizeof() and layout (offsets). This two problems were repeated through the thread, but it is my fault that I did not write a more formal paper where these things are described so nobody wastes time to re-iterate the same thing. In current implementation indeed not easy to fill in this gap, but IMHO is doable. 

Not sure if I mentioned in my original post, but this black/white story is mainly intended for "big classes" which will not be candidate for call by value.

Classes are treated as general categories, regardless of size. This proposal will need to work for classes of any size, even empty ones.

The point I tried to make is that not knowing sizeof() at compilation time may may cause some implementation issues for for call by value or reference. If it turns out that will be impossible to implement call by value or by reference, it should not be considered as a catastrophe and the proposal rejected for this reason. But this may be just a false alarm due to my limited knowledge of how call by reference and by value is translated. Once I have time will have a look on how code is generated for different layouts etc.

[Chris Gary]

*snip* Thus cannot see too much reason not to have separate black. can you give an example pls.

Consumer:

class public button
{
public:

    string text;
    drawing::color color;
};

[[dllimport]]
button *make_special_button();

void verb()
{
    // WYSIWYG instantiation of button.
    //
    // sizeof(button) MUST reduce to a constant integer at compile time!
    //
    button btn{};
   
    button *btn_ptr = make_special_button();
   
    btn_ptr->text = "Press me!";
}

PIMPL producer:

// Problem:
//
// What if more than one of these are visible in the same TU?
// Use the most recent definition?
// If they all have the same layout and member ordering, don't complain?
//
class private button
{
public:

    // To save typing, this automatically injects
    // members that aren't explicitly declared
    // in the same declaration order, before any
    // member declared in "private"
   
    // "text" implicitly declared before what follows.
   
    HWND hWnd;
    drawing::color color;
};

[[dllexport]]
button *make_special_button()
{
    return new button{/*Create window, set text, blah...*/};
}

[Chris Gary]

On Thursday, February 19, 2015 at 6:06:01 AM UTC-7, mobiphil wrote:
 

*snip*

The point I tried to make is that not knowing sizeof() at compilation time may may cause some implementation issues for for call by value or reference. If it turns out that will be impossible to implement call by value or by reference, it should not be considered as a catastrophe and the proposal rejected for this reason. But this may be just a false alarm due to my limited knowledge of how call by reference and by value is translated. Once I have time will have a look on how code is generated for different layouts etc.

This is why I keep repeating "sizeof() must reduce to a constant at compile time." As templates constitute what is basically a functional language of categories and simple integer expressions, we would need a way to modify already-compiled binaries to accept a type with an unknown layout and alignment (want some more bytecode with your bytecode?). I believe similar issues were encountered with "export templates"... Imagine some exported dependent template that does not terminate for some odd size of a type; how difficult it might be to isolate that kind of bug (unless we have a solution to the halting problem somewhere). So, keeping things simple, we either forbid sizeof(), new, etc.. without a visible PIMPL, or just invoke WYSIWYG.

[mobi phil]

try to find also my comments in your example

*snip* Thus cannot see too much reason not to have separate black. can you give an example pls.

Consumer:

class public button
{
public:

    string text;  // MOBIPHIL: still needs to be decided if this means a "real" data that extends the interface or automatically a "virtual" one that needs to be resolved by a black

    drawing::color color;
};

[[dllimport]]
button *make_special_button();

void verb()
{
    // WYSIWYG instantiation of button.
    //

    // sizeof(button) MUST reduce to a constant integer at compile time! MOBIPHIL: not necessarily! if behind the scene the compiler could generate sthg. like char *buffer =  alloca(button_sizeof()); new (buffer) blahblah...

    //
    button btn{};
   
    button *btn_ptr = make_special_button();
   
    btn_ptr->text = "Press me!";
}

PIMPL producer:

// Problem:
//
// What if more than one of these are visible in the same TU?

// Use the most recent definition? MOBIPHIL: you will get duplicated symbol for symbols (either functions or globals, depending on the implementation) that are defining the sizeof and offsets of visible members

// If they all have the same layout and member ordering, don't complain?
//
class private button
{
public:

    // To save typing, this automatically injects
    // members that aren't explicitly declared
    // in the same declaration order, before any

    // member declared in "private"  MOBIPHIL: no real objections against injection... see further comment below

   
    // "text" implicitly declared before what follows.
   
    HWND hWnd;
    drawing::color color;
};

[[dllexport]]
button *make_special_button()
{
    return new button{/*Create window, set text, blah...*/};
}

for me the black will define some symbols that need to be available at link time, such symbols are black_sizeof(), and offsets of the attributes that are accessed from consumers either as setter/getter or through this "virtual member" mechanism that makes sense. Any "white" declaration will create dependency on such symbols, thus if the black is not available then these symbols would be not defined. Remember that I am still proposing two approaches for symbol resolution

-> link time (link time inlining of setters/getters ... thus the virtual data members)

-> load time with pointers

The injection you mention in your example can make sense but there are two contradictory approaches, I was riding more the second:

1. what you mention here that the memebres get automatically injected from the "public" (white) and a link should be made someway between the two. This will not allow however the option to have "real" public members. Nevertheless allowing "real" public members would make implementation even more difficult.

2. The other way is to have a distinction between "virtual" and normal members on the "white". Virtual ones would be linked with the one on the black. 

In the design I am envisaging you do not need the factory method make_special_button. However it will work like a factory method.

[mobi phil]

*snip*

The point I tried to make is that not knowing sizeof() at compilation time may may cause some implementation issues for for call by value or reference. If it turns out that will be impossible to implement call by value or by reference, it should not be considered as a catastrophe and the proposal rejected for this reason. But this may be just a false alarm due to my limited knowledge of how call by reference and by value is translated. Once I have time will have a look on how code is generated for different layouts etc.

This is why I keep repeating "sizeof() must reduce to a constant at compile time." As templates constitute what is basically a functional language of categories and simple integer expressions, we would need a way to modify already-compiled binaries to accept a type with an unknown layout and alignment (want some more bytecode with your bytecode?). I believe similar issues were encountered with "export templates"... Imagine some exported dependent template that does not terminate for some odd size of a type; how difficult it might be to isolate that kind of bug (unless we have a solution to the halting problem somewhere). So, keeping things simple, we either forbid sizeof(), new, etc.. without a visible PIMPL, or just invoke WYSIWYG.

can we please someway agree that if sizeof has to reduce to a constant at compile time is in serious contradiction what this proposal try to achive?

The point is that you add minimum to the header, you freeze it, and then you do your development cycles in the cpp... Once the data layout is in the black, you cannot have the sizeof.

About your argument above, do not say no, but one has to analyze how much size is affecting templates and how much templates can do with the data that stays in the white. Speculations in one direction or other do not really help to much.

[Chris Gary]

This should illustrate my argument a little more clearly:

namespace _impl{

    using AnySmallBfr = uintptr_t[3];
   
    struct _basic_any_blob
    {
        virtual void _destroy() noexcept = 0;
       
        // Destructor not called!
    };

    template<
        typename InnerType
        bool isSmall = (
        sizeof(InnerType) <= sizeof(AnySmallBfr)
        )
    >
    struct _any_blob : _basic_any_blob
    {
        InnerType _that;
       
        void _destroy() noexcept override
        {this->_that->~InnerType();}
       
        // Rest of implementation
    };

    template<typename InnerType>
    struct _any_blob<InnerType, false>
    {
        // Too big, need to allocate separately
        InnerType *_that_ptr;
       
        void _destroy() noexcept override
        {delete this->_that_ptr;}
       
        // Rest of implementation
    };

}// namespace _impl

class any
{
    _impl::AnySmallBfr _small_bfr;
   
public:

    template<typename ValueType>
    any(ValueType &&value)
    {
        // Decides between small and large
        // backing store.
        //
        // If sizeof(ValueType) <= sizeof(_small_bfr),
        // placement-new into _small_bfr.
       
        // This should compile statically, but...
        //
        // Are you suggesting we automate code generation
        // to handle structures like _any_blob<> parameterized
        // by "white" types?
        //
        new(this->_small_bfr) _impl::_any_blob<
            decay<ValueType>
            >{forward<ValueType>(value)};
    }
   
    ~any() noexcept
    {
        reinterpret_cast<_impl::_basic_any_blob &>(
            this->_small_bfr
            )._destroy();
    }
};

Consumer using an 'any':

class public thing
{
public:

    string in_thing;
};

void verb()
{
    // What happens if the resulting object's
    // size depends on something loaded externally?
    //
    // Also: What if we have multiple implementations linked? Which 'thing' is chosen?
    //
    any wut = thing{};
}

Producer with another thing:

// Too big for internal buffer of 'any'
//
class private thing
{
public:

    string in_thing;
    int hidden_int;
    float hidden_float;
    char hidden_chars[16];
};

[Chris Gary]

I need to point out the sample I gave was hacked together in a few minutes and is riddled with bugs. The point here is illustrated in the constructor of 'any'.

[mobi phil]

sorry, first before I try to understand the point you try to make with the example, let's clarify one thing: you are intentionally using sizeof (probably to explain your concern), whereas it is clear that what I am proposing can be implemented if the dependency on the sizeof is eliminated or can be delayed to link time or load time.

again: not all classes can be used in all situations. A class can be used in a certain situation (template parameter etc.) if it implements a certain protocol. In our case the class does not implement the compile time sizeof protocol, thus it cannot be used in situation where this protocol is needed. This does not mean that the proposal has no value etc. We are playing with concept not with a universal machine, or theory of everything :)

 

--

---
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposal...@isocpp.org.
To post to this group, send email to std-pr...@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/std-proposals/.

[Thiago Macieira]

On Thursday 19 February 2015 14:05:59 mobi phil wrote:
> The point I tried to make is that not knowing sizeof() at compilation time
> may may cause some implementation issues for for call by value or
> reference. If it turns out that will be impossible to implement call by
> value or by reference, it should not be considered as a catastrophe and the
> proposal rejected for this reason. But this may be just a false alarm due
> to my limited knowledge of how call by reference and by value is
> translated. Once I have time will have a look on how code is generated for
> different layouts etc.

I can tell you right now: without knowing the size, passing by value in the
current way "pass-by-value" is understood will be impossible, period.

Passing by reference or by pointer should be possible, since those already
work even on incomplete types.

A solution for the by-value case is to treat it as passing by
pointer/reference in the ABI. That is, in most ABIs, the compiler will have to
reserve stack space for the dynamic size, copy the content, then pass the
address of that space.

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
PGP/GPG: 0x6EF45358; fingerprint:
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358

[mobi phil]

I can tell you right now: without knowing the size, passing by value in the
current way "pass-by-value" is understood will be impossible, period.

Passing by reference or by pointer should be possible, since those already
work even on incomplete types.

A solution for the by-value case is to treat it as passing by
pointer/reference in the ABI. That is, in most ABIs, the compiler will have to
reserve stack space for the dynamic size, copy the content, then pass the
address of that space.

that was exactly my concern, however, without any intention to offend: do you exactly know the implementation difference between call by reference and call by value? I am afraid from the called function point of view that is NO difference. On the caller side you will have the original object and a copy of it will be created before the call. Now if the original object can be created with alloca then the copy also can be created with alloca. Do not see any difference here.

[Thiago Macieira]

On Thursday 19 February 2015 21:41:52 mobi phil wrote:
> that was exactly my concern, however, without any intention to offend: do
> you exactly know the implementation difference between call by reference
> and call by value?

Yes.

Under certain constraints, even aggregates might be passed in registers. This
really depends on the ABI and the psABI in question, so YMMV quite a lot. You
can read more about this in my blog:
http://www.macieira.org/blog/2012/02/the-value-of-passing-by-value/

Any type that can't be passed in registers is passed by hidden reference: a
copy is spilled to the stack and its address is passed in place of the type
itself.

Black or white types will always need to be passed by hidden reference.

> I am afraid from the called function point of view that
> is NO difference. On the caller side you will have the original object and
> a copy of it will be created before the call. Now if the original object
> can be created with alloca then the copy also can be created with alloca.
> Do not see any difference here.

[mobi phil]

Black or white types will always need to be passed by hidden reference.

missing some optimization opportunities should not be interpreted as the end of the world or as to say "period" like you stated earlier

>>I can tell you right now: without knowing the size, passing by value in the
>>current way "pass-by-value" is understood will be impossible, period.

 

can you please tell me, why you said it would be impossible? Now you seem to have reduced the "impossible" to the handicap of not being able to pass in registers. I would love to have time to read your blog and other millions of books, article on the topic. Best is to look at generated code, no blog/book/video/etc. can tell me more than. What I see so far that it is feasable. If some mini optimization opportunities are lost, well, you always have to tread sthg. for sthg. else...

I am personally ok with this little "disadvantage". If I want to hunt for that little piece of cake, then I will move my blacks to the white and get it once the project reached a stable phase. I am also thinking about some details on how to move black back if optimization is to be achieved.

Unfortunately most of the projects I worked on there was less worry about such optimization opportunities like passing an object in registers rather than on stack. Happened often that this brilliant managers did read in some fancy local newspapers that was mad to tell "-O{1,2,3}" to the compiler. It is also true that 15 years ago optimizer were often producing buggy code.  

Do not forget that QT pimple object cannot be passed in registers... Because he is still a pointer... 

So, lets take things easy. Let's do not adventure to far, before we did not fix simple things. Let's do not optimize before we agreed on basic stuff... Would alloca knowing the size from a function or global variable solve the problem of sizeof of black?

[Thiago Macieira]

On Saturday 21 February 2015 04:24:05 mobi phil wrote:
> > Black or white types will always need to be passed by hidden reference.
>
> missing some optimization opportunities should not be interpreted as the
> end of the world or as to say "period" like you stated earlier

I never meant that it was the end of the world. I wrote quite explicitly that
it could never be done as it is currently done, that's it.

> >>I can tell you right now: without knowing the size, passing by value in
>
> the
>
> >>current way "pass-by-value" is understood will be impossible, period.
>
> can you please tell me, why you said it would be impossible? Now you seem

How many registers do you reserve for the structure if you don't know its
size? Selecting the registers dynamically means all other parameters would be
dynamically determined, which seems to me to be horribly inefficient.

It's more efficient to allocate stack space, pass by hidden reference and keep
all the other parameters at fixed positions.

> to have reduced the "impossible" to the handicap of not being able to pass
> in registers. I would love to have time to read your blog and other
> millions of books, article on the topic.

Well, if you're not going to read up on the subject, how can we discuss the
subject?

> Best is to look at generated code,
> no blog/book/video/etc. can tell me more than. What I see so far that it is
> feasable. If some mini optimization opportunities are lost, well, you
> always have to tread sthg. for sthg. else...

How do you think I came up with the content of the blog? By compiling and
generating code, plus reading the ABI manuals.

> I am personally ok with this little "disadvantage". If I want to hunt for
> that little piece of cake, then I will move my blacks to the white and get
> it once the project reached a stable phase. I am also thinking about some
> details on how to move black back if optimization is to be achieved.

Once a black box, always a black box: as you don't know the other side, you
must assume the other side doesn't have access to the white box.

> Do not forget that QT pimple object cannot be passed in registers...
> Because he is still a pointer...

The Private class can't, but the outer, public class could be.

Though for Qt, classes that can be passed in registers are rare because most
are either ref-counted or non-copyable and neither of those types can be
passed in registers.

> So, lets take things easy. Let's do not adventure to far, before we did not
> fix simple things. Let's do not optimize before we agreed on basic stuff...
> Would alloca knowing the size from a function or global variable solve the
> problem of sizeof of black?

Yes. if the compiler can do:

ClassName object = otherObject;

It can pass by value.

[mobi phil]

How many registers do you reserve for the structure if you don't know its
size? Selecting the registers dynamically means all other parameters would be
dynamically determined, which seems to me to be horribly inefficient.

without pretending to be an assembler or code generation guru, I would say one for the pointer and accept that no such optimisations are possible. If such optimisations are desired in some cases, then don't use from the beginnig black/white or move back to normal class at certain moment in the development when the interface is stable. 

 

> to have reduced the "impossible" to the handicap of not being able to pass
> in registers. I would love to have time to read your blog and other
> millions of books, article on the topic.

Well, if you're not going to read up on the subject, how can we discuss the
subject?

My curiosity is there, but not the time unfortunately. Though the devil often is hidden in details, I would like to know first at "high level" what are the pitfalls and what are the possibilities. But at this stage we clarified that both passing by value/reference and pointer would be possible without knowing the size at compilation time.

If we agree that code-generation-time optimization decision like passing through registrer depends on sizeof and sizeof is not available, do not see the point for me to spend lot of time to understand the details.

 

> Best is to look at generated code,
> no blog/book/video/etc. can tell me more than. What I see so far that it is
> feasable. If some mini optimization opportunities are lost, well, you
> always have to tread sthg. for sthg. else...

How do you think I came up with the content of the blog? By compiling and
generating code, plus reading the ABI manuals.

great. Reading of your post is on my todo. Right now I am busy to finish this mini project for gdb interface.

> I am personally ok with this little "disadvantage". If I want to hunt for
> that little piece of cake, then I will move my blacks to the white and get
> it once the project reached a stable phase. I am also thinking about some
> details on how to move black back if optimization is to be achieved.

Once a black box, always a black box: as you don't know the other side, you
must assume the other side doesn't have access to the white box.

Sorry... did not understand this as a whole, but pb. not important

 

> Do not forget that QT pimple object cannot be passed in registers...
> Because he is still a pointer...

The Private class can't, but the outer, public class could be.

Was talking indeed about the private. So far probably there were no serious complaints about it. And qt is nice and fast.

 

Though for Qt, classes that can be passed in registers are rare because most
are either ref-counted or non-copyable and neither of those types can be
passed in registers.

So, I am happy that we both come to the came conclusion. So I should not worry too much that missing the registers would be a problem in case of blacks. 

 

> So, lets take things easy. Let's do not adventure to far, before we did not
> fix simple things. Let's do not optimize before we agreed on basic stuff...
> Would alloca knowing the size from a function or global variable solve the
> problem of sizeof of black?

Yes. if the compiler can do:

        ClassName object = otherObject;

It can pass by value.

Well, then this topic is rather clear.

What I still need to work out is how private data member access (inside black) will be expressed (syntax) and implemented. The topic was touched in earlier emails. It was proposed to use public/private keywords for black/white and eventually the keyword virtual, for members in white that are "implemented' in he black. The question is either one should go with :

-> functions or

-> pointers

Manually written or automatically generated (also to be decided) functions could be inlined at link time (llvm lto, hopefully this technique will catch-up). This would generate faster code but has the disadvantage that will not be able to load classes from shared libraries which would be another advantage of this proposal. Probably both implementations would make sense, the question is how to make the distinction. 

 

[iskan...@gmail.com]

I'm sorry to involve in this way, but could you please state what is/are the goals you want to achieve by this suggestion/proposal?

I haven't read all the posts under this topic, I'll just summarize the goals I've noticed:

1) separate public interface from internal implementation details

2) enable change of internal implementation without the need of recompilation code that only uses the public interface of the class/struct.

Please could you add other goals which I've missed? I'm sorry if I completely misunderstood your ideas.

An old way to achieve goals 1&2: abstract interface with pure virtual methods:

// white.h:
class IWidget {
public:
  virtual void set(int) = 0;
  virtual int get() const = 0;
};

// black.h:
class WidgetImpl : public IWidget {
public:
  virtual void set(int i) override { x = i; }
  virtual int get() const { return x; }

private:
  int x;
};

... now if you decide to change the implementation of WidgetImpl, there's no need to recompile code that uses only IWidget.

I can't imagine setup where this same behavior would be achieved by using non-virtual methods. Let's consider code using the black/white keywords (I might missed the proposed syntax, please correct me if I'm wrong):

// white.h:
class white Widget {
public:
  void set(int i) { setInternal(i); }
  int get() const { return getInternal(); }
};

// black.h:
class black Widget {
private:
  void setInternal(int i) { x = i; }
  int getInternal() const { return getInternal2(); }

private:
  int getInternal2() const { return getInternal3(); }
  int getInternal3() const { return x; }

private:
  int x;
};

Then let's assume a use of Widget:

// foo.cpp:
#include "white.h"
void foo() {
  Widget w;
  w.set(2);
  std::cout << w.get() << endl;
}

... here I think any reasonable compiler would inline the call of Widget::get to just reading directly the private member Widget::x (there is no virtual call, no obstacles (if not the black/white keywords)).

But by inlining, the compiled object file of foo.cpp now holds information about not only internal structure of black Widget but also about its implementation. So whenever the implementation of black Widget changes, foo.cpp has to be recompiled to refresh that information.

IMHO either:

a) foo can see the entrails of black Widget -> then foo.cpp must be recompiled on every black Widget change. (we didn't achieve goal #2)

or:

b) foo can be optimized only up to the implementation of white Widget, not further. -> then it seems to me similar to calling a virtual method. (Yes, it might be more efficient then calling an explicitly virtual method. But I'm not sure if all this brings enough value).

[mobi phil]

anybody is welcome to be involved. Nevertheless your example just
introduces a level the dependency but does note break it. The entity
that will create WidgetImpl will still be dependent on the layout of
the class and you are introducing a parallel hierarchy exactly like
PIMPL. Plus dependency wise this is worst IMHO that PIMLP as the the
code that creates WidgetImpl still depends on WidgetImpl, or you
introduce a third hierarchy of factories.

> I can't imagine setup where this same behavior would be achieved by using non-virtual methods.

Yes, PIMPL does sthg. similar, but not with virtuals but with
delegation. The proposal wants to simplify all this.

Did you check yourself that, or it is pure assumption. Such
assumptions are making discussions more difficult. You are practically
delegating the work to the other side to prove the opposite. The
proposal will contains reccomandation to do such inlining at link time
which in my opinion.


Sorry, I am afraid that the compiler cannot inline something it is not
aware about. When you compile a unit that includes white.h (above) is
not aware about black.h, unless you explicitly include it, which does
not result from your example! How does your foo.cpp know about
black.h? Just magically??


Please verify first your assumptions and come back with a cleaner example.

regards,
mobiphil

[iskan...@gmail.com]

You're right, I must apologize for my confusion in what compiler and linker does. The inlining is done in linker, not in compiler, so my assumption about foo.cpp object file is completely wrong. 

As I look at it, this proposal can cause a lot of work to be moved from compiler to linker. I think it would be interesting to do some benchmarking if a complete rebuild (compilation and linking) after an implementation change is much faster then in current design.

If we move a lot of work out of compilation, then it's quite prospective, that the compilation itself will be faster. -> there's a less difference if the compilation actually happens or not.

[mobi phil]

> You're right, I must apologize for my confusion in what compiler and linker
> does. The inlining is done in linker, not in compiler, so my assumption
> about foo.cpp object file is completely wrong.

As far as I know at the moment there is no inlining done in linker in
legacy compilation. Some new features of llvm lto allows link time
inlineing and the proposal among others will try to build on that
feature.

> As I look at it, this proposal can cause a lot of work to be moved from
> compiler to linker. I think it would be interesting to do some benchmarking
> if a complete rebuild (compilation and linking) after an implementation
> change is much faster then in current design.

Obviously no benchmarking will be possible till it is implemented.
Link time inlining is an option of the proposal, but I assume that the
link time inlining will be faster if it will not have to re-inline for
objects that did not change, so the build system itself will have an
impact on this.

[inkwizyt...@gmail.com]

I think that is possible to benchmark 90% of this functionality today. Only thing we lack is `sizeof` but this can be work around by using `std::aligned_storage` that have size margin (you should be able add couple of new members before you run out of space in it).


Another thing is that on C++ VLA committee reject runtime `sizeof`, do you think they will allow it in your proposal? Even if we allow it you still can't add members of that type to another class. Its because every access of members declared after it will require function call for correct offset.
I think it would be better if we require that "black" declare max size of "white" part. This will made it a lot of easer to implement (e.g. it will be possible to do today with lot of boilerplate) and pass committee.

[mobi phil]

> I think that is possible to benchmark 90% of this functionality today. Only
> thing we lack is `sizeof` but this can be work around by using
> `std::aligned_storage` that have size margin (you should be able add couple
> of new members before you run out of space in it).

sorry, how did you calculate this 90%? And bench marking what exactly?
The design is not yet even ready, we do not know what needs to be
changed in the compilers and linkers. Benchmarking which build? Can
you please make more clear statements?

> Another thing is that on C++ VLA committee reject runtime `sizeof`, do you
> think they will allow it in your proposal?

Did they reject in the past? Do they always reject? You assume they
are going to reject? Can you also pls. be more clear here?

> Even if we allow it you still
> can't add members of that type to another class. Its because every access of
> members declared after it will require function call for correct offset.

Sorry, this one not clear either? You mean function call to get the
offset? If you have the function to get the offset, why would it be a
problem. You are contradicting yourself in the same statement. Another
option to offset function is the link time inlining. Please read
through the thread, you may have some of the questions clarified.

> I think it would be better if we require that "black" declare max size of
> "white" part. This will made it a lot of easer to implement (e.g. it will be
> possible to do today with lot of boilerplate) and pass committee.

Max size is out of question. It is hack, I would personally downwoat
sthg. like that. How would you decide on max size? You will have 300
hundred classes and you will spend 30% of effort in adjusting the max
sizes? That would work in o toy application, but not on giant
projects. Would you just waste memory because design is bad and memory
is cheap?

[inkwizyt...@gmail.com]

On Wednesday, February 25, 2015 at 7:20:03 PM UTC+1, mobiphil wrote:

> I think that is possible to benchmark 90% of this functionality today. Only
> thing we lack is `sizeof` but this can be work around by using
> `std::aligned_storage` that have size margin (you should be able add couple
> of new members before you run out of space in it).

sorry, how did you calculate this 90%? And bench marking what exactly?
The design is not yet even ready, we do not know what needs to be
changed in the compilers and linkers. Benchmarking which build? Can
you please make more clear statements?

This is approximation, meaning of this was that is possible to recreate most of this using current tools, but without any syntax support.
 

> Another thing is that on C++ VLA committee reject runtime `sizeof`, do you
> think they will allow it in your proposal?

Did they reject in the past? Do they always reject? You assume they
are going to reject? Can you also pls. be more clear here?

All proposal that didn't have fixed size objects avoided declaring `sizeof` for them. e.g.
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3875.pdf
And I recall some discussion about it, that people form committee didn't like it.
I tired find link to it but right now I dint find it, maybe someone els remember this?

> Even if we allow it you still
> can't add members of that type to another class. Its because every access of
> members declared after it will require function call for correct offset.

Sorry, this one not clear either? You mean function call to get the
offset? If you have the function to get the offset, why would it be a
problem. You are contradicting yourself in the same statement. Another
option to offset function is the link time inlining. Please read
through the thread, you may have some of the questions clarified.

Consider this classes:

class black A1 { void foo(); };
class black A2 { void foo(); };


class B
{
    int i;
    A1 a1;
    int c;
    A2 a2;
    int d;
};

int main()
{
    B b;
    b.c = 5; //require function call to get offset value.
    b.d = 3; //2x call for A1 and A2
    b.a2.foo();
    temp<&B::d> something; //error, this require runtime values.
    int i = sizeof(b); //runtime value too
}

 

> I think it would be better if we require that "black" declare max size of
> "white" part. This will made it a lot of easer to implement (e.g. it will be
> possible to do today with lot of boilerplate) and pass committee.

Max size is out of question. It is hack, I would personally downwoat
sthg. like that. How would you decide on max size? You will have 300
hundred classes and you will spend 30% of effort in adjusting the max
sizes? That would work in o toy application, but not on giant
projects. Would you just waste memory because design is bad and memory
is cheap?

 No, I'm more concern with complexity of this when another objects depends on run time size of another object.

[mobi phil]

>> sorry, how did you calculate this 90%? And bench marking what exactly?
>> The design is not yet even ready, we do not know what needs to be
>> changed in the compilers and linkers. Benchmarking which build? Can
>> you please make more clear statements?
>>
> This is approximation, meaning of this was that is possible to recreate most
> of this using current tools, but without any syntax support.

Still not clear what you want to measure. Even if it would be already
implemented you would have hard time to measure. You measure what
against what? Build times would be clearly 1000 times faster if you do
not have to recompile 1000 object files with same complexity each time
you change the definition of a class they are depending on. The same
time you can have a simple build where this will bring zero added
value, more it will be a little bit slower. About runtime performance
difference: there will be to solutions. link time inlining or "runtime
sizeof". The first will be a bit slower to build, the second will be
slower to run. How much slower? dependends on the situation. If you
have 50 times access to the same member inside one function the
compiler is supposed to cache the offset. So the degradation will not
be probaly that significant, but again will depend on the situation.
If getters/setters will be inlined by the linker, I expect almost zero
degradation.

Please define what and how you want to measure.

>> > Another thing is that on C++ VLA committee reject runtime `sizeof`, do
>> > you
>> > think they will allow it in your proposal?
>>
>> Did they reject in the past? Do they always reject? You assume they
>> are going to reject? Can you also pls. be more clear here?
>>

> All proposal that didn't have fixed size objects avoided declaring `sizeof`
> for them. e.g.
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3875.pdf
> And I recall some discussion about it, that people form committee didn't
> like it.
> I tired find link to it but right now I dint find it, maybe someone els
> remember this?

Please do not confuse with some proposals that had as object
dynamically growing objects or anything in that direction. In this
proposal the size is not known at compile time, but it will not change
at runtime. The proposal however keeps it open dynamically loaded
classes and in that case offset and size have to be dynamic.

Did you think how you would implement sizeof at runtime? Without being
a compiler or assembler guru, I see three solutions (all these details
were described already in this thread):

1. having the linker going through all the generated code and replace
occurences of some "markers" with the sizeof available at link time
from the black
2. indirect reference. The sizeof would be an offset in a global table
that is arranged by the dynamic loader
3. with function()

The same applies to offsets

[inkwizyt...@gmail.com]

On Wednesday, February 25, 2015 at 8:56:53 PM UTC+1, mobiphil wrote:

>> sorry, how did you calculate this 90%? And bench marking what exactly?
>> The design is not yet even ready, we do not know what needs to be
>> changed in the compilers and linkers. Benchmarking which build? Can
>> you please make more clear statements?
>>
> This is approximation, meaning of this was that is possible to recreate most
> of this using current tools, but without any syntax support.

Still not clear what you want to measure. Even if it would be already
implemented you would have hard time to measure. You measure what
against what? Build times would be clearly 1000 times faster if you do
not have to recompile 1000 object files with same complexity each time
you change the definition of a class they are depending on. The same
time you can have a simple build where this will bring zero added
value, more it will be a little bit slower. About runtime performance
difference: there will be to solutions. link time inlining or "runtime
sizeof". The first will be a bit slower to build, the second will be
slower to run. How much slower? dependends on the situation. If you
have 50 times access to the same member inside one function the
compiler is supposed to cache the offset. So the degradation will not
be probaly that significant, but again will depend on the situation.
If getters/setters will be inlined by the linker, I expect almost zero
degradation.

Please define what and how you want to measure.

 
 I see I miss your point, because I think of runtime benchmarking for the whole time.

 

>> > Another thing is that on C++ VLA committee reject runtime `sizeof`, do
>> > you
>> > think they will allow it in your proposal?
>>
>> Did they reject in the past? Do they always reject? You assume they
>> are going to reject? Can you also pls. be more clear here?
>>

> All proposal that didn't have fixed size objects avoided declaring `sizeof`
> for them. e.g.
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3875.pdf
> And I recall some discussion about it, that people form committee didn't
> like it.
> I tired find link to it but right now I dint find it, maybe someone els
> remember this?

Please do not confuse with some proposals that had as object
dynamically growing objects or anything in that direction. In this
proposal the size is not known at compile time, but it will not change
at runtime. The proposal however keeps it open dynamically loaded
classes and in that case offset and size have to be dynamic.

But in both behavior of `sizeof` will change. Probably biggest problem will be using it in metaprograming simple code like that: `test<(sizeof(x) > 4)>`.

I think only first one is good for this. But "markers" will probably don't work, because multiple black can affect one `sizeof`.
Probably more possible implementation would be list of instruction address that need added value of black size to const operand.

Second have two problems, one that require constant keeping this table in cache when you use black objects another is that all offset instruction will require couple step to calculate final offset.

Third one will probably not pass because of overhead compared to accessing meber by const offset. Its posible to store value, but it will be recalculated for each new context.
And if it done properly it probably end up as second point.

Overall most of my concerns could be avoided by first implementation, but template problem is unfixable.

[mobi phil]

> I see I miss your point, because I think of runtime benchmarking for the
> whole time.

Even if we talk about runtime bench-marking, the measurement would be
complex. You cannot measure without defining the problem space. If you
have suggestions, please define what you would like to measure. Just
throwing in numbers and random ideas does not help.

> But in both behavior of `sizeof` will change. Probably biggest problem will
> be using it in metaprograming simple code like that: `test<(sizeof(x) >
> 4)>`.

May I kindly ask you to go through all emails. This problem was
already addressed. The proposal does not intend to replace all the
usage of all the classes in all the templates in all the world. There
are classes that do not provide other kind of protocols that make them
unusable in thousands of other templates. The proposal will not
satisfy the interface for sizeof at compiltime and with that this part
of topic is closed.

>> 1. having the linker going through all the generated code and replace
>> occurences of some "markers" with the sizeof available at link time
>> from the black
>> 2. indirect reference. The sizeof would be an offset in a global table
>> that is arranged by the dynamic loader
>> 3. with function()
>>
>> The same applies to offsets
>
>
> I think only first one is good for this. But "markers" will probably don't
> work, because multiple black can affect one `sizeof`.
> Probably more possible implementation would be list of instruction address
> that need added value of black size to const operand.
>
> Second have two problems, one that require constant keeping this table in
> cache when you use black objects another is that all offset instruction will
> require couple step to calculate final offset.

> Third one will probably not pass because of overhead compared to accessing
> meber by const offset. Its posible to store value, but it will be
> recalculated for each new context.
> And if it done properly it probably end up as second point.
>
> Overall most of my concerns could be avoided by first implementation, but
> template problem is unfixable.

did you ever heard of the term trading something against something?
Did you hear about the theory of relativity? If you get something then
often you have to give up something else. So it is with this proposal!
It does not pretend to make the word much better. It wants to be a
solution for a certain problem. During development if one realises
that the black/white means serious performance, it should be possible
to make the black/white into an original class

[David Krauss]

On 2015–02–27, at 6:31 AM, mobi phil <mo...@mobiphil.com> wrote:

May I kindly ask you to go through all emails. This problem was
already addressed. The proposal does not intend to replace all the
usage of all the classes in all the templates in all the world. There
are classes that do not provide other kind of protocols that make them
unusable in thousands of other templates. The proposal will not
satisfy the interface for sizeof at compiltime and with that this part
of topic is closed.

You can summarily stop a part of this discussion so easily now, but once the committee realizes that you’re proposing sizeof expressions that aren’t compile-time constants, they’ll stop the entire proposal equally swiftly. I’ve not been following this thread, but you might be better off saying that sizeof cannot be applied to certain types at all, and introducing a new runtime function to take its place if necessary.

By the way, the VLA/“classes of runtime size” proposal did not provide for varying object size at runtime, to my knowledge.

[David Krauss]

On 2015–02–27, at 7:44 AM, David Krauss <pot...@gmail.com> wrote:

By the way, the VLA/“classes of runtime size” proposal did not provide for varying object size at runtime, to my knowledge.

I mean, changing the size of an object already created, as suggested a couple messages ago. Of course it allowed for deciding the size of an object upon construction.

[mobi phil]

> I mean, changing the size of an object already created, as suggested a
> couple messages ago. Of course it allowed for deciding the size of an object
> upon construction.

Jumping in and while landing grabbing a random apple from the tree,
would not bring too much added value, isn't it? ;)

Can you please read more carefully the thread and avoid statements
like "changing the size of an object already created, as suggested a
couple messages ago..". At least myself I did not make such
irresponsible statements. You confused with variations on the idea
about loading the full class from a dynamic library which means
together with sizeof() and equivalent functions and offset providing
functions that satisfy the white interface.

You make me repeating the same trivial statement yet the x'th time:
the runtime "sizeof" detail is an option. The main proposal is
focusing on link time inlining. Thus you missed the point, it wont be
compile time constant sizeof, it wont't be runtime, but it wants to be
link time constant. For the rest if you bother, just go through the
thread.

[David Krauss]

> On 2015–02–27, at 9:36 AM, mobi phil <mo...@mobiphil.com> wrote:
>
> Can you please read more carefully the thread and avoid statements
> like "changing the size of an object already created, as suggested a
> couple messages ago..". At least myself I did not make such
> irresponsible statements.

“Please do not confuse with some proposals that had as object dynamically growing objects or anything in that direction.” You said this in reference to N3875, and I just wanted to clarify that N3875 (and various related or similar papers) don’t have that either, yet they have run into trouble.

I don’t think it makes much difference whether a size is determined at link/load time or upon a function call.

> For the rest if you bother, just go through the thread.

This thread has become insanely long. Where am I supposed to start? Perhaps now is a good time to start compiling your findings into a comprehensive paper.

[mobi phil]

> I don’t think it makes much difference whether a size is determined at link/load time or upon a function call.

The statement is ambiguous. While it will obviously make difference
for the compiler, as the feature has to be implemented, it will also
make huge difference if a function call could be replaced by linker
with a "link time constant" that is link-time-inlined. For some
reasons LLVM guys are working hard on such optimizations. It is easy
to reject by blaming potential run-time performance degradation, but
if that is excluded then
If C++ committee thinks only in terms of technology of 20 years ago,
indeed there will be plenty of reasons to refuse anything.

Paper will be ready soon, please do not blame me for the thread
becoming long, could have ignored all replies but that was obviously
not my interest.