issues with std::variant

[wthol...@gmail.com]

I'm using std::variant in my code for one very important type (`Node`). This variant has 68 different types in it. There are two issues:

1. Compile times are extremely slow (~30s per file), due to variant visitation. I'm an indie developer and I don't want to have to buy a new computer (might only be a factor of two faster anyway).

2. If I hire a contractor to help, they might be baffled by the c++ fancyness, such as:

Node setExposedPosition(Node node, Vec2 p) {
std::visit(overloaded {
[p](auto& n, decltype(n.exposedPosition)* = nullptr) {
n.exposedPosition = p;
},
[](auto&...) { }
}, node);
return node;
}

(This kind of stuff is all over the place, because it eliminates a LOT of redundant code)

Now that's pretty cool pattern matching, but let's face it, this would be much more understandable:

Node setExposedPosition(Node node, Vec2 p) {
if(hasField(node, EXPOSED_POSITION_BIT)) {
node.exposedPosition = p;
}
return node;
}

I've found that if I simply make Node a struct with every possible data member from all the variant types, it only increases the size of Node by a factor of two, which is totally fine. (I could use std::optional for each member, but that would increase the size, and potentially be more dynamic than I need)

Does that sound like the approach you would take?

thanks!

[Öö Tiib]

On Wednesday, 4 December 2019 18:37:50 UTC+2, Taylor wrote:
> I'm using std::variant in my code for one very important type (`Node`). This variant has 68 different types in it. There are two issues:
>
> 1. Compile times are extremely slow (~30s per file), due to variant visitation. I'm an indie developer and I don't want to have to buy a new computer (might only be a factor of two faster anyway).
>
> 2. If I hire a contractor to help, they might be baffled by the c++ fancyness, such as:
>
> Node setExposedPosition(Node node, Vec2 p) {
> std::visit(overloaded {
> [p](auto& n, decltype(n.exposedPosition)* = nullptr) {
> n.exposedPosition = p;
> },
> [](auto&...) { }
> }, node);
> return node;
> }
>
> (This kind of stuff is all over the place, because it eliminates a LOT of redundant code)

Huh? The design feels odd indeed.
I have usually stuff in variants that have rare things if anything
at all in common.

>
> Now that's pretty cool pattern matching, but let's face it, this would be much more understandable:
>
> Node setExposedPosition(Node node, Vec2 p) {
> if(hasField(node, EXPOSED_POSITION_BIT)) {
> node.exposedPosition = p;
> }
> return node;
> }

External setter? I do not have setter members no idea why I
would want external setters.

>
> I've found that if I simply make Node a struct with every possible data member from all the variant types, it only increases the size of Node by a factor of two, which is totally fine. (I could use std::optional for each member, but that would increase the size, and potentially be more dynamic than I need)
>
> Does that sound like the approach you would take?

No. Only by factor of two? Then those are rather similar classes.

From 68 mostly similar to each other "node" classes I would make plain
old polymorphic hierarchy.
If there is need to keep objects of these classes in a single
container then I would use Boost.PolyCollection for efficiency.
https://www.boost.org/doc/libs/1_71_0/doc/html/poly_collection.html

That design beats both your variant and your monster struct blindfolded
with right hand tied behind back I can bet.

> thanks!

Oh ... no problem.

[Taylor]

Thanks for your help!

On Wednesday, December 4, 2019 at 10:26:52 AM UTC-8, Öö Tiib wrote:
> On Wednesday, 4 December 2019 18:37:50 UTC+2, Taylor wrote:
> > I'm using std::variant in my code for one very important type (`Node`). This variant has 68 different types in it. There are two issues:
> >
> > 1. Compile times are extremely slow (~30s per file), due to variant visitation. I'm an indie developer and I don't want to have to buy a new computer (might only be a factor of two faster anyway).
> >
> > 2. If I hire a contractor to help, they might be baffled by the c++ fancyness, such as:
> >
> > Node setExposedPosition(Node node, Vec2 p) {
> > std::visit(overloaded {
> > [p](auto& n, decltype(n.exposedPosition)* = nullptr) {
> > n.exposedPosition = p;
> > },
> > [](auto&...) { }
> > }, node);
> > return node;
> > }
> >
> > (This kind of stuff is all over the place, because it eliminates a LOT of redundant code)
>
> Huh? The design feels odd indeed.
> I have usually stuff in variants that have rare things if anything
> at all in common.

Consider what the function does: it's checking (statically) if the struct has a particular member and setting it.

>
> >
> > Now that's pretty cool pattern matching, but let's face it, this would be much more understandable:
> >
> > Node setExposedPosition(Node node, Vec2 p) {
> > if(hasField(node, EXPOSED_POSITION_BIT)) {
> > node.exposedPosition = p;
> > }
> > return node;
> > }
>
> External setter? I do not have setter members no idea why I
> would want external setters.
>
> >
> > I've found that if I simply make Node a struct with every possible data member from all the variant types, it only increases the size of Node by a factor of two, which is totally fine. (I could use std::optional for each member, but that would increase the size, and potentially be more dynamic than I need)
> >
> > Does that sound like the approach you would take?
>
> No. Only by factor of two? Then those are rather similar classes.
>
> From 68 mostly similar to each other "node" classes I would make plain
> old polymorphic hierarchy.

I used to do a traditional polymorphic hierarchy: it's harder to make value types, and requires much more boilerplate.

> If there is need to keep objects of these classes in a single
> container then I would use Boost.PolyCollection for efficiency.
> https://www.boost.org/doc/libs/1_71_0/doc/html/poly_collection.html
>
> That design beats both your variant and your monster struct blindfolded
> with right hand tied behind back I can bet.

I'm using https://github.com/arximboldi/immer and can't change my containers.

[Öö Tiib]

On Wednesday, 4 December 2019 21:07:37 UTC+2, Taylor wrote:
> Thanks for your help!
>
> On Wednesday, December 4, 2019 at 10:26:52 AM UTC-8, Öö Tiib wrote:
> > On Wednesday, 4 December 2019 18:37:50 UTC+2, Taylor wrote:
> > > I'm using std::variant in my code for one very important type (`Node`). This variant has 68 different types in it. There are two issues:
> > >
> > > 1. Compile times are extremely slow (~30s per file), due to variant visitation. I'm an indie developer and I don't want to have to buy a new computer (might only be a factor of two faster anyway).
> > >
> > > 2. If I hire a contractor to help, they might be baffled by the c++ fancyness, such as:
> > >
> > > Node setExposedPosition(Node node, Vec2 p) {
> > > std::visit(overloaded {
> > > [p](auto& n, decltype(n.exposedPosition)* = nullptr) {
> > > n.exposedPosition = p;
> > > },
> > > [](auto&...) { }
> > > }, node);
> > > return node;
> > > }
> > >
> > > (This kind of stuff is all over the place, because it eliminates a LOT of redundant code)
> >
> > Huh? The design feels odd indeed.
> > I have usually stuff in variants that have rare things if anything
> > at all in common.
>
> Consider what the function does: it's checking (statically) if the struct has a particular member and setting it.

Do not let complex syntax to confuse you ... magic does not exist.
It can't. It generates switch-case that checks *dynamically* behind scenes
what of those 68 is in that particular variant instance "node" and then
sets the member.

> > > Now that's pretty cool pattern matching, but let's face it, this would be much more understandable:
> > >
> > > Node setExposedPosition(Node node, Vec2 p) {
> > > if(hasField(node, EXPOSED_POSITION_BIT)) {
> > > node.exposedPosition = p;
> > > }
> > > return node;
> > > }
> >
> > External setter? I do not have setter members no idea why I
> > would want external setters.
> >
> > >
> > > I've found that if I simply make Node a struct with every possible data member from all the variant types, it only increases the size of Node by a factor of two, which is totally fine. (I could use std::optional for each member, but that would increase the size, and potentially be more dynamic than I need)
> > >
> > > Does that sound like the approach you would take?
> >
> > No. Only by factor of two? Then those are rather similar classes.
> >
> > From 68 mostly similar to each other "node" classes I would make plain
> > old polymorphic hierarchy.
>
> I used to do a traditional polymorphic hierarchy: it's harder to make value types, and requires much more boilerplate.

Value types in sense that little, immutable objects, passed by value?
These do not make sense in context of your post at all.

On the contrary, it is boilerplate of piles of those inefficient
switches from 68 cases all around your code that traditional
polymorphic hierarchy would remove with just one cheap level of
indirection added. I have measured it enough to predict that
program would be at least 4 times quicker, data about 4 times
smaller and executable also about twice smaller.

> > If there is need to keep objects of these classes in a single
> > container then I would use Boost.PolyCollection for efficiency.
> > https://www.boost.org/doc/libs/1_71_0/doc/html/poly_collection.html
> >
> > That design beats both your variant and your monster struct blindfolded
> > with right hand tied behind back I can bet.
>

> I'm using https://github.com/arximboldi/immer ...

I do not know the lib but it is seems from afar that public common
setters sound like total bullshit with those immers that are
immutable after construction. Come again and tell what you really
need.

> ... and can't change my containers.

That is defect of mind. Software is *soft*. It is like drawing on
sand not carving on obsidian.

Most projects usually combine data initially from rather few container
types like:
std::array, std::vector, std::unordered_set, std::unordered_map.
Then, later, when the needs become more clear those are maintained
to be more special, precisely suitable containers.

[Taylor]

I'm not confused. The determination of whether a struct has a member is done statically via SFINAE. Specifically `decltype(n.exposedPosition)* = nullptr`. Of course *which* struct is of course dynamic.

>
> > > > Now that's pretty cool pattern matching, but let's face it, this would be much more understandable:
> > > >
> > > > Node setExposedPosition(Node node, Vec2 p) {
> > > > if(hasField(node, EXPOSED_POSITION_BIT)) {
> > > > node.exposedPosition = p;
> > > > }
> > > > return node;
> > > > }
> > >
> > > External setter? I do not have setter members no idea why I
> > > would want external setters.
> > >
> > > >
> > > > I've found that if I simply make Node a struct with every possible data member from all the variant types, it only increases the size of Node by a factor of two, which is totally fine. (I could use std::optional for each member, but that would increase the size, and potentially be more dynamic than I need)
> > > >
> > > > Does that sound like the approach you would take?
> > >
> > > No. Only by factor of two? Then those are rather similar classes.
> > >
> > > From 68 mostly similar to each other "node" classes I would make plain
> > > old polymorphic hierarchy.
> >
> > I used to do a traditional polymorphic hierarchy: it's harder to make value types, and requires much more boilerplate.
>
> Value types in sense that little, immutable objects, passed by value?
> These do not make sense in context of your post at all.

Not necessarily immutable, but yes passed by value. I'm using value-oriented design.

>
> On the contrary, it is boilerplate of piles of those inefficient
> switches from 68 cases all around your code that traditional
> polymorphic hierarchy would remove with just one cheap level of
> indirection added. I have measured it enough to predict that
> program would be at least 4 times quicker, data about 4 times
> smaller and executable also about twice smaller.

This is not runtime performance critical code. (That's all on another thread)

>
> > > If there is need to keep objects of these classes in a single
> > > container then I would use Boost.PolyCollection for efficiency.
> > > https://www.boost.org/doc/libs/1_71_0/doc/html/poly_collection.html
> > >
> > > That design beats both your variant and your monster struct blindfolded
> > > with right hand tied behind back I can bet.
> >
> > I'm using https://github.com/arximboldi/immer ...
>
> I do not know the lib but it is seems from afar that public common
> setters sound like total bullshit with those immers that are
> immutable after construction. Come again and tell what you really
> need.
>
> > ... and can't change my containers.
>
> That is defect of mind. Software is *soft*. It is like drawing on
> sand not carving on obsidian.

Of course, but it takes time to make bigger changes.

>
> Most projects usually combine data initially from rather few container
> types like:
> std::array, std::vector, std::unordered_set, std::unordered_map.
> Then, later, when the needs become more clear those are maintained
> to be more special, precisely suitable containers.

There are big advantages to be had with using those Immer containers which I would not want to abandon.

cheers

[David Brown]

On 04/12/2019 17:37, wthol...@gmail.com wrote:
> I'm using std::variant in my code for one very important type
> (`Node`). This variant has 68 different types in it. There are two
> issues:
>
> 1. Compile times are extremely slow (~30s per file), due to variant
> visitation. I'm an indie developer and I don't want to have to buy a
> new computer (might only be a factor of two faster anyway).
>

There are many possibilities for speeding up compilation. But we'd need
a lot more information to be able to tell. I can only give some vague
suggestions (in no particular order), without knowing which might apply.

1. If your computer is very slow, get a better one - or upgrade the one
you have (more memory is often helpful).

2. Consider changing compilers if the compiler is slow.

3. If you are working with lots of files in the project, make sure you
are using a good build system with parallel builds.

4. If you have everything in a few huge files, split them up (and use a
parallel build system).

5. If you are using Windows and have parallel builds and lots of files,
especially with gcc or clang, try Linux. If you are stuck with Windows,
try with an SSD instead of a hard disk.

6. Look at pre-compiled header support.

7. Check the compiler flags. You'll want some optimisation, but be
careful about using some of the more powerful ones (especially ones that
are not part of any -O option), as they can involve a great deal of
compiler effort. But if you have no optimisation, the compiler will
have to generate huge amounts of code, which can be slower than optimising.

8. Consider something other than variant. A variant needs quite a bit
of compile-time effort to get its static type safety and convenience,
and a 68 type variant is going to involve a very large amount of
compile-time effort.

[Alf P. Steinbach]

I, on the other hand, am confused.

I remember reading the original blog posting (on a blog by some
"experts", I think Dave Abrahams was there) about functionality like
`overloaded`, to make a set of lambdas available via a single function
that dispatches to them as if with overload resolution.

But now I fail to see, I don't remember, quite how it worked.

As I see it the OP shouldn't assume that readers would be familiar with
`overloaded`.

When I can't imagine it and only vaguely remember reading about it, then
I think most regulars here know just as little about it as I. And
delving into links to third party web pages. Well that's just not done. :)


[snip]

- Alf

[Öö Tiib]

That is exactly what I meant.
Compiler generates dynamic switch over 68 cases, each case block
made with SFINAE for type handled in that block. Basically just
odd syntax sugar to hide long switch-case. You yourself said you
have very numerous of those. Therefore certain combo of classical
dynamic polymorphism and classical visitor pattern will win the
huge variant of everything (my modules handle rarely over 50 classes)
in all areas easily and without much doubt.

>
> >
> > > > > Now that's pretty cool pattern matching, but let's face it, this would be much more understandable:
> > > > >
> > > > > Node setExposedPosition(Node node, Vec2 p) {
> > > > > if(hasField(node, EXPOSED_POSITION_BIT)) {
> > > > > node.exposedPosition = p;
> > > > > }
> > > > > return node;
> > > > > }
> > > >
> > > > External setter? I do not have setter members no idea why I
> > > > would want external setters.
> > > >
> > > > >
> > > > > I've found that if I simply make Node a struct with every possible data member from all the variant types, it only increases the size of Node by a factor of two, which is totally fine. (I could use std::optional for each member, but that would increase the size, and potentially be more dynamic than I need)
> > > > >
> > > > > Does that sound like the approach you would take?
> > > >
> > > > No. Only by factor of two? Then those are rather similar classes.
> > > >
> > > > From 68 mostly similar to each other "node" classes I would make plain
> > > > old polymorphic hierarchy.
> > >
> > > I used to do a traditional polymorphic hierarchy: it's harder to make value types, and requires much more boilerplate.
> >
> > Value types in sense that little, immutable objects, passed by value?
> > These do not make sense in context of your post at all.
>
> Not necessarily immutable, but yes passed by value. I'm using value-oriented design.

Yeah I noticed that you both pass and return that variant as value
in your examples. Are you sure that "value-oriented design" means
that "always make copies when possible"?

> > On the contrary, it is boilerplate of piles of those inefficient
> > switches from 68 cases all around your code that traditional
> > polymorphic hierarchy would remove with just one cheap level of
> > indirection added. I have measured it enough to predict that
> > program would be at least 4 times quicker, data about 4 times
> > smaller and executable also about twice smaller.
>
> This is not runtime performance critical code. (That's all on another thread)

It was you who said that people "might be baffled by the c++ fancyness"
of your code. There are no performance, no robustness, no clarity,
no readability ... what *is* the point?

> >
> > > > If there is need to keep objects of these classes in a single
> > > > container then I would use Boost.PolyCollection for efficiency.
> > > > https://www.boost.org/doc/libs/1_71_0/doc/html/poly_collection.html
> > > >
> > > > That design beats both your variant and your monster struct blindfolded
> > > > with right hand tied behind back I can bet.
> > >
> > > I'm using https://github.com/arximboldi/immer ...
> >
> > I do not know the lib but it is seems from afar that public common
> > setters sound like total bullshit with those immers that are
> > immutable after construction. Come again and tell what you really
> > need.
> >
> > > ... and can't change my containers.
> >
> > That is defect of mind. Software is *soft*. It is like drawing on
> > sand not carving on obsidian.
>
> Of course, but it takes time to make bigger changes.

True. That is why I keep my stuff very modular. Bit more work
with making and keeping the architecture clean but lot easier to
replace the parts with better or even to have (optional) alternatives
side-by-side.

> >
> > Most projects usually combine data initially from rather few container
> > types like:
> > std::array, std::vector, std::unordered_set, std::unordered_map.
> > Then, later, when the needs become more clear those are maintained
> > to be more special, precisely suitable containers.
>
> There are big advantages to be had with using those Immer containers which I would not want to abandon.

Huh? When you are collector of worthless recommendations then ask for
advice but keep the situation that you are in and constraints that you
have ... in secrecy. Instead make confusing claims with strange
examples. ;) That immer seemed to be all for performance in concurrent
processing. But just plain old "favor immutability" does not need
special libraries for it so it did not make me to feel interested.

[Taylor]

On Wednesday, December 4, 2019 at 1:39:20 PM UTC-8, Alf P. Steinbach wrote:
> I, on the other hand, am confused.
>
> I remember reading the original blog posting (on a blog by some
> "experts", I think Dave Abrahams was there) about functionality like
> `overloaded`, to make a set of lambdas available via a single function
> that dispatches to them as if with overload resolution.
>
> But now I fail to see, I don't remember, quite how it worked.
>
> As I see it the OP shouldn't assume that readers would be familiar with
> `overloaded`.

This is my point (2): that it's fancy and new and people might be confused by it. That's one reason I'm looking for other options :)

[Taylor]

It doesn't.

>
> > > On the contrary, it is boilerplate of piles of those inefficient
> > > switches from 68 cases all around your code that traditional
> > > polymorphic hierarchy would remove with just one cheap level of
> > > indirection added. I have measured it enough to predict that
> > > program would be at least 4 times quicker, data about 4 times
> > > smaller and executable also about twice smaller.
> >
> > This is not runtime performance critical code. (That's all on another thread)
>
> It was you who said that people "might be baffled by the c++ fancyness"
> of your code. There are no performance, no robustness, no clarity,
> no readability ... what *is* the point?

Well, I don't claim to write perfect code :)

The point with that example was to be able to express things concisely. Performance doesn't matter that much (within reason of course).

I actually think it's fairly readable, but still a bit confusing for folks who aren't really up to date on "modern" c++.

I'm not trying to be secretive. The immer containers offer memory efficient immutability, and other advantages. The author of Immer has some great talks which I would recommend watching:

https://youtu.be/_oBx_NbLghY
https://youtu.be/sPhpelUfu8Q

cheers

[Alf P. Steinbach]

I wouldn't describe myself as “new”.

I see that the `overloaded` class is just a few lines of C++17 code and
is exposed on the cppreference.com page about `std::visit`.

You could/should simply have added a comment “For `overloaded`, see
<url: https://en.cppreference.com/w/cpp/utility/variant/visit#Example>”.

Re the problem you're asking about, it seems clear that there is some
commonality of the classes that has not been factored out / abstracted,
but that the rest of the code relies on.

By stuffing everything into variants you're throwing away critical
static type information, replacing it with dynamic type information and
checking. A solution should involve not throwing away the static type
information, or at least throwing away less of it. And voilá.


- Alf

[Öö Tiib]

On Thursday, 5 December 2019 02:28:14 UTC+2, Taylor wrote:
>
> I'm not trying to be secretive. The immer containers offer memory efficient immutability, and other advantages. The author of Immer has some great talks which I would recommend watching:
>
> https://youtu.be/_oBx_NbLghY
> https://youtu.be/sPhpelUfu8Q

Each tool has its use but hope of silver bullets makes you to sell snake
oil to yourself. You wrote "it only increases the size of Node by a

factor of two, which is totally fine".

You seemingly hope that immer wins the consequences: twice more storage
used per object and twice more cache misses somehow back? Again, there
can't be any magic, sorry.

That is why Boost.PolyCollection wins:
1) We get twice the memory efficiency compared with your original variant
and quadruple memory efficiency compared with your "factor of two" monster.
2) We get readability and maintainability of classical dynamic
polymorphism compared with tons of half-hidden switch-cases.
3) We get meaningful layout of data not some kind of flags that
tell what members do make sense and what members don't.
4) We get code-base with what work is possible to divide between contractors.
5) There are nothing bleeding edge fancy about all of it.

[Taylor]

PolyCollection is cool and I appreciate the suggestion. Unfortunately, it alone doesn't have the structural sharing I need in order to implement undo/redo efficiently. I would probably have to use a memento pattern instead, which is error prone.

[Vir Campestris]

On 04/12/2019 21:34, David Brown wrote:
> 5. If you are using Windows and have parallel builds and lots of files,
> especially with gcc or clang, try Linux.  If you are stuck with Windows,
> try with an SSD instead of a hard disk.

I'd try the SSD before I went to the pain of switching OS.

(I still haven't found a Linux IDE as good as visual studio)

Andy

[Öö Tiib]

I do not use Memento pattern ever. Memento had on 95% of cases
when I met it started as naive, locally invented square wheel. Time
had turned it into annoying spaghetti that just complicated everything.
It was often misused for other somewhat similar purposes (like
cut/copy/paste or drag and drop). As result over half of the issues
in these projects were directly caused by that dreaded Memento.

Why not to use serialization capabilities for most of such purposes?
Everything worth undoing or rolling back wants soon to become
something that is worth storing persistently and even worth
transferring to far-away sites. There are literal piles of serialization
libraries available (like Protobufs, Flatbuffers, Cereal, Cap'n Proto,
Bond, Apache Avro, Thrift, MessagePack and just name it). The
popular ones are done by good programmers, tested by lot of
people and so tend to be adequate and far from naive. Pick one
that is language- and platform-neutral ... that provides also wider
opportunities of automatic black-box testing of your modules. ;)

[Taylor]

On Thursday, December 5, 2019 at 8:17:37 AM UTC-8, Öö Tiib wrote:
> Why not to use serialization capabilities for most of such purposes?
> Everything worth undoing or rolling back wants soon to become
> something that is worth storing persistently and even worth
> transferring to far-away sites. There are literal piles of serialization
> libraries available (like Protobufs, Flatbuffers, Cereal, Cap'n Proto,
> Bond, Apache Avro, Thrift, MessagePack and just name it). The
> popular ones are done by good programmers, tested by lot of
> people and so tend to be adequate and far from naive. Pick one
> that is language- and platform-neutral ... that provides also wider
> opportunities of automatic black-box testing of your modules. ;)

I use Flatbuffers for serialization. It's pretty good. But it doesn't easily provide the structural sharing that I need to make undo/redo memory-efficient. For example, if my document contains large data (images, audio, etc.) I don't want to have to save that out after every user interaction. (I'm simplifying a bit: Flatbuffers is a DAG so it can express structural sharing, however I think it would be tricky to implement an efficient undo system with it.)

I have another app which doesn't use structural sharing, and I had to lower the max undos, after some users complained (to put it mildly) about storage usage.

[Taylor]

On Wednesday, December 4, 2019 at 8:13:24 PM UTC-8, Alf P. Steinbach wrote:
> On 05.12.2019 01:18, Taylor wrote:
> > On Wednesday, December 4, 2019 at 1:39:20 PM UTC-8, Alf P. Steinbach wrote:
> >> I, on the other hand, am confused.
> >>
> >> I remember reading the original blog posting (on a blog by some
> >> "experts", I think Dave Abrahams was there) about functionality like
> >> `overloaded`, to make a set of lambdas available via a single function
> >> that dispatches to them as if with overload resolution.
> >>
> >> But now I fail to see, I don't remember, quite how it worked.
> >>
> >> As I see it the OP shouldn't assume that readers would be familiar with
> >> `overloaded`.
> >
> > This is my point (2): that it's fancy and new and people might be confused by it. That's one reason I'm looking for other options :)
>
>
> I wouldn't describe myself as “new”.
>
> I see that the `overloaded` class is just a few lines of C++17 code and
> is exposed on the cppreference.com page about `std::visit`.
>
> You could/should simply have added a comment “For `overloaded`, see
> <url: https://en.cppreference.com/w/cpp/utility/variant/visit#Example>”.
>
> Re the problem you're asking about, it seems clear that there is some
> commonality of the classes that has not been factored out / abstracted,
> but that the rest of the code relies on.

If I were to use inheritance to factor out commonality, I would need to use multiple inheritance and it could get messy. So for example, there's the base class Node, then there might be ExposedNode, and ColorfulNode, LabeledNode.

Or I could use interfaces, such has HasExposedPosition, HasColor, HasLabel, etc. I did this before and it's much more code than the variant stuff.

(And of course using virtual functions complicates the use of value types, since you need to avoid slicing.)

The SFINAE thing (setExposedPosition) does this all in under 10 lines of code, and automatically works for any new type that has an exposedPosition attribute. It's pretty cool in that way.

>
> By stuffing everything into variants you're throwing away critical
> static type information, replacing it with dynamic type information and
> checking. A solution should involve not throwing away the static type
> information, or at least throwing away less of it. And voilá.

I'm not sure what information I'm throwing away there. Could you elaborate?

thanks!

[Taylor]

FWIW, my 2013 laptop already has a SSD.

Thing is, this computer is fast enough for almost everything I do. So it would be good to save the money.

[Öö Tiib]

On Thursday, 5 December 2019 19:44:54 UTC+2, Taylor wrote:
> On Thursday, December 5, 2019 at 8:17:37 AM UTC-8, Öö Tiib wrote:
> > Why not to use serialization capabilities for most of such purposes?
> > Everything worth undoing or rolling back wants soon to become
> > something that is worth storing persistently and even worth
> > transferring to far-away sites. There are literal piles of serialization
> > libraries available (like Protobufs, Flatbuffers, Cereal, Cap'n Proto,
> > Bond, Apache Avro, Thrift, MessagePack and just name it). The
> > popular ones are done by good programmers, tested by lot of
> > people and so tend to be adequate and far from naive. Pick one
> > that is language- and platform-neutral ... that provides also wider
> > opportunities of automatic black-box testing of your modules. ;)
>
> I use Flatbuffers for serialization. It's pretty good. But it doesn't easily provide the structural sharing that I need to make undo/redo memory-efficient. For example, if my document contains large data (images, audio, etc.) I don't want to have to save that out after every user interaction. (I'm simplifying a bit: Flatbuffers is a DAG so it can express structural sharing, however I think it would be tricky to implement an efficient undo system with it.)

Sounds again like some kind of straw-man argument ... who would serialize
whole document to store erasure or change of one image? Only the state and
location of image before change needs to be serialized for to be capable
of restoring it.

> I have another app which doesn't use structural sharing, and I had to lower the max undos, after some users complained (to put it mildly) about storage usage.

No wonder. Look back at your OP. Should you replace unions of 68 classes
with one class with all possible members in those classes? Uhh. I thought
anyone wielding wasteful stuff like C# or Java half-decently or horrible
stuff like Python or PHP expertly will beat C++ of that guy in all of
clarity, maintainability, speed and storage usage.

[Taylor]

On Thursday, December 5, 2019 at 11:14:39 AM UTC-8, Öö Tiib wrote:
>
> Sounds again like some kind of straw-man argument ... who would serialize
> whole document to store erasure or change of one image? Only the state and
> location of image before change needs to be serialized for to be capable
> of restoring it.

Sure, I could serialize diffs, but with significant additional complexity (it's a bit close to the memento pattern). If you know of a way that flatbuffers makes that easier, that would be good to know. I couldn't find anything searching around.

[David Brown]

On 05/12/2019 16:07, Vir Campestris wrote:
> On 04/12/2019 21:34, David Brown wrote:
>> 5. If you are using Windows and have parallel builds and lots of
>> files, especially with gcc or clang, try Linux.  If you are stuck with
>> Windows, try with an SSD instead of a hard disk.
>
> I'd try the SSD before I went to the pain of switching OS.
>

People vary as to what they consider a "pain" regarding an OS. Much has
to do with familiarity, both of the OS and the tools you have on it. I
use both Windows and Linux, and have no doubts to which I find most
efficient for development work. But I fully appreciate that others have
different preferences, experiences and requirements.

As for switching to an SSD, that will require less change in habits or
tools, but it is not always an easy job with an existing system. If you
are using Linux, I wouldn't bother with the SSD if your machine has
plenty of ram (and upgrading ram is usually easy), but it makes a bigger
difference on Windows.

> (I still haven't found a Linux IDE as good as visual studio)
>

I've never been an MSVS fan - again, it's subjective. But I believe MS
Visual Studio Code has a good reputation on Linux.

[Öö Tiib]

On Thursday, 5 December 2019 22:37:20 UTC+2, Taylor wrote:
> On Thursday, December 5, 2019 at 11:14:39 AM UTC-8, Öö Tiib wrote:
> >
> > Sounds again like some kind of straw-man argument ... who would serialize
> > whole document to store erasure or change of one image? Only the state and
> > location of image before change needs to be serialized for to be capable
> > of restoring it.
>
> Sure, I could serialize diffs, but with significant additional complexity (it's a bit close to the memento pattern).

What diffs? Chain of reversible operations.
First operation: Erase: that picture from there.
Next operation Undo: put that picture back there.
Next operation Redo: erase that picture again from there.
What have diffs to do with Memento pattern? Utter nonsense.
Read at least Wikipedia article about it.

If you know of a way that flatbuffers makes that easier, that would be good to know. I couldn't find anything searching around.

Flatbuffers? Huh? But it generates all code for you? All for
needed for storing your data into buffer or reading it back from
it. Bit verbose interface for my taste but clear enough. Have you
at least tried the tutorials of it or something?
Very confusing.

[Taylor]

On Thursday, December 5, 2019 at 1:20:11 PM UTC-8, Öö Tiib wrote:
> On Thursday, 5 December 2019 22:37:20 UTC+2, Taylor wrote:
> > On Thursday, December 5, 2019 at 11:14:39 AM UTC-8, Öö Tiib wrote:
> > >
> > > Sounds again like some kind of straw-man argument ... who would serialize
> > > whole document to store erasure or change of one image? Only the state and
> > > location of image before change needs to be serialized for to be capable
> > > of restoring it.
> >
> > Sure, I could serialize diffs, but with significant additional complexity (it's a bit close to the memento pattern).
>
> What diffs? Chain of reversible operations.
> First operation: Erase: that picture from there.
> Next operation Undo: put that picture back there.
> Next operation Redo: erase that picture again from there.
> What have diffs to do with Memento pattern? Utter nonsense.
> Read at least Wikipedia article about it.

The chain of reversible operations (changes) you describe is error prone and requires additional complexity.

As an example, in Photoshop, Adobe has abandoned the chain reversible operations in favor of value oriented design.

I'd recommend watching Sean Parent's talks such as this one: https://www.youtube.com/watch?v=QGcVXgEVMJg.

>
> If you know of a way that flatbuffers makes that easier, that would be good to know. I couldn't find anything searching around.
>
> Flatbuffers? Huh? But it generates all code for you? All for
> needed for storing your data into buffer or reading it back from
> it. Bit verbose interface for my taste but clear enough. Have you
> at least tried the tutorials of it or something?
> Very confusing.

I've used flatbuffers extensively at this point.

[Öö Tiib]

On Thursday, 5 December 2019 23:42:30 UTC+2, Taylor wrote:
> On Thursday, December 5, 2019 at 1:20:11 PM UTC-8, Öö Tiib wrote:
> > On Thursday, 5 December 2019 22:37:20 UTC+2, Taylor wrote:
> > > On Thursday, December 5, 2019 at 11:14:39 AM UTC-8, Öö Tiib wrote:
> > > >
> > > > Sounds again like some kind of straw-man argument ... who would serialize
> > > > whole document to store erasure or change of one image? Only the state and
> > > > location of image before change needs to be serialized for to be capable
> > > > of restoring it.
> > >
> > > Sure, I could serialize diffs, but with significant additional complexity (it's a bit close to the memento pattern).
> >
> > What diffs? Chain of reversible operations.
> > First operation: Erase: that picture from there.
> > Next operation Undo: put that picture back there.
> > Next operation Redo: erase that picture again from there.
> > What have diffs to do with Memento pattern? Utter nonsense.
> > Read at least Wikipedia article about it.
>
> The chain of reversible operations (changes) you describe is error prone and requires additional complexity.

So go ahead and store state of your whole document after each click.
Or maybe even commit it to repo ... how should I care?

> As an example, in Photoshop, Adobe has abandoned the chain reversible operations in favor of value oriented design.

Who is there in Adobe? I know no one from it. Perhaps all the decent
engineers who made it what it is have been eaten out. If whatever
is left of it decides to rewrite their Acrobat in F# then should
we follow?

> I'd recommend watching Sean Parent's talks such as this one: https://www.youtube.com/watch?v=QGcVXgEVMJg.

Scrolled a bit into that video. First he pushed
rawly newed pointer to document then said that it is wrong and used
the most dork tool std::make_shared! I stopped right there. Had it
been std::make_unique I would had guessed that poor kid hasn't tried
polycollections like you. But snake oil salesmen start from such
clear idiocies for to ensure that by minute 7 everybody with at least
half a brain are left away and only total morons remain.

Do not post anymore YouTube crap. Watching such TV and movies is
perhaps good entertainment for retarded and illiterate.

> > If you know of a way that flatbuffers makes that easier, that would be good to know. I couldn't find anything searching around.
> >
> > Flatbuffers? Huh? But it generates all code for you? All for
> > needed for storing your data into buffer or reading it back from
> > it. Bit verbose interface for my taste but clear enough. Have you
> > at least tried the tutorials of it or something?
> > Very confusing.
>
> I've used flatbuffers extensively at this point.

Sorry, it does not make any sense what is supposed to be the
issue with it. Decent serialization library with bit more
verbose interface than I would use.

[Christian Gollwitzer]

Am 05.12.19 um 18:59 schrieb Taylor:

> On Wednesday, December 4, 2019 at 8:13:24 PM UTC-8, Alf P. Steinbach wrote:
>> On 05.12.2019 01:18, Taylor wrote:
>> Re the problem you're asking about, it seems clear that there is some
>> commonality of the classes that has not been factored out / abstracted,
>> but that the rest of the code relies on.
>
> If I were to use inheritance to factor out commonality, I would need to use multiple inheritance and it could get messy. So for example, there's the base class Node, then there might be ExposedNode, and ColorfulNode, LabeledNode.
>
> Or I could use interfaces, such has HasExposedPosition, HasColor, HasLabel, etc. I did this before and it's much more code than the variant stuff.
>

I'm far from understanding the problem, but factoring out common members
does not mean you nead "class A: public B" inheritance. You could also
have a "struct Node" and then stuff the things that vary into the
variant, which is a member of "Node". This is done in C with unions very
often e.g. for byte code interpreters, where all the values have a type
tag, a reference count, an alternative representation, and then a union
which contains the real data.

Christian

[Christian Gollwitzer]

Am 05.12.19 um 18:44 schrieb Taylor:

> On Thursday, December 5, 2019 at 8:17:37 AM UTC-8, Öö Tiib wrote:
>> Why not to use serialization capabilities for most of such purposes?
>> Everything worth undoing or rolling back wants soon to become
>> something that is worth storing persistently and even worth
>> transferring to far-away sites. There are literal piles of serialization
>> libraries available (like Protobufs, Flatbuffers, Cereal, Cap'n Proto,
>> Bond, Apache Avro, Thrift, MessagePack and just name it). The
>> popular ones are done by good programmers, tested by lot of
>> people and so tend to be adequate and far from naive. Pick one
>> that is language- and platform-neutral ... that provides also wider
>> opportunities of automatic black-box testing of your modules. ;)
>
> I use Flatbuffers for serialization. It's pretty good. But it doesn't easily provide the structural sharing that I need to make undo/redo memory-efficient. For example, if my document contains large data (images, audio, etc.) I don't want to have to save that out after every user interaction.

We used such a scheme for undo/redo in a project (written in another
language, not C++). The document there consists of 3D meshes composed
into a 3D scene and attached properties like material, transformation,
... So the actual data was quite small (few kb for a realistic document,
even if stored in text format) compared to multi-MB of 3D mesh data.
Even before we introduced undo/redo, the serialization format did not
include the 3D meshes. There is a global storage of the big objects, a
hash map indexed by the MD5 of the binary mesh data, and the document
simply references these. On I/O, only the transformation data is
serialized into a compact format and a second pass reads/writes the
binary object table. The on-disc format in the end was a ZIP file, this
is similar to modern text writer formats. Just rename a docx-file to
.zip and open it up, then you'll see it uses the same scheme.

Undo/redo with this scheme was instant, almost no memory consumption, so
we could provide unlimited undo.

Christian

[Öö Tiib]

On Friday, 6 December 2019 10:58:47 UTC+2, Christian Gollwitzer wrote:
> The on-disc format in the end was a ZIP file, this
> is similar to modern text writer formats. Just rename a docx-file to
> .zip and open it up, then you'll see it uses the same scheme.

Yes, lot of projects use that format: folders of utf8 text and
portable binary files zipped, just that json instead of xml is
more usual.

[Taylor]

Thanks Christian. That would be an improvement and cut down on the variant visitation (it would help compilation time on one file in particular). I don't think it would go far enough though in improving my compilation time.

[Taylor]

On Thursday, December 5, 2019 at 3:41:38 PM UTC-8, Öö Tiib wrote:
>
> Scrolled a bit into that video. First he pushed
> rawly newed pointer to document then said that it is wrong and used
> the most dork tool std::make_shared! I stopped right there.

You should continue watching. He's just using shared_ptr in his example. A production implementation would use a small-object optimization. In fact, central to his type erasure scheme is reducing the amount of dynamic allocation.

[Öö Tiib]

On Friday, 6 December 2019 19:44:14 UTC+2, Taylor wrote:
> On Thursday, December 5, 2019 at 3:41:38 PM UTC-8, Öö Tiib wrote:
> >
> > Scrolled a bit into that video. First he pushed
> > rawly newed pointer to document then said that it is wrong and used
> > the most dork tool std::make_shared! I stopped right there.
>
> You should continue watching. He's just using shared_ptr in his example.

Huh? Each shared_ptr usage is dynamic allocation of sole object plus
complicating your data from tree to directed acyclic or may be even
introducing directed cycles into your data. Additionally it is rather
expensive (in both performance and storage usage) smart pointer. It
has usages but rare or else you turn your program into as error prone
and inefficient mess as typical Java or C# programs are.
So video where you are casting it into code without apparently no
thought applied is not worth watching. It proves that you are either
snake oil salesman or noob.

> A production implementation would use a small-object optimization. In fact, central to his type erasure scheme is reducing the amount of dynamic allocation.

Blah blah blah blah? If something can't be written as document then
it is *never* worth watching as video either. I have already posted link
to document and open source implementation of one way how to reduce
dynamic allocations:
<https://www.boost.org/doc/libs/1_71_0/doc/html/poly_collection.html>
C++ is powerful so there are several other ways and each can be
used and discussed in separation or in combination.

[Manfred]

On 12/7/2019 1:43 PM, Öö Tiib wrote:
> On Friday, 6 December 2019 19:44:14 UTC+2, Taylor wrote:
>> On Thursday, December 5, 2019 at 3:41:38 PM UTC-8, Öö Tiib wrote:
>>>
>>> Scrolled a bit into that video. First he pushed
>>> rawly newed pointer to document then said that it is wrong and used
>>> the most dork tool std::make_shared! I stopped right there.
>>
>> You should continue watching. He's just using shared_ptr in his example.
>
> Huh? Each shared_ptr usage is dynamic allocation of sole object plus
> complicating your data from tree to directed acyclic or may be even
> introducing directed cycles into your data. Additionally it is rather
> expensive (in both performance and storage usage) smart pointer.

The point about shared_ptr in the video is that the speaker starts
presenting a use of shared_ptr, then he shows how bad it is and replaces
it for a while with unique_ptr. However, in the end he resumes the use
of shared_ptr, but only as pointer to const, as a mean to reduce copies.
So it is less stupid than you posed it.

That said, I don't like shared_ptr either.

It
> has usages but rare or else you turn your program into as error prone
> and inefficient mess as typical Java or C# programs are.

Totally agree on this.

[Öö Tiib]

On Saturday, 7 December 2019 19:17:57 UTC+2, Manfred wrote:
> On 12/7/2019 1:43 PM, Öö Tiib wrote:
> > On Friday, 6 December 2019 19:44:14 UTC+2, Taylor wrote:
> >> On Thursday, December 5, 2019 at 3:41:38 PM UTC-8, Öö Tiib wrote:
> >>>
> >>> Scrolled a bit into that video. First he pushed
> >>> rawly newed pointer to document then said that it is wrong and used
> >>> the most dork tool std::make_shared! I stopped right there.
> >>
> >> You should continue watching. He's just using shared_ptr in his example.
> >
> > Huh? Each shared_ptr usage is dynamic allocation of sole object plus
> > complicating your data from tree to directed acyclic or may be even
> > introducing directed cycles into your data. Additionally it is rather
> > expensive (in both performance and storage usage) smart pointer.
> The point about shared_ptr in the video is that the speaker starts
> presenting a use of shared_ptr, then he shows how bad it is and replaces
> it for a while with unique_ptr.

Does not matter. Usage of shared_ptr is often outright defect.
Usage of lot of single object allocations/deallocations is
not defect (but usually weak design).

> However, in the end he resumes the use
> of shared_ptr, but only as pointer to const, as a mean to reduce copies.
> So it is less stupid than you posed it.

Ok, I agree, not so utterly stupid. Making the shared objects immutable
makes it at least valid to process our now directed acyclic data in
embarrassingly parallel manner. So we at least can burden all the
cores (or even GPUs in our typical target architecture) next to
effortlessly when needed.

But the prices of dynamic memory management per object, overhead
from expensive pointer, incapability to realize that equivalent
immutable object that we are creating now is already shared
elsewhere and additional trouble during serialization still remain.
I by default resolve those proactively by having special (smart)
factory and special (smart) pointers but shared_ptr can also be
good enough when used like that.

However when data logically really has directed cycles in it (that
happens ultra rarely) then I use likes of Parallel BGL.
<https://www.boost.org/doc/libs/1_71_0/libs/graph_parallel/doc/html/index.html>
At least some hope that someone else is testing it too.

> That said, I don't like shared_ptr either.
>
> It
> > has usages but rare or else you turn your program into as error prone
> > and inefficient mess as typical Java or C# programs are.
> Totally agree on this.

So I'm just bit allergic to it. The unreliable stuff that has
resulted from misuse of shared_ptr IMHO keeps damaging reputation
of C++. Unsure what book taught to put it everywhere ... it should
be burnt.