[boost] Library proposal - indexed_array
Julien Blanc via Boost
I've written and released a (rather small) library which allows
defining arrays with arbitrary index mechanisms (hence the name,
indexed_array). This especially allows indexing with scoped enums, use
non zero-starting indexes, or use sparse indexes. This is not a view,
but an array in the sense that it owns the data, and provides the same
guarantees and functionalities as std::array, such as triviality
propagation, being able to reside in read-only program sections, etc.
It also supports multidimensional indexing, and provides an
indexed_span template class to access lower dimension content (which
can be used outside indexed_array).
Finally, it provides an opt-in safer way to initialize the array
content, which is especially useful when using complex and/or changing
indexing schemes (like based on externally generated files).
The library is available here:
https://github.com/Julien-Blanc-tgcm/indexed_array
And the doc is here:
https://julien-blanc-tgcm.github.io/indexed_array/
It requires C++17 and mp11, and has first-class support for describe-d
enums.
If there is some interest, i'd like to propose it for inclusion in
boost. So i'm looking for people who would endorse it.
Thanks and best regards,
Julien
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Gavin Lambert via Boost
> I've written and released a (rather small) library which allows
> defining arrays with arbitrary index mechanisms (hence the name,
> indexed_array). This especially allows indexing with scoped enums, use
> non zero-starting indexes, or use sparse indexes. This is not a view,
> but an array in the sense that it owns the data, and provides the same
> guarantees and functionalities as std::array, such as triviality
> propagation, being able to reside in read-only program sections, etc.
How does it compare with Boost.MultiIndex?
Julien Blanc via Boost
> On 2/06/2022 17:57, Julien Blanc wrote:
>> I've written and released a (rather small) library which allows
>> defining arrays with arbitrary index mechanisms (hence the name,
>> indexed_array). This especially allows indexing with scoped enums, use
>> non zero-starting indexes, or use sparse indexes. This is not a view,
>> but an array in the sense that it owns the data, and provides the same
>> guarantees and functionalities as std::array, such as triviality
>> propagation, being able to reside in read-only program sections, etc.
>
> How does it compare with Boost.MultiIndex?
I need to fill the feature comparison table, thanks for suggesting
including MultiIndex. Unless i'm mistaken, multi-index is intended for
dynamic-sized containers, and for providing different ways of accessing
the same set of elements. It is never trivial, and cannot reside in a
read-only program section (this is an important use case for me,
targetting MCUs running program from flash). Its goal is to provide fast
access via different ways (i see it as an equivalent of database
indexes).
On the other side, indexed_array provides a single indexing mechanism,
guarantees triviality propagation and contiguity of elements.
So, i believe they address different use cases.
Regards,
Julien
Joaquin M López Muñoz via Boost
> Le 2022-06-02 08:06, Gavin Lambert via Boost a écrit :
>> On 2/06/2022 17:57, Julien Blanc wrote:
>>> I've written and released a (rather small) library which allows
>>> defining arrays with arbitrary index mechanisms (hence the name,
>>> indexed_array). This especially allows indexing with scoped enums, use
>>> non zero-starting indexes, or use sparse indexes. This is not a view,
>>> but an array in the sense that it owns the data, and provides the same
>>> guarantees and functionalities as std::array, such as triviality
>>> propagation, being able to reside in read-only program sections, etc.
>>
>> How does it compare with Boost.MultiIndex?
>
>
> So, i believe they address different use cases.
Hi Julien,
I've taken a look at your docs and, yes, your proposal covers a totally
different space
than Boost.MultiIndex.
Best,
Joaquín M López Muñoz
Dominique Devienne via Boost
<bo...@lists.boost.org> wrote:
> Le 2022-06-02 08:06, Gavin Lambert via Boost a écrit :
> > How does it compare with Boost.MultiIndex?
>
> I need to fill the feature comparison table
Can you also compare it with https://github.com/serge-sans-paille/frozen ?
More use-case examples would be nice too. For when it's applicable and
useful. --DD
Maximilian Riemensberger via Boost
On 6/2/22 07:57, Julien Blanc via Boost wrote:
> Hi,
>
> I've written and released a (rather small) library which allows
> defining arrays with arbitrary index mechanisms (hence the name,
> indexed_array). This especially allows indexing with scoped enums, use
> non zero-starting indexes, or use sparse indexes. This is not a view,
> but an array in the sense that it owns the data, and provides the same
> guarantees and functionalities as std::array, such as triviality
> propagation, being able to reside in read-only program sections, etc.
>
> It also supports multidimensional indexing, and provides an
> indexed_span template class to access lower dimension content (which
> can be used outside indexed_array).
>
> Finally, it provides an opt-in safer way to initialize the array
> content, which is especially useful when using complex and/or changing
> indexing schemes (like based on externally generated files).
The library looks very interesting. I think I could be very useful for defining custom error codes (enums) and their mapping to custom error strings. This is relatively simple with a std::array if the codes are contiguous and starting from 0. Your library would make it as easy for arbitrary code values, like http codes. Very useful.
>
> The library is available here:
>
> https://github.com/Julien-Blanc-tgcm/indexed_array
>
> And the doc is here:
>
> https://julien-blanc-tgcm.github.io/indexed_array/
>
> It requires C++17 and mp11, and has first-class support for describe-d
> enums.
Is there a particular reason to require c++17? I guess some std::array features that got added? I would assume that it should be possible to make it compile under C++14 (maybe using some boost array) since most constexpr things should already be available there.
>
> If there is some interest, i'd like to propose it for inclusion in
> boost. So i'm looking for people who would endorse it.
I endorse it.
Thanks for sharing.
Best,
Max
Julien Blanc via Boost
> On Thu, Jun 2, 2022 at 8:59 AM Julien Blanc via Boost
> <bo...@lists.boost.org> wrote:
> > Le 2022-06-02 08:06, Gavin Lambert via Boost a écrit :
> > > How does it compare with Boost.MultiIndex?
> >
> > I need to fill the feature comparison table
>
> Can you also compare it with
> https://github.com/serge-sans-paille/frozen ?
I was not aware of the existence of this library. After a quick look,
frozen::unordered_map has some similarities in what it offers. The main
difference that comes to mind is that it is designed as a map, which
means it stores both the key and the value. This has some consequences
in the size of the data. Thanks for pointing this library, I'll include
it in the feature comparisons.
> More use-case examples would be nice too. For when it's applicable
> and useful. --DD
Max gave a very good one with http error codes. I'll include some more.
Thanks for your feedback,
Julien
Julien Blanc via Boost
>
> On 6/2/22 07:57, Julien Blanc via Boost wrote:
> >
>
>
> The library looks very interesting. I think I could be very useful
> for defining custom error codes (enums) and their mapping to custom
> error strings. This is relatively simple with a std::array if the
> codes are contiguous and starting from 0. Your library would make it
> as easy for arbitrary code values, like http codes. Very useful.
Thanks for your feedback. This is indeed a very good use case.
>
> Is there a particular reason to require c++17? I guess some
> std::array features that got added? I would assume that it should be
> possible to make it compile under C++14 (maybe using some boost
> array) since most constexpr things should already be available there.
The first reason it requires C++17 is because this is what i am
targetting at the moment. However, making it C++14-compliant would
require a few changes:
* the library use template auto syntax in several places, this allows
writing indexed_array<float, interval<1, 10>>. In C++14 one would need
to write indexed_array<float, interval<int, 1, 10>>, or resort to
macros.
* it makes use of std::is_invokable to correctly disable the default
aggregate constructor if given a safe_arg initializer list. This can be
worked around (iirc is_invokable can be emulated in C++14)
* it also makes use of std::apply for multi-dimensional indexing.
Likewise, this can be emulated in C++14.
* there are probably a few others that i'm missing. In short: it can be
done, but sticking to C++17 avoided reimplementing several now-standard
library features.
> >
> > If there is some interest, i'd like to propose it for inclusion in
> > boost. So i'm looking for people who would endorse it.
>
> I endorse it.
>
> Thanks for sharing.
Thanks for your feedback, and thanks for endorsing the library.
Regard,
Maximilian Riemensberger via Boost
> Le jeudi 02 juin 2022 à 11:00 +0200, Maximilian Riemensberger a écrit :
>> On 6/2/22 07:57, Julien Blanc via Boost wrote:
>>
>> Is there a particular reason to require c++17? I guess some
>> std::array features that got added? I would assume that it should be
>> possible to make it compile under C++14 (maybe using some boost
>> array) since most constexpr things should already be available there.
>
> The first reason it requires C++17 is because this is what i am
> targetting at the moment. However, making it C++14-compliant would
> require a few changes:
>
> * the library use template auto syntax in several places, this allows
> writing indexed_array<float, interval<1, 10>>. In C++14 one would need
> to write indexed_array<float, interval<int, 1, 10>>, or resort to
> macros.
> * it makes use of std::is_invokable to correctly disable the default
> aggregate constructor if given a safe_arg initializer list. This can be
> worked around (iirc is_invokable can be emulated in C++14)
> * it also makes use of std::apply for multi-dimensional indexing.
> Likewise, this can be emulated in C++14.
> * there are probably a few others that i'm missing. In short: it can be
> done, but sticking to C++17 avoided reimplementing several now-standard
> library features.
I see. The first one is definitely a language limitation. But I guess it's mostly an inconvenience to the users of the library if they are on C++14. The other ones as far as I can see have counterparts in boost that would work in C++14 as well. So I guess it could be done without too much hassle if there is a need and the library would be included in boost. Either way, it's very useful regardless of the std version details.
Best,
Max
Julien Blanc via Boost
> On 6/2/22 13:30, Julien Blanc wrote:
> >
> >
>
> I see. The first one is definitely a language limitation. But I guess
> it's mostly an inconvenience to the users of the library if they are
> on C++14. The other ones as far as I can see have counterparts in
> boost that would work in C++14 as well. So I guess it could be done
> without too much hassle if there is a need and the library would be
> included in boost. Either way, it's very useful regardless of the
> std version details.
I'm already depending on mp11, so adding a dependency to more boost
libraries (i have been stuck too long with c++11 and old boost
releases, and completely missed that hof provides is_invocable and
apply) is not an issue.
I would need, however, to provide an unified syntax for declaring
intervals for C++14 and C++17, otherwise a compiler update would break
user code. I think this can be done only by using a macro (which would
be mandatory only for C++14 code, and provided in C++17 for backward
compatibility). Anyhow, I created a github issue to track any progress
on this side.
For those interested, i filled up the feature comparison table (
https://julien-blanc-tgcm.github.io/indexed_array/comparison.html ) and
added some use cases in the readme file.
Best regards,
Julien
Richard Hodges via Boost
wrote:
> Le jeudi 02 juin 2022 à 14:00 +0200, Maximilian Riemensberger a écrit :
> > On 6/2/22 13:30, Julien Blanc wrote:
> > >
> > > [...]
> > >
> >
> > I see. The first one is definitely a language limitation. But I guess
> > it's mostly an inconvenience to the users of the library if they are
> > on C++14. The other ones as far as I can see have counterparts in
> > boost that would work in C++14 as well. So I guess it could be done
> > without too much hassle if there is a need and the library would be
> > included in boost. Either way, it's very useful regardless of the
> > std version details.
>
> I'm already depending on mp11, so adding a dependency to more boost
> libraries (i have been stuck too long with c++11 and old boost
> releases, and completely missed that hof provides is_invocable and
> apply) is not an issue.
>
> I would need, however, to provide an unified syntax for declaring
> intervals for C++14 and C++17, otherwise a compiler update would break
> user code. I think this can be done only by using a macro (which would
> be mandatory only for C++14 code, and provided in C++17 for backward
> compatibility). Anyhow, I created a github issue to track any progress
> on this side.
>
> For those interested, i filled up the feature comparison table (
> https://julien-blanc-tgcm.github.io/indexed_array/comparison.html ) and
> added some use cases in the readme file.
>
Thank you this has helped me.
I have one question to add:
Are you able to provide one or two motivating use cases? i.e. scenarios in
which I would strongly prefer to use this library rather than a std::array
and static_cast?
Julien Blanc via Boost
> On Fri, 3 Jun 2022 at 08:31, Julien Blanc via Boost
> <bo...@lists.boost.org>
> wrote:
>
> Are you able to provide one or two motivating use cases? i.e. scenarios
> in
> which I would strongly prefer to use this library rather than a
> std::array
> and static_cast?
You mean, in addition to
https://github.com/julien-Blanc-tgcm/indexed_array#sample-use-cases ?
Another benefit that i did not speak of is that you get type safety for
the index. i.e. you cannot use a wrongly typed index:
indexed_array<char const*, user_messages> messages;
std::array<char const*, nb_user_messages> messages_arr;
// ...
play_message(messages_arr[static_cast<int>(tech_message::goon)]); //
boom at runtime
play_message(messages[tech_message::goon]); // fails to compile
I also find the code clearer (the static_cast just adds noise in my
opinion), but that would be a matter of taste.
Regards,
Hadriel Kaplan via Boost
I know I’m late to this email thread topic, but email’s free so why not…
I saw some questions on if the indexed_array would be useful/used.
I don’t have a crystal ball, but I can tell you we have a similar type in my employer’s codebase, and we use it. Like indexed_array, ours is a std::array, all done at compile time, etc.
Ours is purely for enum-based indexing, and only for scoped enums that start at value 0 and increase sequentially – i.e. contiguous, no “holes”. We happen to be able to verify that property of enums at compile-time, and can static_assert that it’s true, so ours just ends up doing a static_cast under-the-hood. (I can’t quite tell what indexed_array does in the end, since it also deals with non-contiguous index values.)
Like indexed_array, ours also allows the user to initialize entries, which was probably the hardest aspect of writing our own.
Had Boost had the indexed_array when we wrote ours, it is extremely likely we would have used Boost’s instead. (we use Boost a lot)
I’m not sure we would have ever used the non-contiguous aspect. We have plenty of non-contiguous enums, but no one’s ever clamored for array support for them. (of course that could just be a self-fulfilling prophecy, since we don’t already have such an array) It definitely appears to make it more complicated, from my quick perusal of indexed_array on github.
To give you some numbers, we’re what I consider a medium-sized codebase, and there are well over 100 different uses of this array in our codebase. That’s approximately a 15:1 ratio of enum types to our array types – i.e., for every 15 enum types in our codebase, we have one of these array types. It surprises me that it’s so many of these arrays, but that’s what the numbers show. Around 1/3rd of them are purely in test code, the rest production.
-hadriel
Richard Hodges via Boost
> Le 2022-06-03 08:44, Richard Hodges via Boost a écrit :
> > On Fri, 3 Jun 2022 at 08:31, Julien Blanc via Boost
> > <bo...@lists.boost.org>
> > wrote:
> >
> > Are you able to provide one or two motivating use cases? i.e. scenarios
> > in
> > which I would strongly prefer to use this library rather than a
> > std::array
> > and static_cast?
>
> You mean, in addition to
> https://github.com/julien-Blanc-tgcm/indexed_array#sample-use-cases ?
>
>
Yes, I'm looking for a real-world use case. The trivial hypothetical cases
offered are building a static association between an enum value and some
other value.
This can be done with switch/case (for example), in about the same amount
of typing, at compile time, in a function.
For example:
constexpr std::uint32_t
base_address(i2c_id id)
{
switch(id)
{
case i2c_id::i2c1: return 0x1060000u;
case i2c_id::i2c2: return 0x1070000u;
case i2c_id::i2c3: return 0x1080000u;
case i2c_id::i2c4: return 0x1090000u;
}
}
constexpr std::uint32_t
offset(i2c_register r)
{
switch(r)
{
case i2c_register::set_clock: return 0;
case i2c_register::read: return 1;
case i2c_register::write: return 2;
case i2c_register::ack: return 3;
}
}
https://godbolt.org/z/bYYq65hc6
Hadriel Kaplan via Boost
> Yes, I'm looking for a real-world use case. The trivial hypothetical cases
> offered are building a static association between an enum value and some
> other value.
> This can be done with switch/case (for example), in about the same amount
> of typing, at compile time, in a function.
OK, now show me how to iterate over it. And how to get a view or slice of it (i.e., std::span or range). And how to get the number of elements. :)
I mean ultimately, it's a container. It's not _only_ useful for key'ed lookup retrieval.
You can change the value of whatever the container holds too, for example. Or maybe I'm misunderstanding your point? (could easily be the case)
Anyway, the main use-case or value of it, in my mind anyway, is actually the one when using Boost.Describe. Because when you're using Describe for an enum, you only need to do this for indexed_array:
indexed_array<string, MyEnum> data;
That's powerful, because you do not need to repeat any code, do not need to repeat any sizing/number of enums, etc. If you add new enumerators to your enum, you don't need to change this instantiation of indexed_array. It's all automatic, and you can't screw up.
The arbitrary intervals and non-contiguous use-cases I'm less convinced by. It's too easy to mess up.
And arguably it's not really a "indexed_array" at that point anyway - the underlying container might happen to be a std::array, but that's an implementation detail. From the user's perspective it's more like a "static_map" or some such. Except... it's not actually necessarily sorted, is it? All the examples show them sorted, but I'm not sure it really is if the user doesn't define it that way? (I'm trying _not_ to read the code - only the docs)
-hadriel
Richard Hodges via Boost
> On Thursday, June 23, 2022, 05:20:01 PM EDT, Richard Hodges via Boost <
> bo...@lists.boost.org> wrote:
>
> > Yes, I'm looking for a real-world use case. The trivial hypothetical
> cases
> > offered are building a static association between an enum value and some
> > other value.
> > This can be done with switch/case (for example), in about the same amount
> > of typing, at compile time, in a function.
>
> <snipped example switch-case code>
>
> OK, now show me how to iterate over it. And how to get a view or slice of
> it (i.e., std::span or range). And how to get the number of elements. :)
>
is somehow pleasing. The ability might be an interesting academic goal.
I am more interested in what real-world use-cases this solves.
boost.describe, wise_enum etc solve the case where (for example) I wish to
serialise and deserialise my enum value as text. This is useful in the real
world as, for example, it allows me to easily and reliably implement a JSON
communications protocol that contains my enum.
Indexed array still has the maintenance burden of having to maintain both
the enum and the associative "array" separately.
>
> I mean ultimately, it's a container. It's not _only_ useful for key'ed
> lookup retrieval.
>
> You can change the value of whatever the container holds too, for example.
> Or maybe I'm misunderstanding your point? (could easily be the case)
>
> Anyway, the main use-case or value of it, in my mind anyway, is actually
> the one when using Boost.Describe. Because when you're using Describe for
> an enum, you only need to do this for indexed_array:
>
> indexed_array<string, MyEnum> data;
>
> That's powerful, because you do not need to repeat any code, do not need
> to repeat any sizing/number of enums, etc. If you add new enumerators to
> your enum, you don't need to change this instantiation of indexed_array.
> It's all automatic, and you can't screw up.
>
> The arbitrary intervals and non-contiguous use-cases I'm less convinced
> by. It's too easy to mess up.
>
> And arguably it's not really a "indexed_array" at that point anyway - the
> underlying container might happen to be a std::array, but that's an
> implementation detail. From the user's perspective it's more like a
> "static_map" or some such. Except... it's not actually necessarily sorted,
> is it? All the examples show them sorted, but I'm not sure it really is if
> the user doesn't define it that way? (I'm trying _not_ to read the code -
> only the docs)
>
presented as text and each flag may not appear more than once?
enum class NerfEnum : char {
wounded = 'W',
buffed = 'B';
burning = 'N';
};
BOOST_DESCRIBE_ENUM(NerfEnum, (wounded, buffed, burning));
JSON input:
{
...
"nerfs": [wounded, buffed, burning]
...
}
indexed_array might then be used to efficiently store the set of flags
associated with this unfortunate character.
But in this case, am I not better off with an unsigned int, where each flag
is represented by a bit?
The "natural code" might be:
unsigned flags = 0;
for(auto&& value : nerfsjson.as_array())
flags |= to_bit(*deserialise<NerfEnum>(value.as_string());
I could imagine that being able to express this bitset with a
container-like interface might make my code more intuitive, but I don't
think that indexed_array addresses this case.
(I appreciate that such a container would have some of the same criticisms
of vector<bool>)
Julien Blanc via Boost
écrit :
> On Fri, 24 Jun 2022 at 02:01, Hadriel Kaplan <hadr...@yahoo.com>
> wrote:
>
> I understand that presenting the type with the look and feel of a
> container is somehow pleasing. The ability might be an interesting
> academic goal.
>
> I am more interested in what real-world use-cases this solves.
Maybe i should have started with some history for what has lead me to
the design of indexed_array.
I've been professionnally working for the past 4 years (and still am)
in the Lift industry. We make heavy use of CANOpen, and especially it's
lift oriented variant, CANOpenLift (DS417). A lift has floors (at least
2) and door faces (usually one or two), which we need to handle in
several places to store characteristic like the type of door, access
controls, etc.
Due to the way CANOpenLift is designed, doors start at zero and goes up
to 3 (4 doors max), but floors start as one (zero indicates the cabin).
And there's a special value, 0xFF, which means all floors. As a
manufacturer extension, we have also a few reserved values (in the
higher range) which indicates specific locations, such as machinery.
What this means in that in our code base, we have a mix of zero-based
arrays and one-based arrays. This has actually led to some errors when
the highest floor was in use (failure to apply the offset will be
transparent if you do it both when reading and writing unless you go
out of bound). Migrating to an indexed_array-like container (the one we
use in our code base is not the one i'm proposing, but an inferior but
C++11 compatible one) has greatly improved readability and actually
fixed a few bugs.
We also have several places where data is indexed by values coming from
the DS417 specification (eg virtual inputs), and they tend to start at
one, but not always (sometimes the zero value is reserved, sometimes
not). The need to support "holes" in the indexing scheme comes from
there, because you have reserved areas for manufacturer specific
extensions.
The floor / door combination is also what incurs the need to provide a
multidimensional indexing. In our code base, we have a BiIndexedArray,
but i find it more elegant to have it as a generalization of
indexed_array.
>
> Indexed array still has the maintenance burden of having to maintain
> both the enum and the associative "array" separately.
That is true, but that's actually where the safe initialization feature
can help you. By checking at compile time that you handle all cases,
any change in the enum will be a compile-time break. Especially handy
when the enum is generated from some description file (that is the case
in our code base). You can do that with some switch/case if you don't
need a contiguous storage, but if you do then you're out of solution.
Another place where i plan to use indexed_array in our code base is for
state machines (we use them a lot in our code base). The state is
defined by both an enum value (the enum contains the list of all
possible states) and a corresponding class, which does the logic. A
pointer to each state is stored in an array (a map is not usable as we
are target mcus with around 256k ram, so any dynamic memory management
is forbidden). This array is initialized at startup, and we have to
make sure the initialization (which is usually in the source file)
follows the order of the enum, which is usually defined in the header
file. This as well actually led to some bugs (not production bugs,
they're seen in the development phase), a compile time check here is a
nice addition.
>
>
> Perhaps there is a case when seeking to parse a number of flags that
> are presented as text and each flag may not appear more than once?
>
> enum class NerfEnum : char {
> wounded = 'W',
> buffed = 'B';
> burning = 'N';
> };
> BOOST_DESCRIBE_ENUM(NerfEnum, (wounded, buffed, burning));
>
> JSON input:
> {
> ...
> "nerfs": [wounded, buffed, burning]
> ...
> }
>
> indexed_array might then be used to efficiently store the set of
> flags associated with this unfortunate character.
> But in this case, am I not better off with an unsigned int, where
> each flag is represented by a bit?
>
> The "natural code" might be:
>
> unsigned flags = 0;
> for(auto&& value : nerfsjson.as_array())
> flags |= to_bit(*deserialise<NerfEnum>(value.as_string());
>
> I could imagine that being able to express this bitset with a
> container-like interface might make my code more intuitive, but I
> don't think that indexed_array addresses this case.
> (I appreciate that such a container would have some of the same
> criticisms of vector<bool>)
This is indeed not the use cases that drove me to design the container.
I don't intend to provide any vector<bool>-like optimization, i don't
think they make a lot of sense (i'm not a big fan of vector<bool>, i
use bitset which i find a lot more convenient).
There surely is some academicity in the design of indexed_array. Its
inspiration came from real world needs, but i have tried to make it
extensible so that it can address more than just my actual needs.
Describe integration was made both because it really simplifies the
usage, and it was a nice stress-test of the design. The same goes for
multidimensional: it was actually a nice way of checking if the design
was general enough.
Last but not least, i'll just add that Ada (yes, Ada!) has had the
following feature for nearly 40 years:
type Index is range 1 .. 5;
arr: array (Index) of int;
This kind of thing is especially liked by embedded developers. I'm not
aware of any C++ library that provides the same functionality.
indexed_array does:
indexed_array<int, interval<1, 5>> arr;
Sure, you may not have a need for this. I guess it depends a lot on the
domain you are working in. Most of the time, an std::map will provide
the abstraction you need, and the additional cost won't be an issue.
But if you don't have any dynamic memory management, then obviously it
can't. indexed_array has been designed with this in mind (that is a
strong requirement in my daily job).
I hope this long reply explains better the motivations behind the
design.
Best regards,
Julien
Julien Blanc via Boost
Hi,
Thanks for this feedback.
> Ours is purely for enum-based indexing, and only for scoped enums that
> start at value 0 and increase sequentially – i.e. contiguous, no
> “holes”. We happen to be able to verify that property of enums at
> compile-time, and can static_assert that it’s true, so ours just
> ends up doing a static_cast under-the-hood. (I can’t quite tell what
> indexed_array does in the end, since it also deals with non-contiguous
> index values.)
The current strategy is the following:
* detect if the set of indexing values is contiguous (ie, no holes).
Duplicate values are removed (this is needed to support enums with
aliases such as first or last).
* If that's the case, use a static_cast and an offset for index
computation. If the offset happens to be zero, this is fully optimized
by the compiler.
* if that's not the case, the current implementation use an mp_foreach
loop (iterate over all indexes to recompute the index in the underlying
array).
This detection is performed at compile time, and has been firstly
designed to support described-enums, which we get as an integer
list-like. Supporting them without run-time penalty was a design goal.
There is indeed a performance penalty with using non-contiguous enums
with the default indexer. Since several of my use cases for
non-contiguous enums involves mostly either two discontinuous ranges, or
a single range + a few single values, i may improve these specific cases
in the future (this is an implementation detail that will not change the
client-side usage of the library).
Regards,
Julien Blanc via Boost
écrit :
>
> The arbitrary intervals and non-contiguous use-cases I'm less
> convinced by. It's too easy to mess up.
> And arguably it's not really a "indexed_array" at that point
> anyway - the underlying container might happen to be a std::array,
> but that's an implementation detail. From the user's perspective it's
> more like a "static_map" or some such. Except... it's not actually
> necessarily sorted, is it? All the examples show them sorted, but I'm
> not sure it really is if the user doesn't define it that way? (I'm
> trying _not_ to read the code - only the docs)
There's one example of unordered indexes in the doc, in basic usage /
Usage with integer lists (at the end of
https://julien-blanc-tgcm.github.io/indexed_array/basicusage.html ).
I don't have any use case for them. It just happens than once you
handle arbitrary integer list to be able to support holes/non
contiguous, you get unordered for free. But yes, i think it can quickly
get pretty messy.
Best regards,
Julien
Hadriel Kaplan via Boost
Ahh, right you want concrete examples of what one would need/do with this thing.
Because yeah, it’s not to convert to a string name. People already do that, as we all know.
OK, so here are some uses in my employer’s codebase – I’ll just list the first 10 I find, in the order my editor finds them:
1. A container of event-collectors, where the enum key represents event types, and the collectors are objects that stream them to various places or save them to files, etc., per type of event.
2. A container of loggers, where the enum key represents a logging type, and each logger is an object with its own log-level, filenames, etc.
3. A container of subprocess execution commands, where the enum key represents the command type, and the object knows the actual command string, args , etc.
4. A container of hashmaps, that map some VRF prefixes to some internal stuff, and the enum key represents route-advertisement types. (we’re a router company)
5. A container of test-inputs/outputs, where the enum key represents the test-input type, and the test-inputs are various routing messages we use in tests. (there’s actually a whole big bunch of these use-cases, so I’ll skip the rest)
6. A container of worker threads, where the enum key represents the thread name+purpose, and the thread is dedicated for that work. (used in a thread-chain)
7. A container of timer-wheels, where the enum key represents the resolution of the timer-wheel.
8. A container of stats/counters, where the enum key represents a specific stat/counter purpose. (there’s a whole bunch of these too, so I’ll skip the rest)
9. A container of another enum type, where the enum key represents some internal components, and the enum values represent their current state. (up, down, disabled, etc.)
10. A container of function objects, where the enum key represents a rule/action to take, and the function objects perform it.
I skipped a bunch of self-similar ones in there, to give a better idea. Some of these are essentially global containers, and some are members of other objects.
Obviously we could have created separate C++ variables for each instance instead, instead of using an std::array – because basically we’re using the enum key as a form of programmatic “name” for variables.
The reason we do it this way instead, however, is because when you have many of these things it is far easier to write generic code to handle them this way.
For example stats/counters: of course at the point in the code that needs to increment/decrement them, that line of code has to identify a specific stat it wants to increment/decrement/whatever, and using an enum to do so vs. variable name is not much different. But when you write the observers of those stats, that need to iterate over them to write them somewhere else, or display them, or reset them all, or whatever, it’s a lot more convenient if they’re already in a container. That code doesn’t need to know variable names. It doesn’t even need to know specific enum values. It just has to know how to access the container. And we have thousands of stats (as do our competitors, and other industries too I’m sure). Obviously some people “register” individual stats into observers, which is fine too, but we don’t need to do that for every stat.
Same goes for test inputs/outputs. We don’t write every test by-hand, for each and every test-input/output we need. We just have generic tests that iterate over all inputs, and verify all outputs. (again, I’m sure a lot of people do this – it’s impractical to do otherwise)
The other advantage to using enums for these types of things, is once you have some mechanism to convert enums-to-strings, you can use those strings too for the generic code aspects. For example in the test inputs/outputs, the enum’s string is used to print out what’s being tested and is written to the test log, so if it fails we know which one. (googletest does this as well anyway via their macros stringizing stuff, but I’m just mentioning it) Another example is that array of threads – the enum strings become the thread string names used in logging for those threads, for example.
And you can do the reverse too – if you can convert strings-to-enums, then you can use strings to find/match entries in those arrays – strings such as might be in your config for example, or even from a command-line.
-hadriel
From: Boost <boost-...@lists.boost.org> on behalf of Richard Hodges via Boost <bo...@lists.boost.org>
Date: Friday, June 24, 2022 at 12:31 AM
I understand that presenting the type with the look and feel of a container
is somehow pleasing. The ability might be an interesting academic goal.
I am more interested in what real-world use-cases this solves.
boost.describe, wise_enum etc solve the case where (for example) I wish to
serialise and deserialise my enum value as text. This is useful in the real
world as, for example, it allows me to easily and reliably implement a JSON
communications protocol that contains my enum.
Hadriel Kaplan via Boost
I think this thing should automatically handle more than just Boost.Describe. I think it should try to handle magic_enum and wise_enum’s generated-enum-info too.
That would make it usable by a much broader user base.
If that’s technically challenging to do, then I think you should approach the authors of them and ask to “normalize” whatever needs to be the same, to make them all usable with this.
I don’t know how they’d respond, but it would be really cool if people could start writing things like index_array that supports all of them.
-hadriel
From: Boost <boost-...@lists.boost.org> on behalf of Richard Hodges via Boost <bo...@lists.boost.org>
Date: Friday, June 24, 2022 at 12:31 AM
Cc: Richard Hodges <hodg...@gmail.com>
Subject: Re: [boost] Library proposal - indexed_array
On Fri, 24 Jun 2022 at 02:01, Hadriel Kaplan <hadr...@yahoo.com> wrote:
boost.describe, wise_enum etc solve the case where (for example) I wish to
serialise and deserialise my enum value as text. This is useful in the real
world as, for example, it allows me to easily and reliably implement a JSON
communications protocol that contains my enum.
Julien Blanc via Boost
écrit :
> By the way Julien, I was going to suggest something eventually that
> might be… controversial, but since Richard brought them up I’ll
> mention it now:
>
> I think this thing should automatically handle more than just
> Boost.Describe. I think it should try to handle magic_enum and
> wise_enum’s generated-enum-info too.
>
> That would make it usable by a much broader user base.
That should not be too hard to do, but could require some additions in
these libraries. Basically, what is needed is a way to transform the
enumeration values list into a corresponding integer list, which then
gets feeded to the default_indexer – and of course a way to do compile-
time detection of the fact that the enum has some introspection (like
has_describe_enumerators). At that point, all the machinery is already
in place. For example, describe integration is only 25 lines of code.
Out of the box this should already work for wise-enum:
indexed_array<X,
interval<wise_enum::enumerators<MyEnum>::range.front(),
wise_enum::enumerators<MyEnum>::range.back()>>
But there is no contiguity check, and it would be much simpler to write
just indexed_array<X, MyEnum>.
> If that’s technically challenging to do, then I think you should
> approach the authors of them and ask to “normalize” whatever needs to
> be the same, to make them all usable with this.
>
> I don’t know how they’d respond, but it would be really cool if
> people could start writing things like index_array that supports all
> of them.
That would indeed be a nice addition. I'll try to see what can be done
here.
Best regards,
Julien
Peter Dimov via Boost
> By the way Julien, I was going to suggest something eventually that might be…
> controversial, but since Richard brought them up I’ll mention it now:
>
> I think this thing should automatically handle more than just Boost.Describe. I
> think it should try to handle magic_enum and wise_enum’s generated-enum-
> info too.
>
> That would make it usable by a much broader user base.
>
> If that’s technically challenging to do, then I think you should approach the
> authors of them and ask to “normalize” whatever needs to be the same, to
> make them all usable with this.
The intent of Boost.Describe is exactly to specify a standard type-based metadata
format for reflecting enums and classes, but whether the authors of these
libraries will be willing to adopt it, is up to them.
Julien Blanc via Boost
écrit :
> might be… controversial, but since Richard brought them up I’ll
> mention it now:
>
> I think this thing should automatically handle more than just
> Boost.Describe. I think it should try to handle magic_enum and
> wise_enum’s generated-enum-info too.
>
> That would make it usable by a much broader user base.
(replying off-list to avoid making too much noise)
Just for information, that's done (for both magic_enum and wise enum).
Adapting for others enum reflection libraries should be pretty
straightforward now with these three examples.
Best regards and thanks for supporting the library !
Julien
Hadriel Kaplan via Boost
> Yes, I'm looking for a real-world use case. The trivial hypothetical cases
> offered are building a static association between an enum value and some
> other value.
> This can be done with switch/case (for example), in about the same amount
> of typing, at compile time, in a function.
OK, now show me how to iterate over it. And how to get a view or slice of it (i.e., std::span or range). And how to get the number of elements. :)
Anyway, the main use-case or value of it, in my mind anyway, is actually the one when using Boost.Describe. Because when you're using Describe for an enum, you only need to do this for indexed_array:
indexed_array<string, MyEnum> data;
That's powerful, because you do not need to repeat any code, do not need to repeat any sizing/number of enums, etc. If you add new enumerators to your enum, you don't need to change this instantiation of indexed_array. It's all automatic, and you can't screw up.
-hadriel