flag types

[Sean Middleditch]

A very common need in many programs is to store a collection of flags, often stored as a collection of bits.  In the past, enums were typically used, often of a form similar to:

  enum flags {

    none = 0,

    first = 0x1,

    second = 0x2,

    third = 0x4,

  };

These would then often be stored in an integral type.  Since the enum value cast to integral type implicitly, you could use expressions like first|third or such with no problem.  Common problems, however, were that the enum constants were injected into the enclosing namespace, any integral valid would implicitly cast to the variable holding the flags, and so on.

C++11 style enum class solve most of these problems, but are an annoyance for flag types as they are designed to be strict enumerations.  Namely, they cannot be used with operators like |, &, ~, or ^, and there is no definitive type in which to store values made of these compositions.

Operating overloading and some cleverness can be employed to get around these problems, but not without pain.

Purely as an initial idea proposal, and this syntax is entirely made-up just to serve for exposition, imagine we had a facilities geared for flags:

   flag_type my_flags {

     first, // value is 1

     second, // value is 2

     third, // value is 4

   };

   flag_type other_flags : unsigned int { // signed types are not legal

    none = 0, // value is 0, of course

    first // value is 1

   };

   sizeof(my_flags) == sizeof(smallest unsigned integral type that can contain all defined values)

   sizeof(other_flags) == sizeof(unsigned int)

   my_flags foo = my_flags::first|my_flags::third; // valid, | is defined for any flag_type to be the expected result, typed as the corresponding flag_type

   my_flags bar = 5; // not valid, no conversion from int to my_flags

   my_flags gar = other_flags::first; // not valid, no conversion from other_flags to my_flags

   my_flags baz = foo&my_flags::first; // & is defined as works as expected, typed as corresponding flag_type

   bool test = baz; // not valid; no implicit conversion from a flag_type to boolean

   if (baz) // valid, implicit boolean context is true if value is not zero, false if it is zero

The idea is that we get all the strictness of enum class while still being usable for flags as pre-C++11 enums were.

The rule for values would be that if a value has an explicit assigned (none = 0), the value is as given.  If no assignment is given, the next power of two from the previous item is chosen, or 1 is chosen if the previous value was 0.  The first value listed is 1, not 0.

   flag_type foo {

     a,         // 1

     b = 3,   // 3

     c,         // 4, next power of 2 after 3

     d = a|c, // 5

     e,         // 8, next power of 2 after 5

     f = d|e,  // 13

     g,         // 16, next power of 2 after 13

     h = 3,   // 3, alias for b

   };

I could see a nullflag_type or something which is implicitly equal to 0 but also implicitly converts to any flag_type, equivalent to nullptr and nullopt.

I'm of the opinion a library solution would be best, but I've been incapable of creating one that works smoothly without a lot of repetition for each flags variant.  Minor language extensions may be needed to make a library approach feasible, such as the ability to inject enum values into another namespace, and the ability for friend operators of a template to override enum class operations (which doesn't work as of GCC 4.8; unsure if that's just a bug or not).

Larger language changes may be necessary to directly support a "flag_type" built-in, if a library approach can't be made that is easy to use and isn't too error-prone.  The best I can do now (which isn't too different from the C++98 days) is to make a template that wraps a particular enum.  This still make flag definition fragile (coders must be conscious of bit patterns; would be less troublesome with binary literals, but only slightly IMO), awkward scoping for values, and enum classes cannot be used if | and & are meant to work with lots of excess casting.

I note as prior art of attention to this problem is the C# flags attribute, which I consider to fall far short of a full solution.  C#'s built-in enum rules are a mix of C++98 and C++11 are better suited to use as flags than either C++ approach currently available.

On a highly related note, something like a std::max_enumeration_value<type>::value would be really great.  It is _extremely_ common to see things like:

   enum class foo {

     first,

     second,

     last // highest "legal" value of the enumeration, +1

   };

Used in loops, asserts, and so on.  Having such a construct both for enums (C++98 and C++11 style) as well as some flag_type would be very handy.

Sensible or off the deep end?

[Nicol Bolas]

You can already do this, since strongly typed enums are proper types. You can simply overload operators |, &, and such for them, disallowing other such operators.

[Sean Middleditch]

And that's part of the "awkward" part. One should not have to
overload the entire set of operators each and every time, just like
one does not have to provide an operator+ for every variant of
integer. I at least have seen a very large number of flag types in
larger applications and frameworks. The only sensible option now is a
set of preprocessor macros (which at least lets you _also_ solve the
other problem of string conversions and introspection, but there's
already a subcommittee looking into that one specifically). It also
still leaves the other issues unaddressed.

> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/a/isocpp.org/d/topic/std-proposals/1RPxJSJ_0z8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> Visit this group at
> http://groups.google.com/a/isocpp.org/group/std-proposals/.
>
>



--
Sean Middleditch
http://seanmiddleditch.com

[Jeffrey Yasskin]

I think std::bitset<N> provides most of the functionality you want
here. You'd define a normal enum class with contiguous values, and
you'd use the bitset type for sets of such enums. With the
std::max_enumeration_value<type> you suggest, one could easily write
an enum_set<enum_t> which deduces the proper N for a bitset. Chandler
(involved with the reflection SG) suggested that the best compiler
interface for this might be to provide a (constexpr) function template
that takes an enum type and returns a range containing all values of
that enum.

It might also be convenient to provide an implicit conversion from
enum_t to enum_set<enum_t>. I suspect we'd need a full solution for
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3682.html#33
(adding conversion operators for enum classes) for that to work.

But see if a bitset will get you most of the way to what you want today.

Jeffrey

> --
>
> ---
> You received this message because you are subscribed to the Google Groups

> "ISO C++ Standard - Future Proposals" group.

> To unsubscribe from this group and stop receiving emails from it, send an

[Richard Smith]

On Sun, Jul 21, 2013 at 9:19 PM, Jeffrey Yasskin <jyas...@google.com> wrote:

I think std::bitset<N> provides most of the functionality you want
here. You'd define a normal enum class with contiguous values, and
you'd use the bitset type for sets of such enums. With the
std::max_enumeration_value<type> you suggest, one could easily write
an enum_set<enum_t> which deduces the proper N for a bitset.

Look at Qt's QFlags for a pre-rolled implementation of something similar: QFlags<enum_type> gives you a type-safe collection of flags, with a complete set of bitwise operations. The only missing piece is that you need to manually specify the values for your enumeration constants as powers of two:

enum flag_values {

  none = 0,

  first = 0x1,

  second = 0x2,

  third = 0x4

};

using flags = QFlags<flag_values>;

Given that you can solve nearly all of this problem in a library, adding a significant new core language feature to support it seems excessive to me.

[Sean Middleditch]

QFlags represents the best you can do now, which is exactly what I was
getting at as a nasty workaround. I already know how to handle flags
today; I'm not exactly new at this game. :)

** QFlags and its ilk is the problem I'd like to solve here, not the
solution. ** Macros, no proper scoping of values, doesn't use the new
C++11 enum classes, can't be used with forward-declared enums, the
problem list goes on.

Again, I'm all for library-only solutions. QFlags ain't it.
std::bitset fails due to not using the type system at all (nothing
stops you from accidentally combining flags1::value with flags2::value
into the same bitset, and there's other issues involving how you'd
define "combined flag" constants in an obvious clean way). Pasting
together operator overloads for every set of flags is repetitive and
error-pone.

If the answer is "suck it up and deal with it," fine, that's what we
do already... but should we have to post C++17? Going to the ideal I
listed originally is probably a bit too much in terms of one-off
semantics, I agree, but surely there's a compromise we could discuss
involving much more minor (and useful elsewhere) tweaks enabling a
very clean and robust standard library solution, rather than "use
macros and throw away the type system" ?

If there were an easier way to define sets of operator overloads for
any enum, it could be a bit easier, maybe? Going back to the "old
style" enums, and ignoring temporarily the problem of combined
constants, a template perhaps could be:

namespace std {
struct nullflag_t { };
static constexpr nullflag_t nullflag;

template <typename EnumType>
// this inheritance is not allowed currently, of course; just a
strawman idea of how one could "inject"
// enum names into a class's namespace, I don't think this is the
ideal syntax here
class flag_set : public EnumType {
public:
typedef std::make_unsigned<typename
std::underlying_type<EnumType>::type>::type int_type;

flag_set() = default;
constexpr flag_set(const std::nullflag_t&) : _value(0) { }
constexpr flag_set(EnumType value) : _value(1 << value) { }
explicit constexpr flag_set(int_type value) : _value(value) { }

explicit constexpr operator int_type() const { return int_type(_value); }

friend constexpr flag_set operator|(flag_set lhs, flag_set rhs)
{ return flag_set(int_type(_value)|int_type(_value)); }
// other relevant bitwise operators

private:
EnumType _value;
};

This is very close to what is done today, plus all that macro nonsense
you see in things like QFlags to get around the lack of ability to
inject scopes.

Being able to inject the enum scope gives two things in this case.
First, it means that you don't have one type name meant for use of
values of a flag set and entirely different one for "constants" of the
value set. Two, it means with some additional unpleasantness you
could also introduce "parallel" names for constant _combinations_ of
flags, albeit with some ugly syntax:

enum class flags_values { first, second, third };
struct flags : public std::flag_set<flags_values> {
constexpr flag_set<flags_values> all = first|second|third;
};

// definition is awkward but not usage is super obvious
auto foo = flags::all & ~flags::second; // decltype(foo) ===
std::flag_set<flags_values>, probably not a serious issue

It of course may be cleaner to just treat the base enum type as is
done today (the actual bit pattern and not a bit offset that must be
shifted) and just continue requiring the client user to remember to
use powers of two for unique bits. I've seen people have trouble with
that in the past but I'm thinking binary literals will make it way
better, probably "good enough," for most developers and relatively
easy to teach.

Without at least that, you end up with something similar to this, if
avoiding macros:

enum class flags { none = 0b0000, first = 0b0001, second = 0b0010,
third = 0b0100 };
constexpr auto flags_all = std::flag_set<flags>(flags::first) |
flags::second | flags::third; // necessary cast of first element

auto foo = flags::first; // just a plain enum class, not a flag set,
auto as "dragon typing"
auto foo = std::flag_set<flags>(flags::first);

auto bar = flags::all ^ flags::second; // oops, 'all' is not part of
the flags scope like every other value
auto bar = flags_all ^ flags::second; // better remember where each
value lives!

auto gaz = flags::first | flags::second; // oops, didn't define all
the operator overloads
auto gaz = std::flag_set<flags>(flags::first) | flags::second; // *sigh*

flags baz = 0; // oops, illegal conversion from int to enum class!
flags baz = std::nullflag; // oops, illegal conversion to enum class!
auto baz = flags::none; // not terrible, but have fun with templates
and be sure to consistently name the none value!

Clearly, the above is non-ideal. QFlags' macros make it slightly less
error-prone in some ways, more so in others.

> --
>
> ---
> You received this message because you are subscribed to a topic in the

> Google Groups "ISO C++ Standard - Future Proposals" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/a/isocpp.org/d/topic/std-proposals/1RPxJSJ_0z8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> Visit this group at
> http://groups.google.com/a/isocpp.org/group/std-proposals/.
>
>

[Martinho Fernandes]

On Mon, Jul 22, 2013 at 9:26 AM, Sean Middleditch <se...@middleditch.us> wrote:
> QFlags represents the best you can do now, which is exactly what I was
> getting at as a nasty workaround. I already know how to handle flags
> today; I'm not exactly new at this game. :)
>
> ** QFlags and its ilk is the problem I'd like to solve here, not the
> solution. ** Macros, no proper scoping of values, doesn't use the new
> C++11 enum classes, can't be used with forward-declared enums, the
> problem list goes on.
>
> Again, I'm all for library-only solutions. QFlags ain't it.
> std::bitset fails due to not using the type system at all (nothing
> stops you from accidentally combining flags1::value with flags2::value
> into the same bitset, and there's other issues involving how you'd
> define "combined flag" constants in an obvious clean way). Pasting
> together operator overloads for every set of flags is repetitive and
> error-pone.
>

I have written a library solution that uses a different approach a while back.

You can see it at
https://github.com/rmartinho/wheels/blob/master/include/wheels/enums.h%2B%2B
(note that the macros are merely for code generation; they do not make
part of the interface).

Usage goes as follows:

enum class some_flags { none = 0, first = 1<<1, second = 1<<2,
third = 1<<3 };
namespace wheels { namespace enums {
template <> struct is_flags<some_flags> : std::true_type {};
} }

// in some scope
{
using namespace wheels::enums::operators;
auto first_and_second = some_flags::first | some_flags::second;

To be honest, I don't think binary literals really make things more
readable; for readability I personally prefer 1<<n, or a constexpr
bit(n) function.

> Without at least that, you end up with something similar to this, if
> avoiding macros:
>
> enum class flags { none = 0b0000, first = 0b0001, second = 0b0010,
> third = 0b0100 };
> constexpr auto flags_all = std::flag_set<flags>(flags::first) |
> flags::second | flags::third; // necessary cast of first element
>
> auto foo = flags::first; // just a plain enum class, not a flag set,
> auto as "dragon typing"
> auto foo = std::flag_set<flags>(flags::first);
>
> auto bar = flags::all ^ flags::second; // oops, 'all' is not part of
> the flags scope like every other value
> auto bar = flags_all ^ flags::second; // better remember where each
> value lives!
>
> auto gaz = flags::first | flags::second; // oops, didn't define all
> the operator overloads
> auto gaz = std::flag_set<flags>(flags::first) | flags::second; // *sigh*
>
> flags baz = 0; // oops, illegal conversion from int to enum class!
> flags baz = std::nullflag; // oops, illegal conversion to enum class!
> auto baz = flags::none; // not terrible, but have fun with templates
> and be sure to consistently name the none value!

`flags{}` is a perfectly valid "zero" value for any enum. You can
either write `flags baz = {};` or `auto baz = flags{};`.

All that said... Do we really want to carry along the bitwise
operators if we are going to give a more serious treatment to flag
sets? I think the only reason to use these is the legacy of using
numbers for flag sets. These days I prefer to use named constexpr
functions, similar to what I have here:
https://gist.github.com/rmartinho/5456207#file-bit_ops-cpp-L86.

[Gabriel Dos Reis]

Sean Middleditch <se...@middleditch.us> writes:

| QFlags represents the best you can do now, which is exactly what I was
| getting at as a nasty workaround. I already know how to handle flags
| today; I'm not exactly new at this game. :)

As you may probably have noted, C++ has never been *perfect* at one
single thing. It's consistently been good, better than the average, at
many things, which one would argue is key to its strength.

I suspect a question that your previous messages have not addressed (at
least not adequately) is whether the need to be perfect at bitmask type
is so urgent that one needs to introduce a fourth or fifth way of
defining a type in C++ -- when there are good library solutions. Note,
this is not a frivolous question: if you want your proposal to gain
traction, you would need to answer it forcefully, convincingly, over and over.

-- Gaby

[Jeffrey Yasskin]

On Jul 22, 2013 12:26 AM, "Sean Middleditch" <se...@middleditch.us> wrote:
> Again, I'm all for library-only solutions.  QFlags ain't it.
> std::bitset fails due to not using the type system at all (nothing
> stops you from accidentally combining flags1::value with flags2::value
> into the same bitset, and there's other issues involving how you'd
> define "combined flag" constants in an obvious clean way).  Pasting
> together operator overloads for every set of flags is repetitive and
> error-pone.

Recall that I suggested an enum_set<enum_t> given an extension for reflecting enum values, not just a raw bitset. It'd be easy to make enum_set <enum_t>'s operator[] take an enum_t and not just an int. With today's language you'd define combined constants with
  auto combined = make_enum_set(value1) | make_enum_set(value2);

Or possibly make_enum_set(value1, value2) if the library wanted.

With the language extension I mentioned to define conversion operators, I suspect you could just use "value1 | value2".

[Sean Middleditch]

On Mon, Jul 22, 2013 at 12:45 AM, Martinho Fernandes
<martinho....@gmail.com> wrote:
>
> I have written a library solution that uses a different approach a while back.

I like your approach. I'll have to give a try and see how it works
out in practice, but it looks very good. I hadn't though to use type
traits and enable_if to constrain generalized global operators for
this case.

I agree that using constexpr or higher level operators makes more
sense than bit operations. I've seen + and - used, but I've never
quite liked those; I can't put good words down as to why.

I also agree that a constexpr bit(n) or the like is better than the
binary literals. I've had no chance to really play with them (no
constexpr yet on a primary compiler our entire industry generally
targets) but I generally today use a macro like:
#define BIT(n) (1U << (n))

The constexpr function version of that combined with a constexpr "next
power of 2" (which is handy for _so_ many other things, of course)
would likely be more than sufficient for defining flagsets.

On Mon, Jul 22, 2013 at 7:09 AM, Gabriel Dos Reis <g...@axiomatics.org> wrote:
> Sean Middleditch <se...@middleditch.us> writes:
> I suspect a question that your previous messages have not addressed (at
> least not adequately) is whether the need to be perfect at bitmask type
> is so urgent that one needs to introduce a fourth or fifth way of
> defining a type in C++ -- when there are good library solutions. Note,
> this is not a frivolous question: if you want your proposal to gain
> traction, you would need to answer it forcefully, convincingly, over and over.

I'm totally in _favor_ of a library solution, just (a) not one
requiring macros, extensive repetition, or awkwardness, (b) able to be
rolled into the standard library so it's there for everyone, and (c)
future resilient presuming modules and such become the norm 10 years
from now (hence no macros). I only propose language extensions if a
library can't be made to work for some reason. Especially language
changes that have few uses outside of one specific use case; those are
the worst burden of all.

I'd like to implement Martinho's solution in a larger project and see
how it settles with a team of other programmers, and come back to this
if it doesn't work out after some hands-on experience with it
(unfortunately we don't have constexpr on one of our teams' target
compilers).

I'm not terribly fond of needing to specialize templates to enforce
type traits (especially if they end up in namespace std in an
"official" implementation), same reasoning as in N3333 (proposing
ASL-friendly std::hash_value) and why rvalue references are a thing
instead of just using manual movable type traits like some of us did
in C++98. Is there an alternative to that in Martinho's example
without any new extensions?

[Richard Smith]

On Mon, Jul 22, 2013 at 12:26 AM, Sean Middleditch <se...@middleditch.us> wrote:

QFlags represents the best you can do now, which is exactly what I was
getting at as a nasty workaround.  I already know how to handle flags
today; I'm not exactly new at this game. :)

** QFlags and its ilk is the problem I'd like to solve here, not the
solution. **

I think drilling into the specific problems with QFlags would be very instructive here. I don't yet see any showstopper problems.

 

Macros,

QFlags has two macros. One of them just expands to "typedef QFlags<Foo> Bar;", and there seems to be no problem with writing that directly. The other is more interesting: it provides an operator| for the underlying enum type, that produces a QFlags value rather than an integer. This seems tricky to handle without a core language change.

 

no proper scoping of values,

You can write flag_values::first and so on. That's admittedly suboptimal, because the flags type and the enum type have different names. That can be fixed with a slightly different mechanism, such as:

// implementation:

template<typename T> struct flags : T {

  typename T::type value;

  flags(typename T::type);

  friend flags operator|(flags a, flags b);

  // ...

  explicit operator typename std::underlying_type<typename T::type>::type() const;

};

// usage:

struct my_flags_impl {

  enum type {

    none = 0,

    first = 0x1,

    second = 0x2,

    third = 0x4

  };

};

using my_flags = flags<my_flags_impl>;

my_flags f = my_flags::first;

my_flags g = (f | my_flags::second) & ~my_flags::third;

doesn't use the new C++11 enum classes,

That would be the tail wagging the dog. Who cares? Also, I don't see any reason why it *wouldn't* work with enum classes.

 

can't be used with forward-declared enums,

Why not? The above approach would work fine with a forward-declared enum.

Depending on the resolution of core issue 1485, there's another approach we can take:

// implementation

template<typename Underlying, typename Tag> struct flags {

  enum type : Underlying;

  flags(type);

  friend flags operator|(flags, flags);

  friend flags operator|(type, type);

  // ...

};

// use

struct my_flags_tag;

using my_flags = flags<int, my_flags_tag>;

template<> enum my_flags::type : int {

  none = 0,

  first = 0x1,

  second = 0x2,

  third = 0x4

};

This approach works today with Clang.

[Sean Middleditch]

On Monday, July 22, 2013 12:14:29 PM UTC-7, Richard Smith wrote:

// use

struct my_flags_tag;

using my_flags = flags<int, my_flags_tag>;

template<> enum my_flags::type : int {

  none = 0,

  first = 0x1,

  second = 0x2,

  third = 0x4

};

I had absolutely no idea this kind of use of forward-declared enums in templates was legal, but in retrospect, it seems obvious.  Excellent.  What would use of this type look like?

  auto foo = my_flags::type::second; // is that 'type' necessary there?

Following up then, is there maybe a call for the constexpr helpers mentioned earlier being in the standard library?  std::bit, std::next_power_of_two/std::next_bit, etc (with better names of course).  Some of these are super trivial to just write (like a constexpr bit), but others are just enough code to get obnoxious even with C++14 constexpr if I had to rewrite for every sample, helper library, and so on (like constexpr next_next_of_two).

[Richard Smith]

On Thu, Jul 25, 2013 at 10:11 AM, Sean Middleditch <sean.mid...@gmail.com> wrote:

On Monday, July 22, 2013 12:14:29 PM UTC-7, Richard Smith wrote:

// use

struct my_flags_tag;

using my_flags = flags<int, my_flags_tag>;

template<> enum my_flags::type : int {

  none = 0,

  first = 0x1,

  second = 0x2,

  third = 0x4

};

I had absolutely no idea this kind of use of forward-declared enums in templates was legal, but in retrospect, it seems obvious.  Excellent.  What would use of this type look like?

  auto foo = my_flags::type::second; // is that 'type' necessary there?

The ::type is not necessary. The purpose of using an explicit specialization of the enum, rather than some more conventional mechanism, is to inject those names into the specialization of the surrounding class template specialization. Core issue 1485 might make the above mechanism ill-formed, but isn't resolved yet, so now would be the time to provide a compelling reason to keep this valid.

Following up then, is there maybe a call for the constexpr helpers mentioned earlier being in the standard library?  std::bit, std::next_power_of_two/std::next_bit, etc (with better names of course).  Some of these are super trivial to just write (like a constexpr bit), but others are just enough code to get obnoxious even with C++14 constexpr if I had to rewrite for every sample, helper library, and so on (like constexpr next_next_of_two).

I think this would make sense, especially for operations which can be mapped to a single instruction on some hardware (count_set_bits, count_trailing_zeroes, ...). If the user writes these, it's not easy to get them to be both valid in constant expressions and efficient outside constant expressions (if you can't rely on inline asm or compiler builtins, you've got to hope your optimizer recognizes the code you write and replaces it with the right operation). If they're part of the implementation, it's trivial.