SECTIONs within a loop

[JBRWilkinson]

Sections are uniquely named based on the line number of the source file. When a test case runs, there's a check to see whether that section has been entered.

This is all good until you come to the case where you might have a loop that covers the section more than once, for example:

TEST_CASE("Section in a Loop", "[section][loop]") {
  for (int x = 0; x < 11; ++x) {
    SECTION("dave", "") {
      const int y(11);
      REQUIRE(x < y);
    }
  }
}

clang++ -I../Catch/include --std=c++98 NamedSections.t.cpp -o NamedSections.bin && ./NamedSections.bin "Section in a loop"

All tests passed (1 assertion in 1 test case)

  }

So Catch found the test case by name and executed it. Once the section "dave" had been entered once, it wasn't entered again.

Consider the case where we have an array of allowed values that we wish to ensure are acceptable to some mechanism:

TEST_CASE("Section testing values", "[section][loop]") {
    const int allowedValues[] = { 1, 2, 3 };
    const size_t allowedValuesSize(sizeof(allowedValues)/sizeof(allowedValues[0]));
    for (int x = 0; x < allowedValuesSize; ++x) {
        SECTION("allowedValues", "") {
            const int y(allowedValues[x]);
            REQUIRE(y < 4);
        }
    }
}

clang++ -I../Catch/include --std=c++98 NamedSections.t.cpp -o NamedSections.bin && ./NamedSections.bin "Section testing values"

All tests passed (1 assertion in 1 test case)

Clearly, this is not what we wanted.

A workaround is to dynamically generate the name of the SECTION:

TEST_CASE("Named Section in a Loop", "[namedsection][loop]") {
  for (int x = 0; x < 11; ++x) {
    std::ostringstream ss;
    ss << "section2." << x;
    SECTION(ss.str(), "") {
      const int y(11);
      REQUIRE(x < y);
    }
  }
}

clang++ -I../Catch/include --std=c++98 NamedSections.t.cpp -o NamedSections.bin && ./NamedSections.bin "Named section in a loop"

All tests passed (11 assertions in 1 test case)

..but this feels like a hack and we do not know the side-effects.

What would be the 'proper' solution?

Anyone any suggestions?

Cheers,

James

[Phil Nash]

This is a known limitation that is also behind this: https://github.com/philsquared/Catch/issues/21

It was a big factor in the rewrite of the section runner recently. Getting that stuff right without introducing issues turns out to be quite tricky so my first concern was to simplify the logic and make it more open to being extended with support for loops and generators.

I've not gone back to add that support for two reasons:

1. I'd wanted to the new code to bed in a for a while and make sure no new regressions come up

2. It's a fairly involved piece of work on it's own which I need a good run at. I'm only working on this on my commute at the moment which makes that tricky.

I probably can't put it off for much longer, though.