Modifying/setting section name during runtime

[Janky Ferenc]

Hey,

Is it possible to modify/set a SECTION name during runtime?

I'd like to do this for adding type info in the detailed test output for type parametrized test cases.

Current implementation:
https://github.com/fecjanky/poly_vector/blob/master/test/include/catch_ext.hpp

Basically I'd like to wrap line 44 Func<T>::execute() in a SECTION() where the section name is set during runtime from a string parameter.

Any other ideas are welcome.

Thanks & BR,

Ferenc

[Martin Hořeňovský]

You should be able to just pass a std::string to the SECTION, e.g.

SECTION("Inputs: i := " + std::to_string(i) + ", j := " + std::to_string(j)) {

    test-code

}

With the next release, we will also be providing a dynamic section macro that uses streams, so you could do this:

DYNAMIC_SECTION("Inputs: i := " << i << ", j := " << j) {

    ...

}

--
You received this message because you are subscribed to the Google Groups "CATCH" group.
To unsubscribe from this group and stop receiving emails from it, send an email to catch-forum+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Phil Nash]

Yes, constructing a SECTION name at runtime inside a templated helper function is exactly how I've done type-parameterised tests in the past.

DYNAMIC_SECTION should make this even easier, but is not necessary (as Martin explained well).

First class type-parameterised tests is on the todo list (has been for a long time) - but I'm working on data-parameterised tests right now ("Generators").

[Janky Ferenc]

Hey,

Thanks for the suggestions. I made the modifications according to your suggestion but an assertion fails in TrackerBase::moveToParent() for m_parent being null.
Do you have any idea what could be the problem?

Thanks & Br

Ferenc

[Phil Nash]

Oh, that's weird. Do you have a minimal reproducible example?

[Martin Hořeňovský]

We have an issue open about that[1, maybe 2], but no-one has been able to provide stable reproduction steps so far. If you can minimize a test case for it, it would be quite welcome.

[1] https://github.com/catchorg/Catch2/issues/1241 

[2] https://github.com/catchorg/Catch2/issues/1210

Martin Hořeňovský

Ferenc

[Phil Nash]

I've just been looking at the code in that area and can see something suspicious.

To test my theory in your case could you try making a small change to your Catch code?

Where the assertion is that's firing (in moveToParent), in the same class (TrackerBase) is a method, fail(), which calls it. That should have lines something like:

if( m_parent )
    m_parent->markAsNeedingAnotherRun();
moveToParent();

Could you change it to:

if( m_parent ) {
    m_parent->markAsNeedingAnotherRun();
    moveToParent();
}

And see if that fixes it?

[Janky Ferenc]

Hey,

The callstack is like
RunContext::sectionEnded() -> TrackerBase::close() -> TrackerBase::close() -> TrackerBase::moveToParent().

I can reproduce it deterministically so if you have any ideas I can try it out.

Phil, maybe a null check is missing here also like in fail() method?

BR,

Ferenc

[Phil Nash]

On Tuesday, 10 July 2018 11:03:00 UTC+1, Janky Ferenc wrote:

Hey,

The callstack is like
RunContext::sectionEnded() -> TrackerBase::close()  -> TrackerBase::close() -> TrackerBase::moveToParent().

Thanks for getting back - that's interesting!

 

I can reproduce it deterministically so if you have any ideas I can try it out.

Phil, maybe a null check is missing here also like in fail() method?

Maybe - more importantly, the code is not telling me what's supposed to be happening here - so my failing is not documenting it (or writing it so the documentation is not needed) - I'll need to dig a bit more.

Do try adding the null check in close() - but I'm not sure, yet, if that's the right place to fix this.

Thanks again for your help with this - this really needs to be rock solid!

Regards,

Phil 

[Janky Ferenc]

Hey,

Update:Wrapping the moveToParent call in null check resulted in infinite recursion.

BR,

Ferenc

[Phil Nash]

Thanks. I suspected there would be more to it :-(

I'll have to take a deeper look a bit later.

[Janky Ferenc]

Please find a sample code below for reproduction (I used Catch v.2.2.0)

#define CATCH_CONFIG_MAIN
#include <catch.hpp>

namespace detail {
    template<template<typename> class Func, typename... Ts>
    struct for_each_type;
}

#define INTERNAL_STRINGIFY_IMPL(X) #X
#define INTERNAL_STRINGIFY(X) INTERNAL_STRINGIFY_IMPL(X)

#define INTERNAL_TYPE_P_TEST_CASE(TESTSTRUCT, NAME, DESCR, TPARAM, ...)        \
  template<typename TPARAM>                                                    \
  struct TESTSTRUCT                                                            \
  {                                                                            \
    static void execute();                                                     \
  };                                                                           \
  TEST_CASE(NAME, DESCR)                                                       \
  {                                                                            \
    ::detail::for_each_type<TESTSTRUCT, __VA_ARGS__>::execute(                 \
      INTERNAL_STRINGIFY(TPARAM));                                             \
  }                                                                            \
  template<typename TPARAM>                                                    \
  void TESTSTRUCT<TPARAM>::execute()

#define TYPE_P_TEST_CASE(NAME, DESCR, TPARAM, ...)                             \
  INTERNAL_TYPE_P_TEST_CASE(                                                   \
    INTERNAL_CATCH_UNIQUE_NAME(                                                \
      ____C_A_T_C_H____T_E_S_T___E_X_T_E_N_S_I_O_N____),                       \
    NAME,                                                                      \
    DESCR,                                                                     \
    TPARAM,                                                                    \
    __VA_ARGS__)

namespace detail {

    template<template<typename> class Func>
    struct for_each_type<Func>
    {
        static void execute(const std::string&) {}
    };

    template<template<typename> class Func, typename T, typename... Ts>
    struct for_each_type<Func, T, Ts...>
    {
        static void execute(const std::string& tParamName)
        {
            SECTION(tParamName, typeid(T).name()) {
                Func<T>::execute();
            }
            for_each_type<Func, Ts...>::execute(tParamName);
        }
    };
}


TYPE_P_TEST_CASE("repor", "[repro]", T, int, double) {

    std::vector<T> vec;

    SECTION("push back") {
        vec.push_back(T{});
        REQUIRE(vec.size() == 1);
    }

    SECTION("empty") {
        REQUIRE(vec.empty());
    }
}

[Janky Ferenc]

After some experimentation I've found out that the assertion is not firing if the description is omitted from the outer SECTION() macro

[Janky Ferenc]

I believe I've found the cause of the assertion fail. Actually I've not omitted the description, but instead I've mangled in the typename to the name of the section and everything was working fine. I belive the problem is that in SectionTracker::acquire
 the previous already completed section is found instead of creating a new one for the current iteration, because the NameAndLocation info is the same. 

By looking at the code I don't think that this can be solved easily since the section mechanism relies on this operation. 

Maybe this could be added to the docs: if a SECTION is created in a recursive context  then some information that depends on the recursion level/iteration count must be mangled into the section name to have different NameAndLocation info.

[Phil Nash]

Oh, that sheds a lot of light on it! In theory I see the issue - but I haven't had a chance to look at the code yet to be sure. Not sure if the lookup name needs to include the whole section path, or if there is some way of ensuring that's not necessary - but either way it definitely sounds like a problem with the Catch code - not something to fix with documenting a workaround.

Thanks!

Unfortunately I won't have a chance to look at this until the end of next week :-(