Best way to check a multi-line string value

[David Wallace]

I've been experimenting with different ways to test a multi-line string value, in an effort to produce more meaningful failure messages when the value changes.  Here's a test that recently failed, as it was originally written:

    TEST_F(StructVendorOptionsTest, CheckWriteHeaderDefault) {
        SSDCVendorOptions options = initVendorSdcFormat(ESDCFormat::eSDCUnknown);
        std::ostringstream oss;
        options.writeHeader(oss);
        std::string expected =
            "# SDC formatting set by user options\n"
            "#   Split Vectors: no\n"
            "#   Add _reg To Register Names: yes\n"
            "#   Generate Exceptions containing get_nets: yes\n"
            "#   Expand Vector based Q ports: no\n"
            "#   Generate Exceptions containing through constraints: yes\n"
            "#   Comment out set_hierarchy_separator: no\n"
            "#   Generate set_clock_groups: yes\n"
            "#   Generate TCL Variable for buses: no\n"
            "#   Insert newlines for readability into TCL: no\n"
            "#   Combine Constraints when possible: no\n"
            ;
        EXPECT_EQ(expected, oss.str());
    }

And here is the failure message when the result changed:

[ RUN      ] StructVendorOptionsTest.CheckWriteHeaderDefault
..\..\..\..\Shared\SDCVendorOptions\UnitTests\TestSDCVendorOptions.cpp(623): error: Value of: oss.str()
  Actual: "# SDC formatting set by user options\n#   Split Vectors: no\n#   Add _reg To Register Names: yes\n#   Generate Exceptions containing get_nets: yes\n#    Expand Vector based Q ports: no\n#   Generate Exceptions containing through constraints: yes\n#   Comment out set_hierarchy_separator: no\n#   Generate set_clock_groups: yes\n#   Use Source Clock in create_generated_clock command: yes\n#    Generate TCL Variable for buses: no\n#   Insert newlines for readability into TCL: no\n#   Combine Constraints when possible: no\n"
Expected: expected
Which is: "# SDC formatting set by user options\n#   Split Vectors: no\n#   Add_reg To Register Names: ye s\n#   Generate Exceptions containing get_nets: yes\n#   Expand Vector based Q ports: no\n#   Generate Exceptions containing through constraints: yes\n#   Comment out set_hierarchy_separator: no\n#   Generate set_clock_groups: yes\n#   Generate TCL Variable for buses: no\n#   Insert newlines for readability into TCL: no\n#   Combine Constraints when possible: no\n"
[  FAILED  ] StructVendorOptionsTest.CheckWriteHeaderDefault (2 ms)

It's not particularly easy to pick out the difference in the long blocks of text, particularly if the difference is more subtle than this one.  One approach I've been using is to split the string up into lines (splitString produces a std::list of strings using the second arg as a splitting char), and use a macro:

    //Macro to facilitate checking one line of a multiline output that has been through splitString:

#define CHECK_NEXT_LINE(vect, iter, expected_str) ASSERT_TRUE((iter) != (vect).end()); EXPECT_EQ((expected_str), *((iter)++));

        auto tmp = NStringUtils::splitString(oss.str(), '\n', true);

        auto li = tmp.begin();

        CHECK_NEXT_LINE(tmp, li, "# SDC formatting set by user options");

        CHECK_NEXT_LINE(tmp, li, "#   Split Vectors: no");

        CHECK_NEXT_LINE(tmp, li, "#   Add _reg To Register Names: yes");

        CHECK_NEXT_LINE(tmp, li, "#   Generate Exceptions containing get_nets: yes");

        CHECK_NEXT_LINE(tmp, li, "#   Expand Vector based Q ports: no");

        CHECK_NEXT_LINE(tmp, li, "#   Generate Exceptions containing through constraints: yes");

        CHECK_NEXT_LINE(tmp, li, "#   Comment out set_hierarchy_separator: no");

        CHECK_NEXT_LINE(tmp, li, "#   Generate set_clock_groups: yes");

        CHECK_NEXT_LINE(tmp, li, "#   Generate TCL Variable for buses: no");

        CHECK_NEXT_LINE(tmp, li, "#   Insert newlines for readability into TCL: no");

        CHECK_NEXT_LINE(tmp, li, "#   Combine Constraints when possible: no");

        EXPECT_EQ(tmp.end(), li);

This produces a more informative set of failure messages, showing which lines are different, starting with the first difference, and identifying the problem: a new line has been inserted in the middle of the output:

[ RUN      ] StructVendorOptionsTest.CheckWriteHeaderDefault

..\..\..\..\Shared\SDCVendorOptions\UnitTests\TestSDCVendorOptions.cpp(604): error: Value of: *((li)++)

  Actual: "#   Use Source Clock in create_generated_clock command: yes"

Expected: ("#   Generate TCL Variable for buses: no")

Which is: "#   Generate TCL Variable for buses: no"

..\..\..\..\Shared\SDCVendorOptions\UnitTests\TestSDCVendorOptions.cpp(605): error: Value of: *((li)++)

  Actual: "#   Generate TCL Variable for buses: no"

Expected: ("#   Insert newlines for readability into TCL: no")

Which is: "#   Insert newlines for readability into TCL: no"

..\..\..\..\Shared\SDCVendorOptions\UnitTests\TestSDCVendorOptions.cpp(606): error: Value of: *((li)++)

  Actual: "#   Insert newlines for readability into TCL: no"

Expected: ("#   Combine Constraints when possible: no")

Which is: "#   Combine Constraints when possible: no"

..\..\..\..\Shared\SDCVendorOptions\UnitTests\TestSDCVendorOptions.cpp(607): error: Value of: li

  Actual: 8-byte object <B0-5D 99-00 00-00 00-00>

Expected: tmp.end()

Which is: 8-byte object <B0-58 99-00 00-00 00-00>

[  FAILED  ] StructVendorOptionsTest.CheckWriteHeaderDefault (2 ms)

I've also tried comparing directly to the list of strings generated by splitString, which produces output similar to the whole string comparison, although a little bit easier to read:

        auto tmp = NStringUtils::splitString(oss.str(), '\n', true);       
        std::list<std::string> expected_list = {

            "# SDC formatting set by user options",

            "#   Split Vectors: no",

            "#   Add _reg To Register Names: yes",

            "#   Generate Exceptions containing get_nets: yes",

            "#   Expand Vector based Q ports: no",

            "#   Generate Exceptions containing through constraints: yes",

            "#   Comment out set_hierarchy_separator: no",

            "#   Generate set_clock_groups: yes",

            "#   Generate TCL Variable for buses: no",

            "#   Insert newlines for readability into TCL: no",

            "#   Combine Constraints when possible: no"

        };

        EXPECT_EQ(expected_list, tmp);

[ RUN      ] StructVendorOptionsTest.CheckWriteHeaderDefault

..\..\..\..\Shared\SDCVendorOptions\UnitTests\TestSDCVendorOptions.cpp(623): error: Value of: tmp

  Actual: { "# SDC formatting set by user options", "#   Split Vectors: no", "#   Add _reg To Register Names: yes", "#   Generate Exceptions containing get_nets: yes", "#   Expand Vector based Q ports: no", "#   Generate Exceptions containing through constraints: yes", "#   Comment out set_hierarchy_separator: no", "#  Generate set_clock_groups: yes", "#   Use Source Clock in create_generated_clock command: yes", "#   Generate TCL Variable for buses: no", "#   Insert newlines for readability into TCL: no", "#   Combine Constraints when possible: no" }

Expected: expected_list

Which is: { "# SDC formatting set by user options", "#   Split Vectors: no", "#   Add _reg To Register Names: yes", "#   Generate Exceptions containing get_nets: yes", "#   Expand Vector based Q ports: no", "#   Generate Exceptions containing through constraints: yes", "#   Comment out set_hierarchy_separator: no", "#   Generate set_clock_groups: yes", "#   Generate TCL Variable for buses: no", "#   Insert newlines for readability into TCL: no", "#   Combine Constraints when

possible: no" }

[  FAILED  ] StructVendorOptionsTest.CheckWriteHeaderDefault (2 ms)

Finally, I have another test that uses matchers to look for specific substrings in the result, which works.  But it doesn't check the ordering of the options or fail if new options are added (as here), so it is less useful as documentation of the complete behavior of the method:

    TEST_F(StructVendorOptionsTest, CheckWriteHeaderDefaultWithMatchers) {

        SSDCVendorOptions options = initVendorSdcFormat(ESDCFormat::eSDCUnknown);

        std::ostringstream oss;

        options.writeHeader(oss);

        std::string result = oss.str();

        EXPECT_THAT(result, HasSubstr("SDC formatting set by user options"));

        EXPECT_THAT(result, HasSubstr("Split Vectors: no"));

        EXPECT_THAT(result, HasSubstr("Add _reg To Register Names: yes"));

        EXPECT_THAT(result, HasSubstr("Expand Vector based Q ports: no"));

        EXPECT_THAT(result, HasSubstr("Generate Exceptions containing through constraints: yes"));

        // These next three are the result of config vars - these are the default settings

        EXPECT_THAT(result, HasSubstr("Generate TCL Variable for buses: no"));

        EXPECT_THAT(result, HasSubstr("Insert newlines for readability into TCL: no"));

        EXPECT_THAT(result, HasSubstr("Combine Constraints when possible: no"));

    }

Are there other approaches that people have found useful for this problem?  Would it be possible to add more built-in support for something like my macro approach in future versions of gtest?

Dave W.  (da...@bluepearlsoftware.com)

[Josh Kelley]

I ab(use) Google Test's AssertionResult predicate assertions for this sort of thing.

For your example, you could do something like

testing::AssertionResult MultiLineStringsMatch(const std::string& expected, const std::string& actual)
{

  if (expected == actual) {

    return testing::AssertionSuccess();

  }

  // From here, you can print as many details as you want,

  // intermixed with regular Google Test output...

  PrettyPrintMultiLineDiff(expected, actual);

  // Or feed them into the assertion failure with <<

  return testing::AssertionFailure()

    << "Some explanatory text:\n"

    << "  More details\n"

    << "  Even more details\n";
}

To use it:

EXPECT_TRUE(MultiLineStringsMatch(expected, oss.str()));

-- 

Josh Kelley

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframework+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.