INSTANTIATE_TEST_CASE_P using Arrays not discoverable with --gtest_list_tests ??

[Kim Gentes]

I am recently charged with making updates to our GTEST unit tests. We are in a MS Visual Studio (2013) environment.   The unit tests execute fine when called to run. However, when I use the --gtest_list_tests switch on the GTEST executable, I see that a number of our tests report some warning emanating from the "GetParam()" method call inside of the individual unit test Setup method.  In all cases where this happens the current written tests are using INSTANTIATE_TEST_CASE_P to generate tests by passing an array into the testing::ValuesIn(<array_namehere>) parameter of the INSTANTIATE_TEST_CASE_P call.

The result of this is that I can build and run the unit tests fine, with appropriate results. However, because I cannot use the --gtest_list_tests parameter with proper results, the unit tests are not all being proper discovered by MS Visual Studio when trying to use their Test Explorer and Code Coverage tools.  But don't let that confuse the issue.

The real problem is that the --gtest_list_tests switch seems to have a problem with unit tests generated using arrays with the INSTANTIATE_TEST_CASE_P feature.  

Is this something others have seen?  Is something different required when using INSTANTIATE_TEST_CASE_P ?

Please help... I have been reading and searching everywhere (including this forum), but haven't seen any similar situations exactly.  I am hoping someone else has encountered this and can tell me the error of our ways...

thank you

[details below]

Here are the details--

To repeat, this works fine to actually execute (run) the unit tests.  But if I run the test in this format :

MyParserUnitTests.exe --gtest_list_tests

I get the following results

[good results of individually created tests]

CParameterParser.
  ParseRealParamRefsReal
  ParseRealParamRefsInteger
CReportParser.
  ParseIntParamRefsAbsoluteValueInt

[strange results of tests written to be instantiated with INSTANTIATE_TEST_CASE_P  ]

ParameterizedFailure/CParameterParserFailures.
  ParseFailure/0  # GetParam() = 8-byte object <28-F8 18-00 F8-CD 17-00>
  ParseFailure/1  # GetParam() = 8-byte object <78-F8 18-00 D8-BA 17-00>
  ParseFailure/2  # GetParam() = 8-byte object <C8-F8 18-00 30-BB 17-00>
  ParseFailure/3  # GetParam() = 8-byte object <18-F9 18-00 B0-CD 17-00>
  ParseFailure/4  # GetParam() = 8-byte object <68-F9 18-00 40-CE 17-00>
... ... ... .... .... .... ....
  ParseFailure/59  # GetParam() = 8-byte object <48-36 19-00 68-BA 14-00>
  ParseFailure/60  # GetParam() = 8-byte object <98-36 19-00 E8-BA 14-00>
  ParseFailure/61  # GetParam() = 8-byte object <E8-36 19-00 C8-49 19-00>
  ParseFailure/62  # GetParam() = 8-byte object <30-3B 19-00 48-4A 19-00>
  ParseFailure/63  # GetParam() = 8-byte object <78-3B 19-00 C8-4A 19-00>

What does this mean??? 

I list the pertinent sections of one of my tests here.  (the .h file first)

///pertinent information in the .h file


const struct ParserFailureData
{
 CString m_Tag;
 CString m_strExprssn;
};


class CParameterParserFailures: public testing::TestWithParam<ParserFailureData>
{
protected:
 virtual void SetUp();
 virtual void TearDown();


 BOOL m_bResult;
 CParameterExpressionNode* m_pExprRoot;
 FunctionClass* m_pFunctionClass;
 CEQParameter* m_pParam;
 CContextConfiguration1* m_ContextConfiguration;
 CTestGroup1* m_TestFormula;
 CString m_strLNF;
 ParserFailureData m_TestValues;


public:
 CParameterParserFailures();
 ~CParameterParserFailures();


 CString GetErrString(const CString& strString1, const CString& strString2);
};

And the CPP test itself...

///// pertinent information from the .cpp unit TEST file




CParameterParserFailures::CParameterParserFailures()
{
 m_bResult = FALSE;
 m_pExprRoot = NULL;
 m_pFunctionClass = NULL;
 m_pParam = NULL;
 m_ContextConfiguration = NULL;
 m_TestFormula = NULL;
}


CParameterParserFailures::~CParameterParserFailures()
{
 //Nothing to do...
}


void CParameterParserFailures::SetUp()
{
 m_TestValues = GetParam();


 // Initialize the Parameter Expression Parser class
 CParameterExpressionParser::InitializeClass();


 if(NULL == m_ContextConfiguration)
 {
 // Declare a Config Context for the test
 m_ContextConfiguration = new CTestContextConfiguration1;
 }


 // Scoping brackets to ensure Config Context object is constructed before Formula object
 {
 // Declare a Formula for the test
 m_TestFormula = new CTestGroup1(INSTANCE,_T("GROUP1"));


 m_pFUNCTIONCLASS = m_ContextConfiguration->GetFUNCTIONCLASS(_T("FUNCTIONCLASS2"));
 EXPECT_TRUE(NULL != m_pFUNCTIONCLASS) << _T("Unable to locate Function class FUNCTIONCLASS2");


 if (NULL != m_pFUNCTIONCLASS)
 {
 m_pParam = m_pFUNCTIONCLASS->GetFunctionParameterByInternalName ( m_TestValues.m_Point );
 EXPECT_TRUE(NULL != m_pParam) << _T("Unable to locate parameter INTEGERPARAM");


 }
 }
}


void CParameterParserFailures::TearDown()
{
 if (NULL != m_pExprRoot)
 {
 delete m_pExprRoot;
 m_pExprRoot = NULL;
 }


 if(NULL != m_TestFormula)
 {
 delete m_TestFormula;
 m_TestFormula = NULL;
 }


 if(NULL != m_ContextConfiguration)
 {
 delete m_ContextConfiguration;
 m_ContextConfiguration = NULL;
 }
}


CString CParameterParserFailures::GetErrString(const CString& strString1, const CString& strString2)
{
 return strString1 + strString2;
}


TEST_P(CParameterParserFailures, ParseFailure)
{
 CString strErrMsg;
 CParameterExpressionParser Parser(&m_strLNF, // Storage location for LNF translation
  NULL,
  m_TestFormula,
  NULL, // Pointer to runtime procedure
  m_pParam); // Pointer to Parameter


 m_bResult = Parser.Parse( m_TestValues.m_strExprssn, m_pExprRoot );
 EXPECT_FALSE(m_bResult) << GetErrString(_T("Parser did not report failure: Exprssn: "),m_TestValues.m_strExprssn) ;
 
 if (TRUE == m_bResult)
 {
 EXPECT_TRUE(NULL == m_pExprRoot) << GetErrString(_T("Parser did not report failure and returned non-NULL root node pointer. Exprssn:"),m_TestValues.m_strExprssn);
 }
}


ParserFailureData Failures[] = {  
 //Target Point, Expression
 {_T("INTEGERPARAM"), _T("99.9")}, //ParseIntegerParamRefsReal
 {_T("INTEGERPARAM"), _T("2/3+9")}, //ParseIntegerParamTestDividePlusOperatorPrecedence
 {_T("INTEGERPARAM"), _T("FUNCTIONCLASS1:1.INTEGERPARAMETER + 3.2")},
 {_T("INTEGERPARAM"), _T("FUNCTIONCLASS1:1.INTEGERPARAMETER + FUNCTIONCLASS1:1.REALPARAMETER")},
 {_T("INTEGERPARAM"), _T("7 + (FUNCTIONCLASS1:1.INTEGERPARAMETER + 3.2)")},
 {_T("INTEGERPARAM"), _T("7 + (FUNCTIONCLASS1:1.INTEGERPARAMETER + FUNCTIONCLASS1:1.REALPARAMETER)")},
 {_T("INTEGERPARAM"), _T("7 - (FUNCTIONCLASS1:1.INTEGERPARAMETER + 3.2)")},
 {_T("INTEGERPARAM"), _T("-3.2")},
 {_T("INTEGERPARAM"), _T("-FUNCTIONCLASS1:1.REALPARAMETER")},
 {_T("INTEGERPARAM"), _T("TRNC FUNCTIONCLASS1:1.REALPARAMETER)")},
 {_T("INTEGERPARAM"), _T("ABS(TRNC(RND(RDUP FUNCTIONCLASS1:1.REALPARAMETER))))")},
 {_T("INTEGERPARAM"), _T("ABS(FUNCTIONCLASS1:1.REALPARAMETER FUNCTIONCLASS1:1.REALREPORT)")},
 {_T("INTEGERPARAM"), _T("ABS(34.6)")} ,
 {_T("INTEGERPARAM"), _T("5 MOD")}, //MOD Missing trailing operand
 {_T("INTEGERPARAM"), _T("5.5 MOD 4")}, //MOD Real into Integer Point MisMatch
 {_T("INTEGERPARAM"), _T("TRUNC(FUNCTIONCLASS1:1.REALPARAMETER) MOD")}, //MOD missing trailing operand
 {_T("INTEGERPARAM"), _T("FUNCTIONCLASS1:1.STRINGPARAMETER MOD FUNCTIONCLASS1:1.INTEGERPARAMETER")}, //MOD invalid operand type
 {_T("REALPARAM"), _T("FUNCTIONCLASS1:1.REALPARAMETER MOD FUNCTIONCLASS1:1.INTEGERPARAMETER")} //MOD Real into REAL Point MisMatch
 };


INSTANTIATE_TEST_CASE_P(ParameterizedFailure,
 CParameterParserFailures,
 testing::ValuesIn(Failures) );

[Zhanyong Wan (λx.x x)]

These are not warnings.  They are comments that tell you what value the parameter has in each case.  Since gtest doesn't know how to print your custom type (ParserFailureData), it dumps the bytes in the object.  You probably want to teach gtest how to print this type -- see https://code.google.com/p/googletest/wiki/AdvancedGuide#Teaching_Google_Test_How_to_Print_Your_Values for how.

How do Visual Studio's Test Explorer and Code Coverage tools discover tests?

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--

Zhanyong

[Kim Gentes]

Thanks for the reply. Yes, this was the issue. As it turns out, once I overloaded the PrintTo method for all the struct/obj types that were being passed through to the Parameterized tests instantiated with INSTANTIATE_TEST_CASE_P  then the gtest parameter --gtest_list_tests properly reports the tests as succeeding and outputs their appropriate class/method heirarchy.

As I guessed , it is this same switch which is being used by MS Visual Studio to discover the tests present in the unit text executable. If you do not fix this issue, your unit tests will not be discovered by the MS Visual Studio test explorer and the code coverage tools because the gtestrunner adapter is not pushing up the information from the unit tests in the expected fashion.

Also, it should be noted that constant arrays of CString will also run into this issue, when that array is used as the parameter into INSTANTIATE_TEST_CASE_P  . This is confusing because I would think GTEST would be able to recognize that type. However, it does not. The fix for this is not as obvious. I have to create a struct that included the single property of the CString, and write a PrintTo method for it.  Kind of a strange thing, but the base type of CString would not be overridden successfully for the PrintTo method, so I had to resort to this to get it all to work. 

[Zhanyong Wan (λx.x x)]

On Wed, Mar 11, 2015 at 12:28 AM, Kim Gentes <k...@kimgentes.com> wrote:

Thanks for the reply. Yes, this was the issue. As it turns out, once I overloaded the PrintTo method for all the struct/obj types that were being passed through to the Parameterized tests instantiated with INSTANTIATE_TEST_CASE_P  then the gtest parameter --gtest_list_tests properly reports the tests as succeeding and outputs their appropriate class/method heirarchy.

To clarify, I think --gtest_list_tests should always properly report the tests, with or without your PrintTo() definition.  The difference is that with the custom PrintTo(), the output is prettier and more human-friendly.

 

As I guessed , it is this same switch which is being used by MS Visual Studio to discover the tests present in the unit text executable. If you do not fix this issue, your unit tests will not be discovered by the MS Visual Studio test explorer and the code coverage tools because the gtestrunner adapter is not pushing up the information from the unit tests in the expected fashion.

Then it seems a bug in Visual Studio to me.  If it tries to use the --gtest_list_tests flag, it should know to filter out the comments in the output.

 

Also, it should be noted that constant arrays of CString will also run into this issue, when that array is used as the parameter into INSTANTIATE_TEST_CASE_P  . This is confusing because I would think GTEST would be able to recognize that type. However, it does not. The fix for this is not as obvious. I have to create a struct that included the single property of the CString, and write a PrintTo method for it.  Kind of a strange thing, but the base type of CString would not be overridden successfully for the PrintTo method, so I had to resort to this to get it all to work. 

I don't understand this.  Please post concrete code to show what you are doing and what errors you were getting.  Thanks, 

These are not warnings.  They are comments that tell you what value the parameter has in each case.  Since gtest doesn't know how to print your custom type (ParserFailureData), it dumps the bytes in the object.  You probably want to teach gtest how to print this type -- see https://code.google.com/p/googletest/wiki/AdvancedGuide#Teaching_Google_Test_How_to_Print_Your_Values for how.

How do Visual Studio's Test Explorer and Code Coverage tools discover tests?

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--

Zhanyong