Regarding Test Case Name with Underscore (_)

[劉至中]

Dear GoogleTest Group Member,

I have a question regarding test case naming in GoogleTest,
Recently, I start to read the sample of GoogleTest, and I found a interested thing in the following source:
http://code.google.com/p/googletest/source/browse/trunk/samples/sample1_unittest.cc

The line 65 and 66 says:
The test case name and the test name should both be valid C++ identifiers.
And you should not use underscore (_) in the names.

Is it a strict rule that I must follow?
I known underscore seems a key word in GoogleTest and it also used in disabled case via the key word DISABLED_
but I have had many test cases that contains underscore in function name and works well,
I am confusing with that, could you please give me some suggestion?
Thank you very much.

Best regards.
Tom Liu

[Joey Oravec]

On Wed, Oct 13, 2010 at 11:18 PM, 劉至中 <liuc...@gmail.com> wrote:

The line 65 and 66 says:
The test case name and the test name should both be valid C++ identifiers.
And you should not use underscore (_) in the names.

Is it a strict rule that I must follow?

 

I don't know for sure, but I looked through the gtest code this morning and I suspect that's an incorrect comment. When you define a TEST() it's going to create a class with this macro defined in gtest-internal.h:

 

#define GTEST_TEST_CLASS_NAME_(test_case_name, test_name) \

  test_case_name##_##test_name##_Test 

 

Of course that restricts you to characters that would be valid in a C++ identifier. The only other restriction I could imagine is if gtest tried to parse names and used underscore as a separater token. But -- I didn't see any evidence of that in gtest.cc around FilterTests().

 

So I didn't see any obvious reason why underscores are off-limits.

 

-joey

[Vlad Losev]

Hi Tom and Joey,

This restriction was put in place to afford us some flexibility in implementing tests. One day we may decide to change the implementation and rely on this assumption. Than the user code that doesn't conform to it may end up broken. So please try keeping your test case names free of underscores. The DISABLED_ test case names are a supported exception to that rule. Please also note that you are free to use underscores in test names, e.g.

TEST(FooTest, FooMethod_WorksAsExpected) {...}

is fully supported.

What's not good is that this rule is not reflected well enough in Google Test's documentation. I will look into articulating it there.

Regards,
Vlad

[Tom]

Hi Joey and Vlad,

Thank you very much for your reply.

Hi Vlad,

Do you mean that we shuold not use underscore in "test case name" but
we can use it in "test name"?
For example:
1. TEST(FooTest , FooMethod_InvalidArgument_WorksAsExpected) {...}
2. TEST(Foo_Test, FooMethod_InvalidArgument_WorksAsExpected) {...}

The item 1 is fully supported by GoogleTest,
However, we should not name test case like item 2.
Is that correct?

On the other hand, does it rule also apply to test fixture?
For example:
3. TEST_F(CFooTest , FooMethod_InvalidArgument_WorksAsExpected) {...}
4. TEST_F(C_Foo_Test, FooMethod_InvalidArgument_WorksAsExpected) {...}

The item 3 is fully supported by GoogleTest,
However, we should not name test case like item 4.
Is that correct?

Best regards.
Tom Liu

On 10月15日, 上午2時07分, Vlad Losev <v...@losev.com> wrote:
> Hi Tom and Joey,
>
> This restriction was put in place to afford us some flexibility in
> implementing tests. One day we may decide to change the implementation and
> rely on this assumption. Than the user code that doesn't conform to it may
> end up broken. So please try keeping your test case names free of
> underscores. The DISABLED_ test case names are a supported exception to that
> rule. Please also note that you are free to use underscores in test names,
> e.g.
>
> TEST(FooTest, FooMethod_WorksAsExpected) {...}
>
> is fully supported.
>
> What's not good is that this rule is not reflected well enough in Google
> Test's documentation. I will look into articulating it there.
>
>
>

> On Thu, Oct 14, 2010 at 7:31 AM, Joey Oravec <joeyora...@gmail.com> wrote:

> > On Wed, Oct 13, 2010 at 11:18 PM, 劉至中 <liucc...@gmail.com> wrote:
>
> >> The line 65 and 66 says:
> >> The test case name and the test name should both be valid C++ identifiers.

> >> *And you should not use underscore (_) in the names*.

[Vlad Losev]

On Thu, Oct 14, 2010 at 11:10 PM, Tom <liuc...@gmail.com> wrote:

Hi Joey and Vlad,

Thank you very much for your reply.

Hi Vlad,

Do you mean that we shuold not use underscore in "test case name" but
we can use it in "test name"?
For example:
1. TEST(FooTest , FooMethod_InvalidArgument_WorksAsExpected) {...}
2. TEST(Foo_Test, FooMethod_InvalidArgument_WorksAsExpected) {...}

The item 1 is fully supported by GoogleTest,
However, we should not name test case like item 2.
Is that correct?

Correct 

On the other hand, does it rule also apply to test fixture?
For example:
3. TEST_F(CFooTest  , FooMethod_InvalidArgument_WorksAsExpected) {...}
4. TEST_F(C_Foo_Test, FooMethod_InvalidArgument_WorksAsExpected) {...}

The item 3 is fully supported by GoogleTest,
However, we should not name test case like item 4.
Is that correct?

Correct. It's the same for fixtures. 

[Tom]

Hi Vlad,

Thank you very much for your reply,
This rule is ok for me,
Currently I only use underscore in "test name", not in "test case" or
"test fixture".
I think it should be updated to formal document for future reference.
Thank you for your quick reply again.

Best regards.
Tom Liu

On 10月15日, 下午2時15分, Vlad Losev <v...@losev.com> wrote:

[Zhanyong Wan (λx.x x)]

The comments are intended.

Underscore (_) is special, as C++ reserves the following to be used by
the compiler and the standard library:

1. any identifier that starts with an _ followed by an upper-case letter, and
2. any identifier that containers two consecutive underscores (i.e.
__) *anywhere* in its name.

User code is prohibited from using such identifiers.

Now let's look at what this means for TEST and TEST_F.

Currently TEST(TestCaseName, TestName) generates a class named
TestCaseName_TestName_Test. What happens if TestCaseName or TestName
contains _?

1. If TestCaseName starts with an _ followed by an upper-case letter
(say, _Foo), we end up with _Foo_TestName_Test, which is reserved and
thus invalid.
2. If TestCaseName ends with an _ (say, Foo_), we get
Foo__TestName_Test, which is invalid.
3. If TestName starts with an _ (say, _Bar), we get
TestCaseName__TestName_Test, which is invalid.
4. If TestName ends with an _ (say, Bar_), we get
TestCaseName_TestName__Test, which is invalid.

So clearly TestCaseName and TestName cannot start or end with _
(Actually, TestCaseName can start with _ -- as long as the _ isn't
followed by an upper-case letter. But that's getting complicated. So
for simplicity we just say that it cannot start with _.).

It may seem fine for TestCaseName and TestName to contain _ in the
middle. However, consider this:

TEST(Time, Flies_Like_An_Arrow) { ... }
TEST(Time_Flies, Like_An_Arrow) { ... }

Now, the two TESTs will both generate the same class
(Time_Files_Like_An_Arrow_Test). That's not good.

So for simplicity, we just ask the users to avoid _ in TestCaseName
and TestName. The rule is more constraining than necessary, but it's
simple and easy to remember. It also gives Google Test some wiggle
room in case its implementation needs to change in the future.

If you violate the rule, there may not be immediately consequences,
but your test may (just may) break with a new compiler (or a new
version of the compiler you are using) or with a new version of Google
Test. Therefore it's best to follow the rule.

--
Zhanyong

[Zhanyong Wan (λx.x x)]

I've update the primer and the FAQ with this rule and the explanation. Cheers,

--
Zhanyong

[Piotrek Martynowicz]

Hi Zhanyong, sorry for reviving an old thread but I need you to clarify something. You wrote that undescores are fine in test names but not in test case names. But then you said (and the current gtest documentation states so) that both test names and test case names shouldn't contain underscores. So which one is it then?

[Zhanyong Wan (λx.x x)]

Hi Piotrek,

On Mon, Feb 8, 2016 at 1:47 AM, Piotrek Martynowicz <the.re...@gmail.com> wrote:

Hi Zhanyong, sorry for reviving an old thread but I need you to clarify something. You wrote that undescores are fine in test names but not in test case names.

I don't remember saying that.  Do you have the original message?

 

But then you said (and the current gtest documentation states so) that both test names and test case names shouldn't contain underscores. So which one is it then?

That's correct.  Please follow the current documentation. 

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--

Zhanyong

[Piotrek Martynowicz]

Oh, wow, I'm sorry, it was Vlad Losev that wrote that, not you. He wrote in this thread that underscores are fine in test names but not in test case names. So I understand that what he wrote is not correct?

Second question, if I'm already using underscores in test case names in my project and they work fine do I really have to change all those names and remove the underscores? What's the risk of something breaking (with future google test releases, I assume)?

[Zhanyong Wan (λx.x x)]

On Mon, Feb 8, 2016 at 11:41 AM, Piotrek Martynowicz <the.re...@gmail.com> wrote:

Oh, wow, I'm sorry, it was Vlad Losev that wrote that, not you. He wrote in this thread that underscores are fine in test names but not in test case names. So I understand that what he wrote is not correct?

Your understanding is correct. 

Second question, if I'm already using underscores in test case names in my project and they work fine do I really have to change all those names and remove the underscores? What's the risk of something breaking (with future google test releases, I assume)?

It's up to you.  I'm no longer actively involved with gtest, so I'll let the current owners answer how likely future gtest releases will break this. 

W dniu czwartek, 14 października 2010 05:18:50 UTC+2 użytkownik Tom napisał:

Dear GoogleTest Group Member,

I have a question regarding test case naming in GoogleTest,
Recently, I start to read the sample of GoogleTest, and I found a interested thing in the following source:
http://code.google.com/p/googletest/source/browse/trunk/samples/sample1_unittest.cc

The line 65 and 66 says:
The test case name and the test name should both be valid C++ identifiers.
And you should not use underscore (_) in the names.

Is it a strict rule that I must follow?
I known underscore seems a key word in GoogleTest and it also used in disabled case via the key word DISABLED_
but I have had many test cases that contains underscore in function name and works well,
I am confusing with that, could you please give me some suggestion?
Thank you very much.

Best regards.
Tom Liu

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--

Zhanyong

[Piotrek Martynowicz]

Thank you, Zhanyong. Btw I was just reading docs/Primer.md and noticed that in their example code they're using underscores as well:

TEST_F(test_case_name, test_name) {
 ... test body ...
}

Well, actually this just presents the syntax but still I don't think it should use underscores if they're discouraged from actual names.

W dniu czwartek, 14 października 2010 05:18:50 UTC+2 użytkownik Tom napisał:

[Zhanyong Wan (λx.x x)]

On Tue, Feb 9, 2016 at 7:59 AM, Piotrek Martynowicz <the.re...@gmail.com> wrote:

Thank you, Zhanyong. Btw I was just reading docs/Primer.md and noticed that in their example code they're using underscores as well:

TEST_F(test_case_name, test_name) {
 ... test body ...
}

Well, actually this just presents the syntax but still I don't think it should use underscores if they're discouraged from actual names.

I think these are meant as meta variables, but I agree that it can be misleading.

Sam, what do you say to changing this example to

TEST_F(TestCaseName, TestName)

?

W dniu czwartek, 14 października 2010 05:18:50 UTC+2 użytkownik Tom napisał:

Dear GoogleTest Group Member,

I have a question regarding test case naming in GoogleTest,
Recently, I start to read the sample of GoogleTest, and I found a interested thing in the following source:
http://code.google.com/p/googletest/source/browse/trunk/samples/sample1_unittest.cc

The line 65 and 66 says:
The test case name and the test name should both be valid C++ identifiers.
And you should not use underscore (_) in the names.

Is it a strict rule that I must follow?
I known underscore seems a key word in GoogleTest and it also used in disabled case via the key word DISABLED_
but I have had many test cases that contains underscore in function name and works well,
I am confusing with that, could you please give me some suggestion?
Thank you very much.

Best regards.
Tom Liu

--

---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--

Zhanyong

[slymz]

Sorry to join in on this old thread, but this restriction is rather unfortunate: it introduces special cases and exceptions when adopting naming conventions in the user space (how to name tests for a class with and without underscores, etc.) where underscores in public names are very common. 

In other words, which one is more confusing, a or b?

a) The following which fails right away while I am coding it in:

TEST(X_Y, Z ){}
TEST(X, Y_Z ){}  // <-- error: 'X_Y_Z_Test' : 'class' type redefinition

b) The following code (which I'd written eons ago) failing in some future version of Google Test, during an infrastructure upgrade, by a downstream team:

TEST( my_fancy_class_test, bla ){}

Instead of the sledgehammer "no-underscore-anywhere", have the following less restrictive alternatives been considered?

1. Actually restricting that  `test_case_name_test_name` to be unique.  First, the no-underscore-anywhere restriction isn't enforced by the framework (looks like an open issue, but at best, can be a runtime enforcement), and judging from the user discussions, there are already plenty of code out there using the "_" without knowing this peculiar restriction.  Second, and more importantly, the restriction that "test_case_name_test_name" be unique, is already enforced by the compiler, at compile time. The error message is not terribly confusing: "class type redefinition".  Clearly TEST() is macro that does name mangling, and it does it in a certain way that what has been just declared clashed. One can easily change it to something else and move on. That's it.

2. Mangling as something like `test_case_name_GTEST_test_name`, and restricting that test suite names can not contain `_GTEST_` (which is much less restrictive than `_`)

3. Implement same behavior with a scope `test_case_name::test_name` rather than a single namespace scope class. (I don't know the code base, so I apologize if this is technically impossible)

On the other related concern that a name must conform to standard (no double underscores or no leading underscore before capital case), this obviously doesn't translate to the restriction that underscores disallowed everywhere (as the FAQ also indicates).  But again, from a frequency-of-unintended-misuse perspective, it is much less (*way less*) common to have leading or trailing underscores in public symbols of a user's facility (former being already a non-google-test issue that the user needs to address), compared to valid uses of underscores, (2) certain compilers already warn when there is a violation, and (3) what exactly is wrong with the framework documenting the restriction that "no leading or trailing underscore in suite names" rather than the overkill no-underscore-anywhere?

Another curious point, quoting the maintainer from 2010 (thread above):

>> This restriction was put in place to afford us some flexibility in implementing [Google Test]. One day we may decide to change the implementation and rely on this assumption

The framework is already brilliant as it is. What is this future potential wiggle room for? Has there actually been a feature or implementation requirement that demanded this assumption be enforced?