[test-more-users] Re: Issue 52 in test-more: test name containing # preceded by unexpected \ character

[codesite...@google.com]

Updates:
Status: Invalid

Comment #1 on issue 52 by schwern: test name containing # preceded by
unexpected \ character
http://code.google.com/p/test-more/issues/detail?id=52

Thanks for your report.

This is expected behavior, # is a reserved TAP character. Directives such
as "skip"
and "todo" follow it. Consider the following:

ok( $test, "name # skip" );

If we didn't escape it, it would come out as:

ok 1 - name # skip

Which is TAP for a skipped test.

note() and diag() do not require this escaping since they are already in a
comment.
Everything after the initial # in the output is ignored by the TAP parser.

I'm going to close this as "invalid" because its the expected behavior.
Please feel
free to reopen it if you want to discuss it further.

[codesite...@google.com]

Comment #2 on issue 52 by rwfranks: test name containing # preceded by
unexpected \ character
http://code.google.com/p/test-more/issues/detail?id=52

Michael,

IMHO it seems deeply unsatisfactory to mix up control/status information
and test
description.

According to the following snippet from the testanything wiki,
unrecognisable
directives (# followed by strange stuff) should be treated as an ordinary
comment.
My fault report was based upon this expectation.

Dick
--


Ignored elements

In order to allow extension of the protocol while maintaining backwards
compatibility, a TAP consumer must ignore certain lines and elements. Any
line which
does not parse must be ignored. Any directive or plan-directive which is not
recognized must also be ignored.

Here's an example of a TAP stream which contains ignored elements.

1..2
Error at line 12
ok 1
ok 2 # BANG

Line 2 should be ignored, it does not parse. The directive "BANG" on test
#2 should
be treated as an ordinary comment, it is not a TAP directive.

[codesite...@google.com]

Comment #3 on issue 52 by schwern: test name containing # preceded by
unexpected \ character
http://code.google.com/p/test-more/issues/detail?id=52

On 2010.5.22 2:13 PM, codesite...@google.com wrote:
> IMHO it seems deeply unsatisfactory to mix up control/status information
> and test description.

There is no mix up, # clearly separates the two.

If you're referring to having them on the same line at all, well, there
isn't a whole
lot to be done about that. TAP is a one line per test protocol.

> According to the following snippet from the testanything wiki,
> unrecognisable
> directives (# followed by strange stuff) should be treated as an
> ordinary comment.
> My fault report was based upon this expectation.

Yes, this is true. But there's no guarantee that the bit after the # in a
test
description isn't a valid directive. Test::More could escape the # if and
only if
there's a valid directive after it, but that bit of complexity is
unnecessary as
there's nothing wrong with escaping the #. In addition, what is not a
directive now
might be a directive later. Then an unescaped # in a test name will be
mistaken for
that directive. This protects test writers from future changes to TAP.

Furthermore, an unescaped # in a test name will be parsed as an unknown
directive
rather than a test name. For example, consider not escaping # in test
names. ok( 1,
"foo # bar" ) would produce...

ok 1 - foo # bar

Which will be parsed as:

number: 1
ok: 1
description: foo
directive: bar

Which was not the intent of the author of the test. They meant:

number: 1
ok: 1
description: foo # bar

Test::More escapes the # in the test name means so test will be parsed as
intended by
the author. It relieves the test author from having to know the details of
TAP.

What is true is that there is no official TAP escape character and
backwhack is a
Test::More convention. Something was needed, it seemed sensible. Then
again, there
is still no complete TAP spec and its still largely negotiation between
Test::Builder
and TAP::Harness.

Was this causing a problem, or is it just something you noticed?

[codesite...@google.com]

Comment #4 on issue 52 by rwfranks: test name containing # preceded by
unexpected \ character
http://code.google.com/p/test-more/issues/detail?id=52

This arose when testing escape substitutions and identifying each test by
interpolating the character sequence into the test description. This gets
confusing
if gratuitous '\'s appear in the test report.

However, when repeating the test a few minutes ago, I found that the
problem has gone
away!

# trivial.t
use Test::More 0.94;
plan tests => 2;

ok( 1, 'test name containing # character' );
ok( 1, 'test name containing \# sequence' );

now produces:

$ perl trivial.t
1..2
ok 1 - test name containing # character
ok 2 - test name containing \# sequence

This works for me.

[codesite...@google.com]

Comment #5 on issue 52 by schwern: test name containing # preceded by
unexpected \ character
http://code.google.com/p/test-more/issues/detail?id=52

> This arose when testing escape substitutions and identifying each test by
> interpolating the character sequence into the test description. This gets
> confusing
> if gratuitous '\'s appear in the test report.

What are you using to interpret the test output? I don't know if
Test::Harness
understands the escaping.

> However, when repeating the test a few minutes ago, I found that the
> problem
> has gone away!

I do not get that result and no change was made to the escaping. Something
is wrong
or you're using an altered version. Can you tell me what operating system
and
version of that OS you're using (for example, Fedora 13) and if you're
using the
system installed version?

[codesite...@google.com]

Comment #6 on issue 52 by rwfranks: test name containing # preceded by
unexpected \ character
http://code.google.com/p/test-more/issues/detail?id=52

TAP::Parser::Grammar.pm has SKIP|TODO hard-coded in a regex at line 145. It
will fail
to match # <random thing>, in which case the entire line is treated as the
test name.

Escaping the # only makes a difference for \#SKIP or \#TODO which I guess
would only
arise when testing test programs and is of little interest to other users.

I overcame my problem by deleting line 750 from a copy of Test::Builder.pm.

The modified version got back into the @INC path accidentally. I am using
Fedora13
and think @INC may have changed since Fedora12. My mistake, I should have
checked the
result more carefully.

My considered view remains that deleting line 750 would be an improvement
for most
users. The penalty is some minor inconvenience for those wishing to use
#SKIP as part
of a test name (that is, not as a directive).

[codesite...@google.com]

Comment #7 on issue 52 by schwern: test name containing # preceded by
unexpected \ character
http://code.google.com/p/test-more/issues/detail?id=52

> TAP::Parser::Grammar.pm has SKIP|TODO hard-coded in a regex at line 145.
> It
> will fail to match # <random thing>, in which case the entire line is
> treated
> as the test name.

Test::Builder does not write to the quirks of TAP::Parser but to the TAP
specification. We deliberately decouple Test::Builder from TAP::Parser.
TAP::Parser's quirks today are not its quirks tomorrow. The above behavior
can
arguably be called a bug, the spec isn't entirely clear on that point, but
it could
have just as easily not hard coded them. The TAP::Parser folks tend to do
a very
strict reading of the spec.

In addition, TAP is a specification which can be extended. Escaping the #
is a
deliberate feature to guard against new TAP directives (skip and todo are
directives). SKIP was added in 1997 and TODO was added in 2001.

> I overcame my problem by deleting line 750 from a copy of
> Test::Builder.pm.

> ...

> My considered view remains that deleting line 750 would be an improvement
> for most users. The penalty is some minor inconvenience for those wishing
> to use #SKIP as part of a test name (that is, not as a directive).

Most users don't even know this feature exists, and that is as it should be.
Test::Builder shields users from the details of TAP, especially the little
things
that could silently cause tests to do weird things. That is far more
important than
an aesthetically displeasing backslash.

It hasn't been made clear why the escape is a problem. I asked earlier
what you're
using to parse the TAP. Is it TAP::Parser, or Test::Harness, or did you
write your
own parser? Or are you eyeballing it? Please show the code which is
affected.

There is one minor bug here, it is that the # is escaped in the comment as
well.

# Failed test 'test name containing \# character'

should technically be

# Failed test 'test name containing # character'