use diff for string in some test

[Lee Duhem]

I think we need some diff-like facility for string in Test::More
(or somewhere else appropriate), these options of diff for files
will be wanted:

-E --ignore-tab-expansion Ignore changes due to tab expansion.
-b --ignore-space-change Ignore changes in the amount of white space.
-w --ignore-all-space Ignore all white space.
-B --ignore-blank-lines Ignore changes whose lines are all blank.
--strip-trailing-cr Strip trailing carriage return on input.

A few days ago, test t/examples/past.t fails because particle's
whitespace clean up on it, some tailing spaces in expect output
have been removed. If we have this facility and use it, this test
will be passed perfectly after particle's whitespace cleanup.

By the way, some naive implement of this "diff" will be enough for
t/examples/past.t and some other tests of parrot.

Lee

[Nicholas Clark]

On Sun, Jan 07, 2007 at 03:34:41PM +0800, Lee Duhem wrote:
> I think we need some diff-like facility for string in Test::More
> (or somewhere else appropriate), these options of diff for files
> will be wanted:
>
> -E --ignore-tab-expansion Ignore changes due to tab expansion.
> -b --ignore-space-change Ignore changes in the amount of white space.
> -w --ignore-all-space Ignore all white space.
> -B --ignore-blank-lines Ignore changes whose lines are all blank.
> --strip-trailing-cr Strip trailing carriage return on input.
>
>
> A few days ago, test t/examples/past.t fails because particle's
> whitespace clean up on it, some tailing spaces in expect output
> have been removed. If we have this facility and use it, this test
> will be passed perfectly after particle's whitespace cleanup.

Shouldn't that have failed tests before it was checked in?

I'm not convinced that allowing slop in expected output is a good idea.
It introduces (more) complexity into the tests, which increases the chance
for false positives (errors in the tests reporting themselves as failures,
distracting developers) and false negatives (tests not spotting real errors)

Nicholas Clark

[Bernhard Schmalhofer]

Nicholas Clark schrieb:

> On Sun, Jan 07, 2007 at 03:34:41PM +0800, Lee Duhem wrote:
>
>> I think we need some diff-like facility for string in Test::More
>> (or somewhere else appropriate), these options of diff for files
>> will be wanted:
>>
>> -E --ignore-tab-expansion Ignore changes due to tab expansion.
>> -b --ignore-space-change Ignore changes in the amount of white space.
>> -w --ignore-all-space Ignore all white space.
>> -B --ignore-blank-lines Ignore changes whose lines are all blank.
>> --strip-trailing-cr Strip trailing carriage return on input.
>>

> Shouldn't that have failed tests before it was checked in?
>
> I'm not convinced that allowing slop in expected output is a good idea.
> It introduces (more) complexity into the tests, which increases the chance
> for false positives (errors in the tests reporting themselves as failures,
> distracting developers) and false negatives (tests not spotting real errors)
>
>

I think the t/examples/past.t is one of the rare cases where ignoring
whitespace would be useful.
Generated code is allowed to change, as long as it still does the wanted
thing.
However, the only purpose of t/examples/past.t is to check that
examples/past/01-sub.pir
is still working. This can be guaranteed with a regexp-based sanity
check, as it is done now.
So there is no immediate need for action.

I propose to use Test::Differences for other reasons. When comparing
texts with eq_or_diff()
it is much easier to see where the deviation is. It is also much easier
to decide wheter the code
or the test is incorrect.

Regards, Bernhard

[Lee Duhem]

On Sun, Jan 07, 2007 at 12:51:46PM +0000, Nicholas Clark wrote:
>
> I'm not convinced that allowing slop in expected output is a good idea.

Sometime allow some slop is convenient, like the situation in
t/examples/past.t

> It introduces (more) complexity into the tests, which increases the chance
> for false positives (errors in the tests reporting themselves as failures,
> distracting developers) and false negatives (tests not spotting real errors)

If it's abused, yes!

Lee