Test Suite: unified ".t" vs. separated "test/.out/" scripts

[David Carr]

Is there a particular reason that the test suite is currently using separate files for test scripts and their expected output?  Personally, I find the unified ".t" format, where the test script and its expected output are stored in a single file, to be much easier to read, understand, and update.  In particular, it's easier to see which command is resulting in which output without having to inject additional echo statements or something similar.  We might also be able to eliminate some of the sed expressions by using the unified test format's support for filtering outputs with regular expressions.

It looks like it should be easy to upgrade the copy of run-tests.py, and then possible to convert the existing tests one at a time to the new format.  If there's interest, I'd be happy to assemble a series of patches in this direction.

[Augie Fackler]

On Aug 22, 2012, at 11:44 PM, David Carr wrote:

> Is there a particular reason that the test suite is currently using
> separate files for test scripts and their expected output? Personally, I
> find the unified ".t" format, where the test script and its expected output
> are stored in a single file, to be much easier to read, understand, and
> update. In particular, it's easier to see which command is resulting in
> which output without having to inject additional echo statements or
> something similar. We might also be able to eliminate some of the sed
> expressions by using the unified test format's support for filtering
> outputs with regular expressions.

.t files are the modern solution, but the way we perform all-version-tests means that any test that needs to run before .t was introduced in run-tests.py needs to stay non-unified. That might be the right justification to drop older versions of hg.

[backlog = high, still planning to review patches]

>
> It looks like it should be easy to upgrade the copy of run-tests.py, and
> then possible to convert the existing tests one at a time to the new
> format. If there's interest, I'd be happy to assemble a series of patches
> in this direction.
>

> --
> You received this message because you are subscribed to the Google Groups "hg-git" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/hg-git/-/4Vnw3c230boJ.
> To post to this group, send email to hg-...@googlegroups.com.
> To unsubscribe from this group, send email to hg-git+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/hg-git?hl=en.
>

[David M. Carr]

I was under the impression that since the copy of run-tests.py that we use is in our codebase (rather than depending on the one in Hg's codebase), it wouldn't be affected by how we do all-version-tests. I'll test it to verify. 

--
David M. Carr
davi...@gmail.com

[Augie Fackler]

On Aug 25, 2012, at 8:18 PM, David M. Carr wrote:

I was under the impression that since the copy of run-tests.py that we use
is in our codebase (rather than depending on the one in Hg's codebase), it
wouldn't be affected by how we do all-version-tests. I'll test it to
verify.

That's true for 'make tests', but not 'make tests-${version}' IIRC

[David M. Carr]

> --
> You received this message because you are subscribed to the Google Groups
> "hg-git" group.
> To post to this group, send email to hg-...@googlegroups.com.
> To unsubscribe from this group, send email to
> hg-git+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/hg-git?hl=en.

You are indeed correct. It looks like, of the versions currently
listed in all-version-tests, 1.7.5 is the earliest one that supports
unified tests.

I played around with it a bit, and I think I've got a tweak to the
makefile that allows using our bundled version of run-tests.py with
any version of Mercurial. It feels slightly inelegant, as it involves
temporarily copying the file into the "crew" working copy, but in
practice I think it's unlikely to cause problems. If you want to look
at it, it's in my patch queue (links below).

https://bitbucket.org/davidmc24/hg-git-queue/src/bff7c80c2603/alternate-run-tests
https://bitbucket.org/davidmc24/hg-git-queue/qseries?apply=t&qs_apply=alternate-run-tests

[Augie Fackler]

On Aug 26, 2012, at 12:05 AM, David M. Carr wrote:

You are indeed correct.  It looks like, of the versions currently
listed in all-version-tests, 1.7.5 is the earliest one that supports
unified tests.

I played around with it a bit, and I think I've got a tweak to the
makefile that allows using our bundled version of run-tests.py with
any version of Mercurial.  It feels slightly inelegant, as it involves
temporarily copying the file into the "crew" working copy, but in
practice I think it's unlikely to cause problems.  If you want to look
at it, it's in my patch queue (links below).

https://bitbucket.org/davidmc24/hg-git-queue/src/bff7c80c2603/alternate-run-tests
https://bitbucket.org/davidmc24/hg-git-queue/qseries?apply=t&qs_apply=alternate-run-tests

Cute. I'm not sure it's really worth bothering though, since 1.7.5 is pretty old. Maybe just drop everything before that and unify?

[David M. Carr]

> --
> You received this message because you are subscribed to the Google Groups
> "hg-git" group.
> To post to this group, send email to hg-...@googlegroups.com.
> To unsubscribe from this group, send email to
> hg-git+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/hg-git?hl=en.

Works for me. Once we've worked through the re-work on my currently
submitted patches, I'll move in that direction (removing support for
pre-1.7.5).

[David M. Carr]

It looks like test-outgoing is currently always skipped. Should I
leave that one as-is (non-unified), or do something else with it? I'm
verifying that other tests work as I unify them, but testing that it's
skipped on the first line isn't particularly meaningful.

[Augie Fackler]

On Sep 6, 2012, at 2:51 PM, David M. Carr wrote:

> It looks like test-outgoing is currently always skipped. Should I
> leave that one as-is (non-unified), or do something else with it? I'm
> verifying that other tests work as I unify them, but testing that it's
> skipped on the first line isn't particularly meaningful.

Hm, is it just that we're only supporting versions that don't support outgoing now? If so, let's go ahead and drop the test.

[David M. Carr]

> --
> You received this message because you are subscribed to the Google Groups "hg-git" group.
> To post to this group, send email to hg-...@googlegroups.com.
> To unsubscribe from this group, send email to hg-git+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/hg-git?hl=en.
>

It looks like you added a comment "This feature is currently
completely broken due to changes in dulwich" back in January along
with an "exit 80" for all versions. Maybe I'll see if I can figure
out what's needed to get outgoing working again, at which point the
test could be unified.

[David Carr]

>> It looks like test-outgoing is currently always skipped.  Should I
>> leave that one as-is (non-unified), or do something else with it?  I'm
>> verifying that other tests work as I unify them, but testing that it's
>> skipped on the first line isn't particularly meaningful.
>
> Hm, is it just that we're only supporting versions that don't support outgoing now? If so, let's go ahead and drop the test.

It looks like you added a comment "This feature is currently
completely broken due to changes in dulwich" back in January along
with an "exit 80" for all versions.  Maybe I'll see if I can figure
out what's needed to get outgoing working again, at which point the
test could be unified.

I've got a patch that appears to fix outgoing support ready.  I'll send to the list along with the other patches I've been working on when the patches I've already submitted make it into bitbucket.org/durin42/hg-git.