TAP ain't "Test All Perl"
Ovid
There's been a fair amount of discussion recently about how to extend TAP, but there's something noticeably absent from this discussion: Java programmers. Or PHP programmers. Or C, or Python, or any other language you can to choose. Personally, I know of TAP implementations written in Perl, C, JavaScript, Python and PHP. I'm sure there are more.
I propose that even though the Perl community has largely been the driving force behind TAP, a new TAP mailing list should be created. This list should be language agnostic and we should try and pull in the major testing players from a variety of different languages. Some of our discussion has been very language-centric and by pulling in people from other communities, I think we can gain a few benefits.
1. Different ways of looking at the problems we face.
2. Wider exposure and adoption of TAP.
3. As some need a break the discussion (like unwinding from OSCON), others are there to keep the flames lit.
My personal thought is that either Andy Lester or Schwern start such a mailing list. Then we need to contact the major players from other languages and get them to join and see how we can make TAP a more widely accepted protocol. We know the benefits of it. Now I recommend that we convince others of those benefits so we can "share the wealth", so to speak.
If anyone likes this idea, I've been accepted to attend Google's Automated Testing Conference in London (http://googleresearch.blogspot.com/2006/04/our-conference-on-automated-testing.html) and I can see what I can do to recruit people. (I was notified of my acceptance considerably later than they said I would be, so it's possible my work might interfere).
Cheers,
Ovid
-- Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Adrian Howard
[snip]
> My personal thought is that either Andy Lester or Schwern start
> such a mailing list. Then we need to contact the major players
> from other languages and get them to join and see how we can make
> TAP a more widely accepted protocol. We know the benefits of it.
> Now I recommend that we convince others of those benefits so we can
> "share the wealth", so to speak.
To be honest, I'm not entirely sure that I could come up with a
really convincing argument for TAP for an audience using JUnit,
TestNG and similar...
Do we have a convincing argument? Maybe I'm just being overly
pessimistic today :-)
> If anyone likes this idea, I've been accepted to attend Google's
> Automated Testing Conference in London (http://
> googleresearch.blogspot.com/2006/04/our-conference-on-automated-
> testing.html) and I can see what I can do to recruit people. (I was
> notified of my acceptance considerably later than they said I would
> be, so it's possible my work might interfere).
Bah. They didn't like me :-)
Adrian
Leon Brocard
> If anyone likes this idea, I've been accepted to attend Google's Automated Testing Conference in London
I shall also be there and shall also speak to people about TAP. I
think about 3 other London.pm-ers will also be there. Thing is, I only
use TAP with Perl. I'll try and play with it in other places before
the conference ;-)
Otherwise, sounds like a good idea. Getting the major players involved
would be the key point. Maybe they are doing things we should be doing
too...
Leon
Michael G Schwern
> To be honest, I'm not entirely sure that I could come up with a
> really convincing argument for TAP for an audience using JUnit,
> TestNG and similar...
>
> Do we have a convincing argument? Maybe I'm just being overly
> pessimistic today :-)
The separation of the display code from the tests themselves is a
bonus. Though one would have to check with folks who actually use
XUnit to be sure this is a problem, I can imagine cases where the test
author used some crappy XUnit implementation and the display of the
tests sucks (its ambiguous, missing information, doesn't provide
enough information, only graphical, etc). And there's little you can
do about that since the test and the display code are bound.
Its also handy for testing over a wire (David Wheeler's Javascript TAP
implementation, for example) or and stream. Testing a running
production program continuously is something I've been wanting to do
for a while (Test::AtRuntime).
There's also the advantage of it being cross-platform for those
projects which involve multiple languages (Java and PHP maybe?)
Finally, XUnit can be coerced into outputting TAP.
Trouble is at the moment all this is still in the prototype stage.
And none of them are killer.
Michael Peters
Michael G Schwern wrote:
> On 8/15/06, Adrian Howard <adr...@quietstars.com> wrote:
>> To be honest, I'm not entirely sure that I could come up with a
>> really convincing argument for TAP for an audience using JUnit,
>> TestNG and similar...
>>
>> Do we have a convincing argument? Maybe I'm just being overly
>> pessimistic today :-)
>
> The separation of the display code from the tests themselves is a
> bonus. Though one would have to check with folks who actually use
> XUnit to be sure this is a problem, I can imagine cases where the test
> author used some crappy XUnit implementation and the display of the
> tests sucks (its ambiguous, missing information, doesn't provide
> enough information, only graphical, etc). And there's little you can
> do about that since the test and the display code are bound.
I think this is the most important benefit. Test runs aren't bound to the single
time it was run with a certain harness. They can be captured and viewed later by
the same developer or shared among developers. If the report is descriptive
enough (catching STDERR, etc) then it can be a great tool for debugging and
would go hand-in-hand with a bug tracker. Imagine getting a bug report of your
test suite failing along with the full TAP output of the test run.
[snip]
> There's also the advantage of it being cross-platform for those
> projects which involve multiple languages (Java and PHP maybe?)
>
> Finally, XUnit can be coerced into outputting TAP.
>
> Trouble is at the moment all this is still in the prototype stage.
> And none of them are killer.
I'm actually hopeful that Smolder (http://sourceforge.net/projects/smolder) or
something like it will show how useful abstracting out the test output from the
harness can be. I'm planning on doing a 1.0 release of it pretty soon if
anyone's interested.
--
Michael Peters
Developer
Plus Three, LP
Ovid
From: Michael Peters <mpe...@plusthree.com>
> Imagine getting a bug report of your
> test suite failing along with the full TAP output of the test run.
That needs to be handled carefully. Currently there's no problem with TAP parsing an infinite stream (Test::AtRuntime).
Ovid
From: Michael G Schwern <sch...@gmail.com>
> Trouble is at the moment all this is still in the prototype stage.
> And none of them are killer.
Is there something wrong with my TAPx::Parser? So far, I think it's the closest implementation we have to the TAP::Harness which has been proposed. It needs some work, but I need to know what people want.
Ovid
From: Ovid <publiuste...@yahoo.com>
> Is there something wrong with my TAPx::Parser?
And as a side note, if anyone likes it and thinks it can be *the* TAP::Harness and has the power to bless that change, I'll happily turn it over.
Joe McMahon
On Aug 15, 2006, at 11:29 AM, Michael Peters wrote:
> I think this is the most important benefit. Test runs aren't bound
> to the single
> time it was run with a certain harness. They can be captured and
> viewed later by
> the same developer or shared among developers. If the report is
> descriptive
> enough (catching STDERR, etc) then it can be a great tool for
> debugging and
> would go hand-in-hand with a bug tracker. Imagine getting a bug
> report of your
> test suite failing along with the full TAP output of the test run.
(App::SimpleScan for the interested). One of its plugins is a
"snapshot" function. When a test fails, simple_scan takes a snapshot
and dumps a diag into the TAP stream. It's easy to post-process this
so that the resulting file can be viewed in a browser with hotlinks
to the snapshots. Our developers like it a lot.
--- Joe M.
Fergal Daly
> On 14 Aug 2006, at 21:53, Ovid wrote:
> [snip]
> > My personal thought is that either Andy Lester or Schwern start
> > such a mailing list. Then we need to contact the major players
> > from other languages and get them to join and see how we can make
> > TAP a more widely accepted protocol. We know the benefits of it.
> > Now I recommend that we convince others of those benefits so we can
> > "share the wealth", so to speak.
>
> To be honest, I'm not entirely sure that I could come up with a
> really convincing argument for TAP for an audience using JUnit,
> TestNG and similar...
XUnit allows heirarchical grouping, TAP does not. DUnit (Delphi's
XUnit) comes with a GUI that shows you a tree of tests/groups,
F
Ovid
From: Fergal Daly <fer...@esatclear.ie>
> XUnit allows heirarchical grouping, TAP does not. DUnit (Delphi's
> XUnit) comes with a GUI that shows you a tree of tests/groups,
This is the sort of input that other's outside of the Perl community might be able to give due to their being familiar with the ins and outs of it.
Side note: TAPx::Parser should easily allow a GUI interface. However, it will be tough to implement until I can get tprove_color (incuded with the distribution) to work. It should produce roughly the same output at tprove_color2. Once tprove_color works, adding a GUI should be trivial.
Collin Winter
> ----- Original Message ----
> From: Fergal Daly <fer...@esatclear.ie>
>
> > XUnit allows heirarchical grouping, TAP does not. DUnit (Delphi's
> > XUnit) comes with a GUI that shows you a tree of tests/groups,
>
> This is the sort of input that other's outside of the Perl community might be able to
> give due to their being familiar with the ins and outs of it.
Python's unittest (a derivative of JUnit) allows you to group related
tests (each test case is a method on a class) and then combine these
groups (via multiple inheritance). However, once composed, the methods
inherited from the parent classes appear in the test output as if they
had been defined on the child class directly. A quick illustration:
>>> class TestLength(unittest.TestCase):
>>> def test_empty(test):
>>> test.assertEqual(len(test.empty), 0)
>>>
>>> # Some other common len()-based tests
>>>
>>> class TestDict(TestLength):
>>> def setUp(test):
>>> test.empty = dict()
>>>
>>> # Some other tests for dict()
>>>
>>> class TestList(TestLength):
>>> def setUp(test);
>>> test.empty = list()
>>>
>>> # Some other tests for list()
When TestDict and TestList were run, they would both run a
test_empty() method; in the standard text output mode, this would show
up as TestDict.test_empty() and TestList.test_empty(), respectively.
All that to say: there'd be no problem integrating a TAP emitter as an
extension to Python's common testing framework. In fact, coding up
such a thing has been on my todo list for a week or two now.
Collin Winter
Fergal Daly
I'm actualy talking about TestSuite objects. Here's some Python
import unittest
class ATest(unittest.TestCase):
def testFooBar(self):
self.assertEquals(foo(), bar())
def testOther(self):
...
class AnotherTest(unittest.TestCase):
def testBuz(self):
...
suite1 = unittest.TestSuite()
suite2.addTests(ATest, AnotherTest)
suite2 = unittest.TestSuite()
suit2.addTests(...)
all_suites = unittest.TestSuite()
all_suites.addTest(suite1, suite2, SomeOtherTest)
all_suites.run()
So all_suites is a TestSuite containing 2 other suites and a test,
both of the suites contain other tests.
So the structure is like this:
all_suites
suite1
ATest
testFooBar
testOther
AnotherTest
testBuz
suite2
...
SomeOtherTest
...
TAP cannot represent this nested structure.
Just as an aside, allowing this kind of nested structure (with nested
plans) solves many of the plan problems that have been brought up
recently. Last time I brought it up I was told YAGNI but if you want
TAP to be used for XUnit then you are going to need it,
F
Collin Winter
> I'm actualy talking about TestSuite objects. Here's some Python
>
>
> So the structure is like this:
>
> all_suites
> suite1
> ATest
> testFooBar
> testOther
> AnotherTest
> testBuz
> suite2
> ...
> SomeOtherTest
> ...
>
> TAP cannot represent this nested structure.
FWIW, I've never come across a situation using unittest where knowing
the structure would be important (though I can imagine such cases).
The cases I've seen have all been along the lines of "I need to
collect all the tests in my test suite, so I'll pull all tests from my
test modules into a suite (or suite-of-suites)". In my experience, the
more important hierarchical information is of the form
$module_name.$test_case_name.$test_method_name, which TAP can handle.
What has your experience been?
Collin Winter
Ovid
From: Collin Winter <col...@gmail.com>
> FWIW, I've never come across a situation using unittest where knowing
> the structure would be important (though I can imagine such cases).
I remember when I first encountered recursion back in the early 80s. Almost all of my programming at the time was in BASIC or assembler nad I couldn't fathom how recursion would be useful to me.
In reality, you never *need* recursion, but sometimes it can be helpful and is often the simplest way of writing things out. If we are going to reach out to other programming communities, we need to understand that even if we disagree with their way of doing things, it doesn't hurt to support those ways so long as it doesn't interfere with other ways of doing things.
Cheers,
Ovid
Fergal Daly
I don't want to get involved in another lengthy argument but this is
not recursion. Recursion involves a function that eventually calls
itself. The example I gave is just nesting, that's just the equivalent
of subroutines. I can't think of an application of recursion in this
situation (I haven't thought very hard!), all I'm asking for is
subroutines (grouping and subgrouping),
F
Fergal Daly
Where this becomes more important is when you start constructing
suites automatically. For example
class SomeTest(unittest.TestCase):
def __init__(self, data):
self.data = data
unittest.TestCase.__init__(self)
def test_foo(self):
# some tests on self.data
def test_bar(self):
# some tests on self.data
list_of_data = [ ... ]
suite = unittest.TestSuite()
for data in list_of_data:
suite.addTest( SomeTest(data) )
suite.run()
now my suite looks like
suite
SomeTest
SomeTest
...
SomeTest
line numbers are no good to me. I haven't ever done this in Python but
I've done the equivalent in Perl. Basically as soon as you want to
have tests that are driven by data sets rather than tests that are all
explicitly written you need structure. It's also necessary for code
reuse.
Finally, it's very nice to see a GUI version where you can expand and
collapse bits of the test tree,
F
Collin Winter
The solution to this particular issue would be to have your SomeTest
class override TestCase.id(), TestCase.__str__() and/or
TestCase.__repr__() to take into account the data attribute.
On the other hand, I can see scenarios where being able to indicate
which suite a test is in, for example, tests that depend on being run in a
certain order.
Collin Winter
Ovid
From: Fergal Daly <fer...@esatclear.ie>
> I don't want to get involved in another lengthy argument but this is
> not recursion.
Sorry if I was unclear. I was only using recursion as an example of something one might see but, if one hadn't used it, not seeing the merit of it. I wasn't trying to imply that what you were describing was recursive in nature.
Fergal Daly
That's necessary but in TAP terms only tests have names whereas here,
each TestCase is actually a group of tests, so TestCase.__str__ has no
place in the TAP output.
F
Fergal Daly
> On 8/15/06, Fergal Daly <fer...@esatclear.ie> wrote:
> > On 15/08/06, Collin Winter <col...@gmail.com> wrote:
> > > The solution to this particular issue would be to have your SomeTest
> > > class override TestCase.id(), TestCase.__str__() and/or
> > > TestCase.__repr__() to take into account the data attribute.
> >
> > That's necessary but in TAP terms only tests have names whereas here,
> > each TestCase is actually a group of tests, so TestCase.__str__ has no
> > place in the TAP output.
>
> *instance* is only a single test, meaning that
> ``str(TestCase('somemethod_name'))`` would invoke TestCase.__str__().
In the python world that's true but for Perl programmers looking at this:
class SomeTest(unittest.TestCase):
def some_test(self):
self.assertEquals(foo(), bar())
self.assertEquals(buz(), qux())
each of those self.assertEquals is a test because it can pass or fail,
Each of those would generate a line of TAP output and possibly some
diagnostics,
F
>
> Collin Winter
>
Collin Winter
> On 15/08/06, Collin Winter <col...@gmail.com> wrote:
> > The solution to this particular issue would be to have your SomeTest
> > class override TestCase.id(), TestCase.__str__() and/or
> > TestCase.__repr__() to take into account the data attribute.
>
> That's necessary but in TAP terms only tests have names whereas here,
> each TestCase is actually a group of tests, so TestCase.__str__ has no
> place in the TAP output.
The TestCase *classes* are groups of tests, but each TestCase
*instance* is only a single test, meaning that
``str(TestCase('somemethod_name'))`` would invoke TestCase.__str__().
Collin Winter
Collin Winter
Ah, yes, quite right. Most of my recent testing has been in Python, so
the details of perl's testing system aren't quite as fresh in my mind
as they used to be.
On the other hand, one thing I do recall clearly is that when I
started doing a lot of testing in Python, I thought it was stupid that
things like ``self.assertEquals(foo(), bar())`` didn't count as
"tests" like they did in perl/TAP.
Collin Winter
Ovid
From: Collin Winter <col...@gmail.com>
> On the other hand, one thing I do recall clearly is that when I
> started doing a lot of testing in Python, I thought it was stupid that
> things like ``self.assertEquals(foo(), bar())`` didn't count as
> "tests" like they did in perl/TAP.
That threw me off when I first encountered jUnit. For much of the testing world, a 'test' is a collection of one or more asserts which verifies that a particular feature behaves the way that it should. So the following from Test::Class will confuse folks not used to Perl:
sub foo : Tests(3) {
my $self = shift;
my $object = $self->class->new;
can_ok $object, 'foo';
ok $object->foo('bar'), '... and it should accept "bar"';
is $object->something, 'something else',
'... and it should properly update the object state';
}
For most of the testing world I've seen outside of Perl, that would be a single test. Terminology will be a barrier.
Cheers,
Ovid
Adrian Howard
On 15 Aug 2006, at 19:14, Michael G Schwern wrote:
> On 8/15/06, Adrian Howard <adr...@quietstars.com> wrote:
>> To be honest, I'm not entirely sure that I could come up with a
>> really convincing argument for TAP for an audience using JUnit,
>> TestNG and similar...
>>
>> Do we have a convincing argument? Maybe I'm just being overly
>> pessimistic today :-)
[excellent reasons snipped]
> Trouble is at the moment all this is still in the prototype stage.
> And none of them are killer.
Yup. While their all good reasons (and why I like TAP) they don't
give any real killer advantage to somebody already using a neato
xUnit framework that's nicely integrated with their IDE.
Adrian
Adrian Howard
On 15 Aug 2006, at 19:14, Michael G Schwern wrote:
[snip]
> The separation of the display code from the tests themselves is a
> bonus. Though one would have to check with folks who actually use
> XUnit to be sure this is a problem
[snip]
I don't think it is. In things like JUnit and TestNG the test runner
is logically separated (it's a different class, defined APIs, etc.)
Cheers,
Adrian
Adrian Howard
On 15 Aug 2006, at 19:29, Michael Peters wrote:
[snip]
> I think this is the most important benefit. Test runs aren't bound
> to the single
> time it was run with a certain harness. They can be captured and
> viewed later by
> the same developer or shared among developers.
Nothing stopping you doing this with JUnit/TestNG/whatever either.
You don't have a protocol, but you do have a standard set of class
based APIs. Not that hard to stick a proxy/recorder type thang in the
middle.
What TAP might give you is the idea of a "standard" though -
something to make it easier for different folk with different tools
to more easily have a common language..
> I'm actually hopeful that Smolder (http://sourceforge.net/projects/
> smolder) or
> something like it will show how useful abstracting out the test
> output from the
> harness can be. I'm planning on doing a 1.0 release of it pretty
> soon if
> anyone's interested.
Me! Me! I've already had a brief play and am considering using it at
$WORK.
Adrian
Adrian Howard
On 15 Aug 2006, at 21:31, Fergal Daly wrote:
[snip]
> XUnit allows heirarchical grouping, TAP does not. DUnit (Delphi's
> XUnit) comes with a GUI that shows you a tree of tests/groups,
It's worse than that actually. These days things like TestNG allow
you to cut up tests into arbitrary groups across the class hierarchy.
You end up with a graph / faceted classification system rather than a
simple hierarchy.
Adrian
Adrian Howard
On 15 Aug 2006, at 22:08, Fergal Daly wrote:
[snip]
> Just as an aside, allowing this kind of nested structure (with nested
> plans) solves many of the plan problems that have been brought up
> recently. Last time I brought it up I was told YAGNI but if you want
> TAP to be used for XUnit then you are going to need it,
You saying Test::Class isn't an xUnit framework ;-)
Adrian
Adrian Howard
On 15 Aug 2006, at 22:19, Collin Winter wrote:
[snip]
> In my experience, the
> more important hierarchical information is of the form
> $module_name.$test_case_name.$test_method_name, which TAP can handle.
How?
Adrian
Adrian Howard
On 15 Aug 2006, at 23:39, Ovid wrote:
[snip]
> That threw me off when I first encountered jUnit. For much of the
> testing world, a 'test' is a collection of one or more asserts
> which verifies that a particular feature behaves the way that it
> should. So the following from Test::Class will confuse folks not
> used to Perl:
>
> sub foo : Tests(3) {
> my $self = shift;
> my $object = $self->class->new;
> can_ok $object, 'foo';
> ok $object->foo('bar'), '... and it should accept "bar"';
> is $object->something, 'something else',
> '... and it should properly update the object state';
> }
>
> For most of the testing world I've seen outside of Perl, that would
> be a single test. Terminology will be a barrier.
Yup. With T::C I tried to always use the phrase "test method" for
what the xUnit folk would normally call a "test" (although the
mapping isn't exact).
Adrian
Fergal Daly
Coure not!
Let me rephrase. If you want TAP to be the protocol which communicates
the test result to the test result analyser then a flat list of tests
throws away too much information,
F
Adrian Howard
On 16 Aug 2006, at 14:45, Geoffrey Young wrote:
[snip]
> I dunno, but I think the ability to
> integrate like this is something that would make a killer feature if
> folks figured out how to leverage (and market) it.
The problem is that the test runners for some frameworks are "cooler"
than Test::Harness ("pretty" GUIs, integrated with all major IDEs,
etc.) so you've got a bit of a hill to climb before you can sell the
extra benefits a "standard" based system might give you.
Adrian (apparently being Mr Pessimistic today - sorry ;)
Geoffrey Young
> Trouble is at the moment all this is still in the prototype stage.
> And none of them are killer.
supporting TAP means you can integrate with Test::Harness. now, I know
that might not seem like much, but we've got quite the number of mature
testing tools over in perl-land that are pretty cool.
for example, because nearly all the popular PHP testing tools now
support TAP (SimpleTest, PHPUnit) PHP test-minded folks don't have to
change what they are doing to take advantage of, for example,
Apache-Test. the combination of both means they can write their tests
the way they always have, but (painlessly) run them in the
httpd-embedded PHP instead of in the CLI... without a browser and
aggregated on the command line, just like they're used to.
so, what else is there in that might work similarly, extending other
languages with cool perl tools? I dunno, but I think the ability to
integrate like this is something that would make a killer feature if
folks figured out how to leverage (and market) it.
--Geoff
Geoffrey Young
>
> On 16 Aug 2006, at 14:45, Geoffrey Young wrote:
> [snip]
>
>> I dunno, but I think the ability to
>> integrate like this is something that would make a killer feature if
>> folks figured out how to leverage (and market) it.
>
>
> The problem is that the test runners for some frameworks are "cooler"
> than Test::Harness ("pretty" GUIs, integrated with all major IDEs,
> etc.) so you've got a bit of a hill to climb before you can sell the
> extra benefits a "standard" based system might give you.
yeah, well that's kinda their problem for me - if perl only came with
one of these cooler interfaces, with pretty colors that I had to
eye-scan I might not be testing as much. now give me something I can
stick in crontab and look at the failure report tomorrow (if anything
fails) and we're all set ;)
but sure, I see your point. in fact, people have made the argument that
Test::Harness integration is actually a stumbling block for PHP testing,
since most PHP developers live in win32 world and only have open a text
editor and their browser (and I guess an ftp client or something), with
no unix-type interface available most of the time. whether this is true
or not, that someone said it would seem to indicate that, yeah, the
command line isn't all that great a tool if you're not already there
using it.
--Geoff
A. Pagaltzis
> On 15 Aug 2006, at 19:29, Michael Peters wrote:
> >I think this is the most important benefit. Test runs aren't
> >bound to the single time it was run with a certain harness.
> >They can be captured and viewed later by the same developer or
> >shared among developers.
>
> Nothing stopping you doing this with JUnit/TestNG/whatever
> either. You don't have a protocol, but you do have a standard
> set of class based APIs.
I don’t know about you, but I tend to prefer a well-defined and
simple wire formats over APIs. :-) In a sense, it’s declarative,
rather than imperative. Integration of heterogenous systems is
much easier that way – even subsystems for which no explicit
format emitter is available can participate without much trouble.
You can easily scale the complexity of the participants at either
end of the wire to match your needs. At one extreme of the scale,
you might telnet to your smoke server and manually tap in (pun
intended) “ok 1”, “not ok 2” responses. Then you can go through
scripts which manually `print "ok 1"` through Test::More all the
way to Test::Class. Your infrastructure can be as simple or
complex as you want or need.
I think this is a powerful benefit.
> Not that hard to stick a proxy/recorder type thang in the
> middle.
Yes, but you have to make extra effort and possibly hack a
little. With TAP, you get that for free. As the name implies,
you can test everything, and with minimum fuss.
Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>
Adrian Howard
On 16 Aug 2006, at 19:38, A. Pagaltzis wrote:
[snip]
> I don’t know about you, but I tend to prefer a well-defined and
> simple wire formats over APIs. :-) In a sense, it’s declarative,
> rather than imperative. Integration of heterogenous systems is
> much easier that way – even subsystems for which no explicit
> format emitter is available can participate without much trouble.
> You can easily scale the complexity of the participants at either
> end of the wire to match your needs. At one extreme of the scale,
> you might telnet to your smoke server and manually tap in (pun
> intended) “ok 1”, “not ok 2” responses. Then you can go through
> scripts which manually `print "ok 1"` through Test::More all the
> way to Test::Class. Your infrastructure can be as simple or
> complex as you want or need.
>
> I think this is a powerful benefit.
You're right it can be a bonus. But to sell that to a group of people
who already have something that works very well, is already
integrated with their IDE, continuous integration package, build
system, etc. you've got a tough job ;-)
Adrian
Fergal Daly
When I was using DUnit, I was doing lots of pointer twiddling (serious
fun in Dephi). If the library I was testing was broken enough it would
crash, taking the whole test GUI with it. I really missed that aspect
of TAP,
F
A. Pagaltzis
> Integration of heterogenous systems is much easier that way –
> even subsystems for which no explicit format emitter is
> available can participate without much trouble. You can easily
> scale the complexity of the participants at either end of the
> wire to match your needs.
Case in point: a rudimentary TAP library for Oracle PL/SQL.
http://use.perl.org/~jdavidb/journal/30641
* Adrian Howard <adr...@quietstars.com> [2006-08-17 09:55]:
> You're right it can be a bonus. But to sell that to a group of
> people who already have something that works very well, is
> already integrated with their IDE, continuous integration
> package, build system, etc. you've got a tough job ;-)
I think it’s likely to be a hard sell precisely because it’s bad
in the ways that IDE-integrated stuff is good, but very good in
the ways that IDE-integrated stuff is bad. Look at the example
above – how much effort would it take you to hook testing for
your SQL scripts into your IDE, compared to the effort it took J.
David B. to write a pretty featureful TAP emitter? He didn’t even
have to write most of that code, 1/3 of it would have been enough
for a primitive library, which is sufficient to hook into
Test::Harness – or whatever other TAP infrastructure we’ll cook
up in the future.
This is exactly the kind of thing I was espousing in my other
mail.
Of course, the problem here is one of world view. A heavily IDE-
centric developer isn’t likely to think in these terms in the
first place.
Perl, OTOH, has a cultural bias towards heterogenous environments
and tool independence. It’s easy to imagine why TAP sprung up
here.
I guess the best marketing opportunity would be to target the
developers who have to deal with at least somewhat heterogenous
systems despite their use of typical IDE+xUnit languages –
they’re the most likely to be feeling the sort of pain from their
current tool that will make them receptive to the benefits of
TAP.
Adrian Howard
On 14 Aug 2006, at 21:53, Ovid wrote:
[snip]
> If anyone likes this idea, I've been accepted to attend Google's
> Automated Testing Conference in London (http://
> googleresearch.blogspot.com/2006/04/our-conference-on-automated-
> testing.html) and I can see what I can do to recruit people. (I was
> notified of my acceptance considerably later than they said I would
> be, so it's possible my work might interfere).
[snip]
If anybody has some spare time
might also be a good place to pimp TAP.
Adrian
Adrian Howard
On 20 Aug 2006, at 17:13, A. Pagaltzis wrote:
[snip]
> * Adrian Howard <adr...@quietstars.com> [2006-08-17 09:55]:
>> You're right it can be a bonus. But to sell that to a group of
>> people who already have something that works very well, is
>> already integrated with their IDE, continuous integration
>> package, build system, etc. you've got a tough job ;-)
>
> I think it’s likely to be a hard sell precisely because it’s bad
> in the ways that IDE-integrated stuff is good,
Of course this issue is nothing to do with any intrinsic property of
TAP. There's nothing stopping TAP being integrated much more tightly
- there just needs to be somebody with that itch to scratch.
[snip]
> I guess the best marketing opportunity would be to target the
> developers who have to deal with at least somewhat heterogenous
> systems despite their use of typical IDE+xUnit languages –
> they’re the most likely to be feeling the sort of pain from their
> current tool that will make them receptive to the benefits of
> TAP.
[snip]
Yup. That sound like a potentially itchy group ;-)
Cheers,
Adrian
Ovid
From: Adrian Howard <adr...@quietstars.com>
> Of course this issue is nothing to do with any intrinsic property of
> TAP. There's nothing stopping TAP being integrated much more tightly
> - there just needs to be somebody with that itch to scratch.
Except that there needs to be some movement on TAP again. There was a flurry of enthusiasm, upgrading the perl-qa wiki with plans for TAP::Harness, talks about subsequent versions of TAP, but then it died. Hell, I have the core in place to move on this, if only I could get an official blessing on it. Having heard nothing, I just gave up.
Cheers,
Ovid
Adrian Howard
I'm not sure who's the "official" here? :-)
Adrian
Ovid
I don't know that anyone is 'official', but I suspect that most in the community who have any idea about perl's qa stuff would probably look to Schwern and Andy. They're kind of the keeper of the keys on the core modules here. I'd like to see more work done on this, but I've not really received feedback (I don't think Shlomi has either), even though, as mentioned, I've done most of the work necessary to put a TAP::Harness in place.
In short, many of the gripes of the Perl community about the known limitations of our testing framework could be mitigated if can get momentum on this again. However, OSCON just ended, Schwern has a new job, Andy has other (good) projects that he's pursuing and work here has stalled. I can't say I blame them because I certainly have bursts of energy followed by periods of needing to recharge, but I don't know what to do. If I've spent a lot of time building something that is deemed unsuitable for the future direction of TAP and the testing framework, I really don't want to continue working on it. If it *is* suitable but needs some work, I'd like to know that, too.
Cheers,
Ovid
Chromatic
> If I were you, I'd assume that silence was a blessing and push on.
I don't know how this blessing might sound, but I do know how the opposite
sounds, and it's much noisier.
-- c
David E. Wheeler
> If I've spent a lot of time building something that is deemed
> unsuitable for the future direction of TAP and the testing
> framework, I really don't want to continue working on it. If it
> *is* suitable but needs some work, I'd like to know that, too.
If I were you, I'd assume that silence was a blessing and push on. If
it's important enough to you, what will happen is that *you'll*
create the momentum and become the keeper of the keys.
So trust your instincts. Do what *you* think is right.
Must my $0.02.
Best,
David
Adrian Howard
[snip]
> If I've spent a lot of time building something that is deemed
> unsuitable for the future direction of TAP and the testing
> framework, I really don't want to continue working on it.
Is it useful to you?
> If it *is* suitable but needs some work, I'd like to know that, too.
For what it's worth: I think what you've done so far is great :-)
The next time I need to fiddle with test harnesses and TAP parsing
I'll be looking at it closely. Until then though I'm not sure that
I've got anything useful to contribute.
Primarily because I'm a lazy SOB and prefer to spend my free time
carousing rather than expending what few mental facilities I have
remaining on difficult things like software design.
The other reason is that I get very nervous designing stuff without a
solid "I need to do Foo". I've found that I really suck at designing
infrastructure up-front, and find it much more effective to extract
infrastructure out of at least semi-working code. That's just my kink.
So if it is useful to you I'd say carry on - I'm sure other folk will
join in when they encounter similar itches. If it isn't, maybe let it
rest a little until it will be directly useful to you.
(desperately hoping this doesn't sound mean :-)
Adrian
Shlomi Fish
I'd like to make use of TAPx::Parser in Test::Run. Another thing I'd like to
do is extract all the non-core functionality from the main Test::Run module
into plugins. I did not completely forget it, just was busy.
Speaking of tests, I wrote Test::Count, and placed it on the CPAN:
http://search.cpan.org/dist/Test-Count/
Its purpose is to keep track of the number of tests in a test script, by
having meta-comments in the code.
Regards,
Shlomi Fish
---------------------------------------------------------------------
Shlomi Fish shl...@iglu.org.il
Homepage: http://www.shlomifish.org/
Chuck Norris wrote a complete Perl 6 implementation in a day but then
destroyed all evidence with his bare hands, so no one will know his secrets.
Joe McMahon
> In short, many of the gripes of the Perl community about the known
> limitations of our testing framework could be mitigated if can get
> momentum on this again. However, OSCON just ended, Schwern has a
> new job, Andy has other (good) projects that he's pursuing and work
> here has stalled. I can't say I blame them because I certainly
> have bursts of energy followed by periods of needing to recharge,
> but I don't know what to do. If I've spent a lot of time building
> something that is deemed unsuitable for the future direction of TAP
> and the testing framework, I really don't want to continue working
> on it. If it *is* suitable but needs some work, I'd like to know
> that, too.
Ovid, I currently need a good TAP parser, and I'm settling on yours
as *my* standard, anyway. It would seem futile to write yet another
one when I've got yours to work with. I will certainly put on my user
hat and start sending you "couldn't it just" mail as soon as
possible. :)
--- Joe M.
A. Pagaltzis
> I've found that I really suck at designing infrastructure
> up-front, and find it much more effective to extract
> infrastructure out of at least semi-working code. That's just
> my kink.
Not just yours; I’m in the same boat. I’m watching this TAP stuff
with great interest but I don’t actually have an acute itch to
scratch so I’m not doing any work that would make me feel
qualified to opine on specific issues.
> (desperately hoping this doesn't sound mean :-)
Didn’t sound so to me.
Ovid
From: Joe McMahon <mcm...@ibiblio.org>
> Ovid, I currently need a good TAP parser, and I'm settling on yours
> as *my* standard, anyway. It would seem futile to write yet another
> one when I've got yours to work with. I will certainly put on my user
> hat and start sending you "couldn't it just" mail as soon as
> possible. :)
Thank you! That's exactly the sort of feedback I need.
Also, if anyone can figure out why tprove_color fails with "Can't dup STDOUT" and tprove_color2 works, I'd be grateful. The former uses callbacks to cleanly output results and the latter is procedural (the former is somewhat procedural, bit I can't refactor until I can nail down the bug). I don't think there *should* be a reason why the former fails as the two are functionally equivalent. I'm wondering if it's some compile time thing as the former has the print statements wrapped in anonymous subroutines, but that doesn't seem right :/
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Adrian Howard
On 21 Aug 2006, at 19:52, Ovid wrote:
> ----- Original Message ----
> From: Joe McMahon <mcm...@ibiblio.org>
>
>> Ovid, I currently need a good TAP parser, and I'm settling on yours
>> as *my* standard, anyway. It would seem futile to write yet another
>> one when I've got yours to work with. I will certainly put on my user
>> hat and start sending you "couldn't it just" mail as soon as
>> possible. :)
>
> Thank you! That's exactly the sort of feedback I need.
[snip]
A thought just occurred.... anybody fancy doing a mini hackathon on
TAPish stuff at YAPC::EU?
Assuming we can think of some nice practical goals :-)
Adrian
Ovid
From: Shlomi Fish <shl...@iglu.org.il>
> I'd like to make use of TAPx::Parser in Test::Run. Another thing I'd like to
> do is extract all the non-core functionality from the main Test::Run module
> into plugins. I did not completely forget it, just was busy.
Thanks! Let me know if there's anything you need changed which can support this.
> Speaking of tests, I wrote Test::Count, and placed it on the CPAN:
>
> http://search.cpan.org/dist/Test-Count/
>
> Its purpose is to keep track of the number of tests in a test script, by
> having meta-comments in the code.
No offense, but I can't tell from the docs how to use this.
Cheers,
Ovid
Ovid
From: Adrian Howard
> Is it useful to you?
> For what it's worth: I think what you've done so far is great :-)
> Primarily because I'm a lazy SOB and prefer to spend my free time
> carousing rather than expending what few mental facilities I have
> remaining on difficult things like software design.
> (desperately hoping this doesn't sound mean :-)
Cheers,
Ovid
Ovid
From: Adrian Howard <adr...@quietstars.com>
> A thought just occurred.... anybody fancy doing a mini hackathon on
> TAPish stuff at YAPC::EU?
Sold! I have some issues I need to work on and TAPx::Parser needs some help, too. We should also post this to our use.perl journals to sucker more folks in.
Cheers,
Ovid
Shlomi Fish
> ----- Original Message ----
> From: Shlomi Fish <shl...@iglu.org.il>
>
> > I'd like to make use of TAPx::Parser in Test::Run. Another thing I'd like
> > to do is extract all the non-core functionality from the main Test::Run
> > module into plugins. I did not completely forget it, just was busy.
>
> Thanks! Let me know if there's anything you need changed which can support
> this.
Sure.
>
> > Speaking of tests, I wrote Test::Count, and placed it on the CPAN:
> >
> > http://search.cpan.org/dist/Test-Count/
> >
> > Its purpose is to keep track of the number of tests in a test script, by
> > having meta-comments in the code.
>
> No offense, but I can't tell from the docs how to use this.
>
This has to be remedied, I guess.
Quick primer:
"# TEST*[[EXPR]]" add [[EXPR]] tests to the total. "# TEST+[[EXPR]]" does the
same.
"# TEST:$VAR1=[[EXPR]];$VAR2=[[EXPR]];" are variable assignments. You can
later use these variables in subsequent lines.
Note that the space after the "#" is significant and "TEST" is always
uppercase. But I'll write this in the POD.
Regards,
Shlomi Fish
> Cheers,
> Ovid
--
Adrian Howard
Like anybody reads my use.perl journal :-)
I've stuck a page on the YAPC::EU wiki at <http://
wiki.birmingham2006.com/wiki.pl?TapHackathon>
Adrian.
A. Pagaltzis
> Like anybody reads my use.perl journal :-)
There is a bunch of people who read all the use.perl journals,
courtesy of <http://use.perl.org/search.pl?op=journals>…
Nik Clayton
> * Adrian Howard <adr...@quietstars.com> [2006-08-22 12:25]:
>> Like anybody reads my use.perl journal :-)
>
> There is a bunch of people who read all the use.perl journals,
Indeed.
/me wishes for a full contents RSS feed.
A. Pagaltzis
> A. Pagaltzis wrote:
> >There is a bunch of people who read all the use.perl journals,
>
> Indeed.
>
> /me wishes for a full contents RSS feed.
So do I. I had a scraper running at one point which would combine
the fulltext from individual journals’ feeds into the sitewide
feed, but it was very rude, downloading up to 30 feeds at every
hour with no support for conditional GETs. Eventually the IP got
blocked[1], though I still don’t know if it was on purpose. It
was unblocked a while ago so I plan to put up a new scraped
fulltext feed, but this time I’ll first make sure to support the
appropriate bandwidth reduction measures before I put it back in
place.
[1]: http://use.perl.org/~Aristotle/journal/28731
David Cantrell
> * Nik Clayton <n...@ngo.org.uk> [2006-08-22 16:50]:
>>A. Pagaltzis wrote:
>>>There is a bunch of people who read all the use.perl journals,
>>>courtesy of <http://use.perl.org/search.pl?op=journals>
> So do I. I had a scraper running at one point which would combine
> the fulltext from individual journals’ feeds into the sitewide
> feed, but it was very rude, downloading up to 30 feeds at every
> hour with no support for conditional GETs. Eventually the IP got
> blocked[1]
Scraping 30 journals every hour does seem somewhat excessive. I fetch
45 use.perl journals once a day using rss2email and no-one's complained.
There's an RSS link embedded in the headers of pages, but weirdly it
doesn't seem to include journals, nor is there any obvious link I can
see for the specific journal RSS feeds. You probably want
http://use.perl.org/~username/journal/rss
--
David Cantrell
Smylers
> Like anybody reads my use.perl journal :-)
Apparently I do -- I saw your journal entry in Bloglines shortly after
reading the above message.
I seeded my Bloglines subscriptions with those on Planet Perl, so maybe
that's why I'm following your feed? If you are on Planet Perl then
probably quite a few folks are reading your output.
Smylers
A. Pagaltzis
> You probably want http://use.perl.org/~username/journal/rss
Actually, <http://use.perl.org/~username/journal/atom> – as well
as <http://use.perl.org/search.pl?op=journals;content_type=atom>.
The scraper pulls the latter and then iterates over the entries,
pulling individual journal feeds to pluck out fulltext and
transplant it, so I end up with the same feed that I get from
<http://use.perl.org/search.pl?op=journals;content_type=atom>,
only with fulltext included.
It’s not necessary to poll every hour actually. I used to do that
because originally I didn’t know about that hub feed and used a
different feed to scrape:
<http://use.perl.org/journal.pl?op=top;content_type=atom>. That
one only lists journals, not individual entries, so you need to
poll very frequently to avoid missing posts.
Additionally, the next version of my scraper will have a cache so
it can send the appropriate `If-Modified-Since` headers on
requests, so that for feeds I’ve already seen the entire req/resp
cycle takes all of 300 bytes or so instead of pulling down the
whole 50KB shebang every time for no reason at all.
(Yeah, I was really rude. I am embarrassed in retrospect.)
Nik Clayton
> * David Cantrell <d.can...@outcometechnologies.com> [2006-08-23 09:35]:
>> You probably want http://use.perl.org/~username/journal/rss
>
> as <http://use.perl.org/search.pl?op=journals;content_type=atom>.
> The scraper pulls the latter and then iterates over the entries,
> pulling individual journal feeds to pluck out fulltext and
> transplant it, so I end up with the same feed that I get from
> <http://use.perl.org/search.pl?op=journals;content_type=atom>,
> only with fulltext included.
Sounds similar to something I got 70% of the way through writing.
N