running test over a set of use cases

16 views

Skip to first unread message

Henry Story

unread,

Feb 28, 2012, 12:18:49 PM2/28/12

to scalate...@googlegroups.com

Hi,

I have a couple of tests that I want to run over a set of input, and find all those inputs that fail.

To give some context I have written a parser using Nomo (an asynchronous parser combinator), and I have a

directory full of files to test my parser against, and resulting files that I can parser with a simpler parser. I want

the tests to report on exactly those tests that failed, and give some reason as to why they failed. Here is

an extract of TurtleParserTest :

  property("The Turtle Parser should parse each of the good files") {
    val base: String = "http://www.w3.org/2001/sw/DataAccess/df1/tests/"
    info("all these files are in "+tstDir)
    val res = for(f <- good) yield {
      val resFileName = f.getName.substring(0,f.getName.length()-"ttl".length())+"out"
      val resultFile = new File(f.getParentFile,resFileName)
      info("testing file "+f.getName+" against result in "+resultFile.getName)
      val otherReading = referenceParser.read(resultFile,base)
      failureOf {
        assert(otherReading.isRight === true, referenceParser + " could not read the " + f + " returned " + otherReading)
        val result = parseTurtleFile(f, base)
        val res = result.user.queue.toList.map(_.asInstanceOf[Triple])
        isomorphicTest(res, otherReading.right.get)
      }
    }
    val errs = res.filter(_!= None)

    assert(errs.size==0,"Not all tests passed. See info for details")
  }

I tried a number of solutions including using Tables but they all stop at the first error.  So I found that I can capture all the 
errors with failureOf. But how do I then go about displaying them nicely? Currently it's all a bit mixed up and difficult to read.
It would be nice if there were a way to set info messages that were error messages. This is because my procedure is to first 
do a general test: find out if the results are isomorphic, and then if that fails, try to find out what the cause may be. So I would
like to collect a number of reasons in each case for why the error could have occurred, 

Here is the current output:


[info] No tests to run for sesame/test:test-only
[info] No tests to run for simple-rdf/test:test-only
[info] SesameTurtleParserSeqTest:
[info] - the Turtle parser should parse the official NTriples test case
[info]   + testing file test.nt at /Volumes/Dev/Programming/w3.org/git/pimp-my-rdf/n3-test-suite/target/scala-2.9.1/classes/www.w3.org/2000/10/rdf-tests/rdfcore/ntriples/test.nt 
[info]   + setting Turtle parser to reading with chunk size of 839 bytes 
[info] - The Turtle Parser should parse each of the good files *** FAILED ***
[info]   Not all tests passed. See info for details (TurtleParserTest.scala:135)
[info]   + all these files are in /Volumes/Dev/Programming/w3.org/git/pimp-my-rdf/n3-test-suite/target/scala-2.9.1/classes/www.w3.org/TR/turtle/tests 
[info]   + testing file base1.ttl against result in base1.out 
[info]   + setting Turtle parser to reading with chunk size of 1487 bytes 
[info]   + testing file base2.ttl against result in base2.out 
[info]   + setting Turtle parser to reading with chunk size of 509 bytes 
[info]   + testing file default.ttl against result in default.out 
[info]   + setting Turtle parser to reading with chunk size of 97 bytes 
[info]   + testing file default1.ttl against result in default1.out 
[info]   + setting Turtle parser to reading with chunk size of 463 bytes 
[info]   + testing file escapedNamespace1.ttl against result in escapedNamespace1.out 
[info]   + setting Turtle parser to reading with chunk size of 1543 bytes 
[info]   + testing file escapedPrefix1.ttl against result in escapedPrefix1.out 
[info]   + setting Turtle parser to reading with chunk size of 1163 bytes 
[info]   + testing file prefix1.ttl against result in prefix1.out 
[info]   + setting Turtle parser to reading with chunk size of 1487 bytes 
[info]   + testing file rdf-schema.ttl against result in rdf-schema.out 
[info]   + setting Turtle parser to reading with chunk size of 709 bytes 
[info]   + graphs were not isomorphic - trying to narrow down on problematic statement 
[info]   + read ntriples file with org.w3.rdf.sesame.TurtleSeqParser$@25410f4b found 124 triples 
[info]   + The number of triples read was 126. The reference number was 124 
[info]   + But perhaps that is due to there being duplicates. The result size with obvious duplicates removed is 124 
[info]   + the last triple read was (http://www.w3.org/2000/01/rdf-schema#, http://www.w3.org/2000/01/rdf-schema#seeAlso, http://www.w3.org/2000/01/rdf-schema-more) 
[info]   + testing file rdfq-results.ttl against result in rdfq-results.out 
[info]   + setting Turtle parser to reading with chunk size of 547 bytes

Bill Venners

unread,

Feb 28, 2012, 12:39:23 PM2/28/12

to scalate...@googlegroups.com

Hi Henry,

Very helpful use case. It feels like a job for a TestFailedException
with a long error message, one for each internal failure, including
perhaps a filename and line number for each. You would not want to see
the successful ones, correct? Just the failed ones in the error
message?

Bill

> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

Henry Story

unread,

Feb 28, 2012, 2:52:25 PM2/28/12

to scalate...@googlegroups.com

On Tuesday, 28 February 2012 18:39:23 UTC+1, Bill Venners wrote:

Hi Henry,
Very helpful use case. It feels like a job for a TestFailedException
with a long error message, one for each internal failure, including
perhaps a filename and line number for each. You would not want to see
the successful ones, correct? Just the failed ones in the error
message?

exactly. The successful ones are not interesting :-)

(though knowing they were called is nice, but that can be dealt with with a small info message.)

I have just dealt with the problem by doing indentation myself and having some

empty info("") lines. And I am finding the problems it is finding extremely useful for

debugging the parser.

The concept of major test, then drill down is an interesting one too. Sometimes

one has functions that can make it dead certain when something is functioning. But

then it is a lot more difficult to find out WHAT is wrong.

In my case knowing that two graphs are isomorphic is easy since the function was written

for us by some logicians. But knowing which relations did not match is a lot more complicated,

and quite important. So a feature to compose error information could be useful here and in other

places I think....

Bill

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

--
Bill Venners
Artima, Inc.
http://www.artima.com

> To post to this group, send email to scalatest-users@googlegroups.com

> To unsubscribe from this group, send email to

> scalatest-users+unsubscribe@googlegroups.com

> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest

Bill Venners

unread,

Feb 28, 2012, 3:24:27 PM2/28/12

to scalate...@googlegroups.com

Hi Henry,

When I added the FailureOf trait this is what I had in mind, but I
wanted more use cases before going further, to make sure I understood
what the actual problems were that needed solving. I think it may make
sense to add some kind of trait that a TestFailedException could mix
in that indicates there are multiple causes. So that reporters can get
at all the original exceptions if they want to, but that the TFE
thrown (the one with multiple causes) would also have a message that
has all the causes formatted nicely. And then adding a method like
assertNoFailures(results: Seq[Option[Throwable]]) to FailureOf would
throw one of those if there were any failures.

Bill

>> > To post to this group, send email to scalate...@googlegroups.com