Removing N/A as a possible response to a test

[Marisa DeMeglio]

Hi all,

Markus and I were just chatting about the EPUB testsuite website that I've been working on, and the idea came up to remove "N/A" as a potential response to tests. Acceptable remaining possibilities are then "Pass", "Fail", and "Not yet answered". Scoring makes much more sense this way -- if a reading system does not support a feature, for example, then it should record a "fail" for those tests. Whereas, responding to a test with "N/A" doesn't really give any useful information. It either passes or fails.

Does anyone have an example where N/A is required to be an option? If there are none and we can indeed remove it, I'd be happy to edit the testsuite content to remove all mentions of it from user instructions.

Marisa

[Patrick Keating]

Scripting is an area that is not required for reading system to implement
but it could be a valuable part of the test suite -and there are probably
more features like this in this spec. Would "Not Yet Supported" be
slightly less inflammatory than "Fail"? (Or perhaps I am being overly
sensitive to language during this needlessly divisive government shitfest)
-but I do agree that a Boolean for each test is the way to go.

--
Patrick Keating
CTO, Bluefire Productions
Your brand. Your app.

>--
>You received this message because you are subscribed to the Google Groups
>"epub-testsuite-discuss" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to epub-testsuite-di...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.

[Marisa DeMeglio]

So instead of "Pass" / "Fail", we could have "Pass" / "Does not pass"? I don't have any strong feelings about the wording, personally.

Marisa

[Patrick Keating]

I don¹t either. But I was trying to say that there could be a 'Fail' do
to an optional feature that is not implemented in the reader -and
wondering whether that would be considered a 'Fail' by most testers since
it was omitted intentionally. Perhaps splitting hairs a bit.

[Matt Garrish]

I like the simplicity of supported/not supported? That way there's no
judgement attached to lack of support.

And as far as what is optional/required, I haven't looked at the web form
lately but does that information translate across from the tests? Those two
pieces of information together should give a pretty accurate picture of
whether required functionality is failing, without actually being so blunt
about it.

Matt

[Marisa DeMeglio]

Sounds good to me. FYI The site indicates "required" or "optional" alongside the test description.

Marisa

[Matt Garrish]

Perfect!

But does this mean we need to revise all the PASS/FAIL wording in the tests?
:(

(I like the question mark at the end of the first sentence in my reply. My
Ralph Wiggum moment for the day.)

[Ori Idan]

I think that N/A means not supported. So I don't see a need to remove it.

-- 

Ori Idan

On Thu, Oct 3, 2013 at 1:56 AM, Matt Garrish <matt.g...@bell.net> wrote:

Perfect!

But does this mean we need to revise all the PASS/FAIL wording in the tests? :(

(I like the question mark at the end of the first sentence in my reply. My Ralph Wiggum moment for the day.)

Matt

-----Original Message----- From: Marisa DeMeglio

Sent: Wednesday, October 02, 2013 6:49 PM

To: epub-testsuite-discuss@googlegroups.com

Subject: Re: Removing N/A as a possible response to a test

Sounds good to me.  FYI The site indicates "required" or "optional" alongside the test description.

Marisa

On Oct 2, 2013, at 3:31 PM, Matt Garrish <matt.g...@bell.net> wrote:

I like the simplicity of supported/not supported? That way there's no judgement attached to lack of support.

And as far as what is optional/required, I haven't looked at the web form lately but does that information translate across from the tests? Those two pieces of information together should give a pretty accurate picture of whether required functionality is failing, without actually being so blunt about it.

Matt

-----Original Message----- From: Marisa DeMeglio
Sent: Wednesday, October 02, 2013 6:08 PM

To: epub-testsuite-discuss@googlegroups.com

email to epub-testsuite-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "epub-testsuite-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to epub-testsuite-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "epub-testsuite-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to epub-testsuite-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "epub-testsuite-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to epub-testsuite-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "epub-testsuite-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to epub-testsuite-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "epub-testsuite-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to epub-testsuite-discuss+unsub...@googlegroups.com.

[Marisa DeMeglio]

Haha .. well yep I guess this makes the rewording more complicated. All the N/A references have to be removed in favor of "Not Supported", and all the Pass/Fails change too. Given the volume of edits required, I was going to wait a little to see if there are any objections before I plow ahead too eagerly.

Marisa

[Marisa DeMeglio]

The problem I see here is that having "Pass/Fail/NA" confuses two questions:

1. Can the RS perform the test? 

2. Why not? Is it that the feature is not implemented (N/A) or is the implementation broken (Fail)?

We really only care about #1 though. Therefore two values (Supported/Not Supported) are enough.

Marisa

To unsubscribe from this group and stop receiving emails from it, send an email to epub-testsuite-di...@googlegroups.com.

[Matt Garrish]

But who makes the determination of what is supported v. what has failed if this is a public form? Is it a failure of the reading system or a lack of support in the core it’s built upon? Is a feature not supported or is it failing on a given platform?

 

Depending on the knowledge level of the person running an evaluation, those distinctions could lead to evaluations with different results, which is one value of reducing to a binary evaluation.

 

Or, asked another way, what does the distinction matter for evaluation? We aren’t trying to derive a feature support matrix, only know what works and what doesn’t work.

 

Matt

To unsubscribe from this group and stop receiving emails from it, send an email to epub-testsuite-di...@googlegroups.com.

[Markus Gylling]

Or, asked another way, what does the distinction matter for evaluation? We aren’t trying to derive a feature support matrix, only know what works and what doesn’t work.

Exactly, we are not capturing the reasons why certain tests don't pass, and this is where the "N/A" option started creating more problems than it solved. By only using optional vs required and passed vs failed, I think we keep things simple both for evaluators, and later also for information consumers. 

[Ori Idan]

Why not simply replace all N/A with Not supported?

[Vincent Gros]

Hello folks,

I totally agree with the Matt's idea "supported/not supported". 

Here is one example to explain why it is important to remove N/A :

The FasterReader seems to have a 100% support of CFI. Great! But in details, there are 3 "PASS" and 4 "N/A": I don't think these results provide a full support of CFI.

So, we need to have an "objective" boolean to clearly represent the readers's performances.

Maybe we can authorize some comments? And thus a user could say "but it is on the todo list, see URL" or "this is a bug, see #issue or #forum". 

Also, if a user sees a "Not supported" test, and he doesn't understand why, he can do the test himself (because there is the reference of the test in the testsuite).

/ Vincent

2013/10/3 Ori Idan <o...@helicontech.co.il>

[Ric Wright]

Sorry to be late to the party, and perhaps I am missing something here, but it seems like there are still three categories needed:

• Required and the test passes:   PASS
• Required and the test fails: FAIL
• Optional and the test passes: PASS
• Optional and the test fails: NOT SUPPORTED

The only problem I see is that there is a fifth case where the feature is optional, it HAS been implemented, but it fails.  The problem is whether the tester knows that it SHOULD work or not. And, IMO, that has to be a judgement call by the tester.  If they are sure the feature has been implemented then they can mark it FAIL.  If they aren't sure they can mark it not supported and the RS implementor/QE can check and update it to FAIL if it SHOULD have worked.

As for the labels themselves, I am fine with FAIL and NOT SUPPORTED.  A failure is a failure in my book.

But there is another, related aspect to this discussion:  reporting the results.  I assume we will get to a point where we can query the DB and export the results in some form that allows one to easily see what is passing, what is not, etc.  Having the ability to easily determine which RS' are implementing and passing optional features vs. required features would be good.

Ric

[Markus Gylling]

Sorry to be late to the party, and perhaps I am missing something here, but it seems like there are still three categories needed:

Its important to keep in mind that we can (and IMO should) differentiate between how the tests are filed during the evaluation process, and how they later are presented. Its also important to keep in mind that in any of these stages, including speculation on why a test failed (because of a bug? because of it not being supported? because of a misinterpretation made by the evaluator? etc) makes things brittle and messy, since many times this information will not be available to the evaluator. 

So - assume the following:  

1) In the test materials themselves, 

* we carry info on whether each test pertains to optional or required ("ctest"/"otest") functionality

* we notate the tests with either of "pass", "fail", or "not yet tested" (the latter obviously being the initial state for all tests' result field)

2) In the eventual presentation (and same holds for db query)

* there is info available on which tests have actually been run at any given point

* there is info on whether each run test has failed or passed

* there is info on optional vs required for each test

This gives the presentation layer the ability to provide separate pass percentages for optional vs required functionality. It provides the presentation layer to use whatever verbiage it wants to describe failure states for both optional and required tests. 

Since we now would also have data on which tests have actually been run, this data can be used to weight the measures; this is important not only because of partial test runs, but also applies directly to test suite updates; whenever a suite update is done, the system will automatically set "not yet tested" to tests that have changed or have been added, which would serve as a direct indicator that the results are not up to date. 

To me, this seems like the least brittle and most straight forward approach.

[Ric Wright]

This works well for me.  Thanks Markus.

[Vincent Gros]

+1 

--

Vincent

[Marisa DeMeglio]

Great, so if there are no objections, I am going to go ahead and make the following changes:

1. Remove "N/A" from the testsuite documents. There are many such statements as "If the reading system does not support scripting, this test should be marked N/A". These will be simply removed.

2. Replace mentions of "Pass" and "Fail" with "Supported" and "Not Supported".

Tests are still going to be classified as "Required" or "Optional", which, combined with a result of "Supported/Not Supported", will give us good unambiguous data when we go to analyze the data later. 

Marisa

[Marisa DeMeglio]

After looking more in depth at the test content, I think that #1 below is a no brainer, but #2 just sounds awkward. For example, now we have this type of wording:

bindings-010

Tests whether bindings on objects are supported.

… test content .. 

If the preceding paragraph reads "PASS", the test passes. 

we could change it to something like:

If the preceding paragraph reads "SUPPORTED", then the test (feature?) is supported.

but I think it's a bit too pedantic and just sounds awkward. Why not just leave it as "pass/fail" in the content, and use the more diplomatic "supported/not supported" on the website form? I think people will be able to figure it out, and it leaves the wording sounding more natural.

Thoughts?

Marisa

[Daniel Weck]

I like when the test result is clear: PASS or FAIL. I like the sound of "feature supported / not supported" on the web form. Will sumitters be able to enter comments? (e.g. "#120 tests failed, but our next product release will support Media Overlays in reflowable mode")

daniel

--

[Patrick Keating]

Seems reasonable to me.

Patrick

Sent from my iPad

[Marisa DeMeglio]

Yes, I'm going to add a "notes" field. Somebody else requested it as well.

Marisa