Proposal to enhance MP specs TCK coverage

[Antoine Sabot-Durand]

Hi all,

And happy new year.

I'd like to propose enhancement for our TCKs to make them more consistant and include information regarding test coverage.

The idea is to get inspiration from the CDI and Bean Validation specs and TCK.

To track test coverage we did the following 

1) Add unique asciidoc section identifier in the spec

2) Create a metadata file listing all spec sections (from 1)) with their id and identifying each statement in the section

3) Add metadata on each TCK test (annotation) to link each test to statement listed in 2)

4) produce a coverage report by checking that all statements listed in 2) has at least a matching test by checking test metadata defined in 3)

Step 2 is the most time consuming but it has good side effect: by forcing us re-reading spec to extract statement it can help to check ambiguous sentence in the text.

In CDI and BV we are using JBoss test Audit [1] to produce test coverage. This lib or a similar one could be imagined to perform these tasks ?

Wdyt ?

Antoine Sabot-Durand

[1] https://github.com/jboss/jboss-test-audit

[Emily Jiang]

+1. I was thinking about this in the past but did not get time to look into it.

Emily

[Kevin Sutter]

I think this idea is consistent with John's thread on 1.4 content...  Focus should be on developer experience -- that can include our implementor's experience as well as users of MicroProfile.
https://groups.google.com/forum/#!topic/microprofile/iQb_O0dc_R0

[Antoine Sabot-Durand]

Hi

On Mon, Jan 8, 2018 at 11:25 PM Kevin Sutter <kwsu...@gmail.com> wrote:

I think this idea is consistent with John's thread on 1.4 content...  Focus should be on developer experience -- that can include our implementor's experience as well as users of MicroProfile.
https://groups.google.com/forum/#!topic/microprofile/iQb_O0dc_R0

Thx Kevin. Well it's more doc and impl quality enhancement than developer experience but yes, it's consistent with John proposal

 

On Monday, January 8, 2018 at 11:43:55 AM UTC-6, Emily Jiang wrote:

+1. I was thinking about this in the past but did not get time to look into it.

How can we move forward on that ? Do you feel like experimenting it on Fault Tolerance spec ? 

 

Emily

On Monday, January 8, 2018 at 3:33:27 PM UTC, Antoine Sabot-Durand wrote:

Hi all,

And happy new year.

I'd like to propose enhancement for our TCKs to make them more consistant and include information regarding test coverage.

The idea is to get inspiration from the CDI and Bean Validation specs and TCK.

To track test coverage we did the following 

1) Add unique asciidoc section identifier in the spec

2) Create a metadata file listing all spec sections (from 1)) with their id and identifying each statement in the section

3) Add metadata on each TCK test (annotation) to link each test to statement listed in 2)

4) produce a coverage report by checking that all statements listed in 2) has at least a matching test by checking test metadata defined in 3)

Step 2 is the most time consuming but it has good side effect: by forcing us re-reading spec to extract statement it can help to check ambiguous sentence in the text.

In CDI and BV we are using JBoss test Audit [1] to produce test coverage. This lib or a similar one could be imagined to perform these tasks ?

Wdyt ?

Antoine Sabot-Durand

[1] https://github.com/jboss/jboss-test-audit

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/155aad93-9b89-4962-8ed7-dc795aa28ef9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emily Jiang]

Antoine,

> How can we move forward on that ? Do you feel like experimenting it on Fault Tolerance spec ?

Experimenting on Fault Tolerance spec is exactly what I was going to recommend. After we have done FT spec, the other specs can use the exactly same pattern.

Emily

[Jeff Mesnil]

Hi Antoine,

+1 that's a very good idea.

Out of curiosity, have you considered the javadoc API as a source of statement too?

An use case for example would be about thrown exceptions. It does not always make sense to mention them in the spec but the TCK should cover such edge cases and it would be great to reference the Javadoc too from your coverage tool.

I also think that your proposal would be a great opportunity to start splitting the api/sec and the TCK as John and Scott have already proposed.

Using this TCK coverage, we could then enhance/improve the TCK without requiring a spec bump.

jeff

[John D. Ament]

On Wednesday, January 10, 2018 at 9:05:53 AM UTC-5, Jeff Mesnil wrote:

Hi Antoine,

+1 that's a very good idea.

Out of curiosity, have you considered the javadoc API as a source of statement too?

An use case for example would be about thrown exceptions. It does not always make sense to mention them in the spec but the TCK should cover such edge cases and it would be great to reference the Javadoc too from your coverage tool.

I think as long as we stop duplicating spec content in the javadocs, its an ok approach.  However, most of the javadoc content today is redundant with the spec contents.

 

I also think that your proposal would be a great opportunity to start splitting the api/sec and the TCK as John and Scott have already proposed.

Using this TCK coverage, we could then enhance/improve the TCK without requiring a spec bump.

I'm not sure if the John referenced is me or John C.  I'm very iffy on this.  I'm OK with having more TCK versions than spec versions.  I think we understand eclipse release policy enough to say that the proper way we should structure is 2 repos per spec, 1 for the spec/api JAR, a second for the TCK JAR.  The proposal from Antoine lends itself towards supporting this better as well, as we can start to see coverage per spec clearer and ensure we don't introduce tests for future features (e.g. we can't do trunk based development).

 

jeff

[Andreas Badelt]

Hi Antoine,

I like this idea as well, and am currently "replicating" the coverage / audit process from beanvalidation for the MVC spec.

Regarding "Step 2": The creation of the metadata file is automated, given you have tagged the "testable" snippets in the spec (asciidoc) first.

Cheers,

Andreas

[Antoine Sabot-Durand]

Hi all,

Back after a few days of flu. I'm glad this idea please some of you.

On Thu, Jan 11, 2018 at 11:12 AM Andreas Badelt <and...@badelt.it> wrote:

Hi Antoine,

I like this idea as well, and am currently "replicating" the coverage / audit process from beanvalidation for the MVC spec.

Regarding "Step 2": The creation of the metadata file is automated, given you have tagged the "testable" snippets in the spec (asciidoc) first.

Can you provide us a link to an example. It's interesting to have this process automated but don't force you to produce a lot of snippet in the spec ? If I go back to CDI, this wouldn't have been realistic since a lot of rules deals with container error detection.

 

Cheers,

Andreas

On Monday, January 8, 2018 at 4:33:27 PM UTC+1, Antoine Sabot-Durand wrote:

Hi all,

And happy new year.

I'd like to propose enhancement for our TCKs to make them more consistant and include information regarding test coverage.

The idea is to get inspiration from the CDI and Bean Validation specs and TCK.

To track test coverage we did the following 

1) Add unique asciidoc section identifier in the spec

2) Create a metadata file listing all spec sections (from 1)) with their id and identifying each statement in the section

3) Add metadata on each TCK test (annotation) to link each test to statement listed in 2)

4) produce a coverage report by checking that all statements listed in 2) has at least a matching test by checking test metadata defined in 3)

Step 2 is the most time consuming but it has good side effect: by forcing us re-reading spec to extract statement it can help to check ambiguous sentence in the text.

In CDI and BV we are using JBoss test Audit [1] to produce test coverage. This lib or a similar one could be imagined to perform these tasks ?

Wdyt ?

Antoine Sabot-Durand

[1] https://github.com/jboss/jboss-test-audit

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/a70e0a0e-c468-44cb-89f5-689a47a8aa49%40googlegroups.com.

[Antoine Sabot-Durand]

On Tue, Jan 9, 2018 at 6:33 PM 'Emily Jiang' via Eclipse MicroProfile <microp...@googlegroups.com> wrote:

Antoine,

> How can we move forward on that ? Do you feel like experimenting it on Fault Tolerance spec ?

Experimenting on Fault Tolerance spec is exactly what I was going to recommend. After we have done FT spec, the other specs can use the exactly same pattern.

 Thanks Emily, so I start to prototype this with FT spec 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/113c7a08-aa1f-43fb-8ba4-29ab5c441830%40googlegroups.com.

[Antoine Sabot-Durand]

On Thu, Jan 11, 2018 at 4:36 AM John D. Ament <john.d...@gmail.com> wrote:

On Wednesday, January 10, 2018 at 9:05:53 AM UTC-5, Jeff Mesnil wrote:

Hi Antoine,

+1 that's a very good idea.

Out of curiosity, have you considered the javadoc API as a source of statement too?

An use case for example would be about thrown exceptions. It does not always make sense to mention them in the spec but the TCK should cover such edge cases and it would be great to reference the Javadoc too from your coverage tool.

I think as long as we stop duplicating spec content in the javadocs, its an ok approach.  However, most of the javadoc content today is redundant with the spec contents.

Idea has to be explored but I tend to agree with John. Duplicating info is spec and javadosc should be totally avoided. It brings very annoying inconsistencies.

 

 

I also think that your proposal would be a great opportunity to start splitting the api/sec and the TCK as John and Scott have already proposed.

Using this TCK coverage, we could then enhance/improve the TCK without requiring a spec bump.

I'm not sure if the John referenced is me or John C.  I'm very iffy on this.  I'm OK with having more TCK versions than spec versions.  I think we understand eclipse release policy enough to say that the proper way we should structure is 2 repos per spec, 1 for the spec/api JAR, a second for the TCK JAR.  The proposal from Antoine lends itself towards supporting this better as well, as we can start to see coverage per spec clearer and ensure we don't introduce tests for future features (e.g. we can't do trunk based development).

 

jeff

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/5307f3d2-215c-4d81-aaef-1b4c33b27ed7%40googlegroups.com.

[Andreas Badelt]

Hi Antoine,

On 16. Jan 2018, at 10:46, 'Antoine Sabot-Durand' via Eclipse MicroProfile <microp...@googlegroups.com> wrote:

Can you provide us a link to an example. It's interesting to have this process automated but don't force you to produce a lot of snippet in the spec ? If I go back to CDI, this wouldn't have been realistic since a lot of rules deals with container error detection.

I had thought that CDI and beanvalidation were using the exact same process - looks like they are using it in opposite directions: For CDI starting in TCK and showing coverage - or tests associated with a section - in spec. Is that correct - and is the tck-audit.xml for CDI then created manually?

You can find examples for the additions to asciidoc in the beanvalidation spec project (https://github.com/beanvalidation/beanvalidation-spec), or in my fork of the MVC spec project:  https://github.com/abadelt/mvc-spec

Look for example for [tck-testable] here:

https://raw.githubusercontent.com/abadelt/mvc-spec/master/spec/src/main/asciidoc/chapters/assertions.asciidoc

The tck-audit.xml creation is then just part of the maven generate-resources phase.

The MVC spec is a bit different in that it collects all assertions again in one appendix section (and there aren’t that many) - so in principle you do not need to have the [tck-testable] markup scattered around the whole document. On the other hand that does not help coverage reporting on different parts of the spec.

Cheers,

Andreas