TCKs

[John D. Ament]

All,

So far for Fault Tolerance and Config, we have published a platform neutral TCK that can be consumed and tested independent of a platform.  It looks like Health is taking a very similar approach.  I'm concerned with the route the JWT Auth module is taking, specifically they are looking to have profiles per container supported.

- This approach doesn't work for closed source vendors.

- This approach means I can only test on latest.

- This approach requires Eclipse IP checks on all of our implementation dependencies (of which we may not be able to pass).

- It doesn't scale.

We had a similar problem with the original conference app, where it required each of us to have a profile in it.  It would be better in opinion if we had one way of consuming specs, and only using that.

Thoughts?

[Alasdair Nottingham]

I agree that it seems wrong for the TCK to be tied to the implementation rather than being platform neutral. I haven’t had a chance to look at what is being done in that repository yet, but based on the emails to this list I share your concern.

Alasdair

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/674f47e1-330f-4db9-90b2-8be824145f05%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sst...@redhat.com]

The problem I have with the current general approach is that is it under documented as there is no sample run_tck.pom that illustrates the full set of steps. At a minimum, there would have to be a sample-xxx-run-tck.pom based on some ASLv2 licensed container for me to support this approach. There is nothing more I dislike than having to fumble around getting a TCK testsuite running due to confusing docs or lacks of docs/examples.

[Alasdair Nottingham]

I agree that there needs to be good documentation, but an approach that excludes implementations based on license, or availability in maven central, seems counter to the goals here.  The issue of having some having profiles built into the TCK and sets up an inequality for implementations. The ones in the TCK will be more valid or more visible than ones that are not. I think we would be better off working out how to make it simple, easy and consistent to run the various TCK’s than to a) have a difference between JWT and the others and b) do something that favour some implementations over others.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/4c692d5b-ffa4-4bae-8418-2e085d6406e5%40googlegroups.com.

[John D. Ament]

I had the same concerns.  FT and Config both have pretty robust "how to run the TCK" documents

https://github.com/eclipse/microprofile-fault-tolerance/blob/master/tck/running_the_tck.asciidoc

https://github.com/eclipse/microprofile-config/blob/master/tck/running_the_tck.asciidoc

I would like to see at least this level of documentation on how to run them.  Can someone else review and provide input?

John

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

[sst...@redhat.com]

There are no implementations in the TCK, only configuration that allows a given vendor stack to run against that TCK.

I would like to see where the Eclipse IP policy indicates that just a reference to a non-conforming licensed library, that only allows a particular stack to be run against the Eclipse technology, is not allowable in an Eclipse code repository.

[sst...@redhat.com]

The Config TCK is an example I had in mind about not being complete as there is no working configuration example. This is an open source project that I would expect to have open source working examples of. Having to piece together a working example from a document, as opposed to taking a working example that one can modify for their environment, are two complete different levels of user experience in my view. 

[Alasdair Nottingham]

I don’t see what Eclipse IP policy has to do with anything. I’m not raising a concern because of some magic understanding I don’t have of the Eclipse IP policy but a concern about maintaining the equality of status of implementations.

Alasdair

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/9a3ea38b-3787-485c-bd97-789c7380099d%40googlegroups.com.

[sst...@redhat.com]

John referenced a concern with regard to Eclipse IP checks on all of our implementation dependencies. I don't view an example profile as an implementation dependency, so I would like to understand how the Eclipse IP views it.

I'm interested in driving implementations and usage over equality. Out of the box working examples is the best way to achieve that in my experience.

[John D. Ament]

Yep I raised it.  If you look at https://www.eclipse.org/projects/dev_process/ip-process-in-cartoons.php there's a graphic indicating the issue - you're adding library dependencies.  This means they have to pass CQ's.

John

[sst...@redhat.com]

Thanks for the link. Following that through to the full process, and then the third party process, I came to this:

https://www.eclipse.org/org/documents/Eclipse_Policy_and_Procedure_for_3rd_Party_Dependencies_Final.pdf

In there, they divide the dependencies into two types: a) "works with" and b) "prerequisite". 

1. a)  Works-with Dependencies:

   1. The Eclipse software does not require the third party software to be present. If the third party software happens to be present, the Eclipse software may call or invoke it. Example: If a web browser is present, clicking on URL's in Eclipse will cause the user's configured web browser to open the URL.

      or

   2. The Eclipse software is designed to work with multiple third party software choices that provide similar functionality - the choice of which to use is up to the user. At least one of those must be a prerequisite (see below) or approved by the EMO for distribution by the project. Example: If a project requires a persistence mechanism, it can allow the user to select from several different implementations.

2. b)  Prerequisite Dependencies: The Eclipse software requires the third party software in order to work correctly or to provide full function. Example: Eclipse requires Apache Ant. 

Based on that, I would classify the use of profile configurations as "works with" dependencies. If that is true, then item 2 of the policy applies:

2. All "works-with" dependencies as determined by the PMC are approved for use by the projects without further EMO review. 

So, the actual determination of the type of dependency needs to be clarified. That is formally detailed as:

The Eclipse Foundation policy on the use of third party software is as follows:

1. It will be the responsibility of each PMC to document all "works with" and "prerequisite" dependencies between Eclipse Foundation code and non-Eclipse Foundation code. As part of this process, the PMCs will be expected to make a determination whether a dependency is a “works with” or a “prerequisite”.

a. These discussions and decisions must occur transparently either via email on the public PMC mailing list, or in the minutes of meetings distributed to the public PMC mailing list.

2. These decisions must be documented by the project in their IP log to ensure their visibility during release and other review checkpoints.

3. All PMCs are expected to seek the advice of the EMO if they are uncertain with respect to a determination on whether a dependency is “works with” or “prerequisite”.

4. As part of their determination, the PMC is expected to review any issues with respect to license compatibility. If the dependency is distributed under any license other than those documented by the EMO as compatible with the EPL, the PMC must request the guidance of the EMO. 

[Alasdair Nottingham]

Does the works-with type ii dependency mean to require at least one implementation be a prereq? If so (which might be an overly strict interpretation), I don’t see how you do that without essentially essentially creating a reference implementation (or at least a most favoured implementation).

Alasdair

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/a4cbbae7-f75b-495a-8a0e-e3c4c442183a%40googlegroups.com.

[sst...@redhat.com]

No, a works-with relationship to third party code only requires some integration pattern in the TCK that third party code has to conform with in order to work with the vendor agnostic code in the repository. It can be a CDI dependency injection requiring a vendor provided producer, java.util.ServiceLoader service provider usage, etc. 

For the microprofile-jwt-auth TCK, the security integration requirements often require a vendor specific descriptor in the JAX-RS deployment in order to accomplish the configuration of the authentication and authorization stack as that is a detail left undefined by Java EE. I would also view such configurations as works-with or even a non-dependency as the descriptor itself would be licensed under ASLv2, and any reference to a schema that may have incompatible license terms is really a dependency of the runtime container parsing of the descriptor, which the TCK code is not involved with.

[Werner Keil]

>You also need to specify which tests you want to run, e.g. create a file tck-suite.xml in your project which contains >the following content:

Sounds odd, what is the purpose of the TCK if everyone can decide which tests to run?;-)

It may not apply to an individual feature like Health or Failover, etc. unless they are modular, but combining several TCKs a kind of "profiles" like we did for JSR 363 https://github.com/unitsofmeasurement/unit-tck#profiles might be better to know which tests are executed. Otherwise a TCK that people can just mutually compose makes little sense.

Werner

[Ondro Mihályi]

In short my points are:

 - it should be easy to setup a project to run all TCK tests against an implementation

 - it makes sense that the TCK also provides an example project that runs the TCK against a dummy implementation (all tests may fail)

In fact, I really like the idea of an example project to run the TCK with an implementation. We didn't do it for the TCK itself, mostly because we didn't have time and weren't focused on this. But I created such example project internally for the Payara engineering team to implement the config feature so I confirm that the need is definitely there. However, I'm still strongly against delivering a compliant RI with the feature, the example project should contain only a dummy implementation that would fail with most tests.

To Scott and others, who discuss the IP rights and CQs - I suggest to stop that conversation here. It's irrelevant if we don't want to have any dependency on external implementations in the TCK and I think that most people really want to avoid that. It's not about IP or CQ, it's about the principle of being neutral to any implementation. MP should provide an easy way to test any implementation but should really stay neutral to any of them.

To Werner, I agree that the TCK should be executed as a whole to qualify that an implementation is compatible. 

But there are some major differences to JSRs:

 - MP implementations are not certified

     - that means anybody can claim they have a compatible implementation without proving it

 - the TCK is open and available for anybody to run against any implementation

     - it should be possible to run it yourself against any implementation and find out for yourself (for some impls it would be easy, for others you would need to create an arquillian adapter if it's not freely available)

Another factor is that having the ability to specify which tests are executed simplifies the development of a compatible implementation - during development, it's easier to focus on a subset of tests when you just run those and turn off the others.

So, in the end, it makes perfect sense to be able to execute a subset of tests if the TCK is open and easily executable by anybody, which is to a great extent true for MP Config and Fault Tolerance. Both can be simplified even more and we should attempt to do it.

--Ondro

[sst...@redhat.com]

This example project your speaking of sounds like a stub RI. For Config that is a simple thing to do, but in general, there are too many aspects that have to be stubbed out to not rely on some existing implementation. For example, we can have stub ServerAuthModule(SAM) code for the MP-JWT TCK, but scaffolding something to actually get that SAM invoked with the bearer token from JAX-RS client code in a unit test is way too much effort. Are you suggesting something different?

It is a good point that the TCK should have a base test driver configuration that ensures all tests are run by default. 

I'm still going to argue that the easiest way to do this is to include profile configurations in the TCK. When implementation X wants to be recognized for passing the TCK, they submit their profile fragment along with the test run summary in a PR. It's still a self auditing system that has the benefit of allowing anyone else to understand how to run that implementation and verify it for themselves. I really don't see how this promotes one vendor over another.

[Ondro Mihályi]

I was talking about a really dummy stub, which doesn't even have to compile or run - the example project should be just an example and there's no reason it should work without a proper implementation.

On the other hand, it's an interesting idea to use profiles to implicitly certify an implementation - however, I see some problems here.

First, implementations are created very often after the final version of a feature is released. Profiles would have to be added after the release, which in fact means they would be added to the next snapshot release - at least this is how we work in the config repo. A solution would be to have a separate "TCK runner" repository, which would be independent from feature lifecycles and would contain configuration for every MP version and profiles for every submitted and accepted implementation.

Second issue is the question who and how we would ensure that the implementation passes the TCK before we add the profile for it? This can be solved to, e.g. by a Jenkins job, but we need to consider it too.

In the end, I still think that it's much simpler for the MP project to avoid having unnecessary dependencies on any implementation and only provide an example project into which an implementation has to be added as a dependency or which would use a dummy implementation. 

I don't believe this is too much work, even for JWT. And if it is, then it's possible to create a project tied to a specific implementation, keep it outside MP and later just remove the dependency on the implementation and add it as an example project into MP (obviously it wouldn't even be executable but it's not essential for an example project to get started with building an implementation)

--Ondro

[sst...@redhat.com]

On Tuesday, August 1, 2017 at 3:04:16 PM UTC-7, Ondro Mihályi wrote:

I was talking about a really dummy stub, which doesn't even have to compile or run - the example project should be just an example and there's no reason it should work without a proper implementation.

Ok, so for JWT it certainly is helpful to show how to interact with the bearer token via various security ingress contexts such as and JAX-RS container interceptor, a SAM, etc. I originally had that in the JWT project under and ri subproject. It did have dependencies on implementations to simplify some aspects, but those could be culled out. I have added a sub topic to the TCK agenda item for next week's call on this topic.

 

On the other hand, it's an interesting idea to use profiles to implicitly certify an implementation - however, I see some problems here.

First, implementations are created very often after the final version of a feature is released. Profiles would have to be added after the release, which in fact means they would be added to the next snapshot release - at least this is how we work in the config repo. A solution would be to have a separate "TCK runner" repository, which would be independent from feature lifecycles and would contain configuration for every MP version and profiles for every submitted and accepted implementation.

The TCK should have minor releases often and in between the spec and api releases because it should be an evolving entity with unit tests added and corrected as implementations go through passing it, and spec meanings are clarified. The implementation profiles still could be placed into another repository that has even more frequent releases and pulled in as a submodule as desired. Maybe the TCK release should be completely independent of the api and spec releases. I'll get back to this below.

 

Second issue is the question who and how we would ensure that the implementation passes the TCK before we add the profile for it? This can be solved to, e.g. by a Jenkins job, but we need to consider it too.

 

In my mind this is trivially handled with a pull request that includes the profile AND the TCK test summary.txt/xml file. There is no need to actually verify that the profile produces the results claimed. This is in fact more than is required for Java EE certification, at least when I was doing it as you simply indicated that your implementation passed. Sun did reserve the right to audit your results, but that is a given here as anyone is free to take the profile and run it against the available TCK.

In the end, I still think that it's much simpler for the MP project to avoid having unnecessary dependencies on any implementation and only provide an example project into which an implementation has to be added as a dependency or which would use a dummy implementation. 

So first, by "unnecessary dependencies on any implementation" do you include in that a maven profile that references a specific implementations dependencies?

Second, here is where the transition from an example usage to something that is actually is run by the TCK is fuzzy for me. Maybe it makes sense to wire up some of the example code to test basic behaviors, but there will be more involved tests where that is not practical and what is in the TCK repository cannot run successfully without an implementation. So how do I verify that the TCK itself is in a state to release if it's baseline has errors?

I would argue that there has to be one passing implementation before a TCK can be released. Without this it is a complete unknown. We could have a release process that first puts out the api and spec portions of a MP features, and then have a time box period where conforming implementation profiles are collected to produce the x.0 release of the TCK.

[John D. Ament]

Scott,

On Tuesday, August 1, 2017 at 6:36:23 PM UTC-4, sst...@redhat.com wrote:

On Tuesday, August 1, 2017 at 3:04:16 PM UTC-7, Ondro Mihályi wrote:

I was talking about a really dummy stub, which doesn't even have to compile or run - the example project should be just an example and there's no reason it should work without a proper implementation.

Ok, so for JWT it certainly is helpful to show how to interact with the bearer token via various security ingress contexts such as and JAX-RS container interceptor, a SAM, etc. I originally had that in the JWT project under and ri subproject. It did have dependencies on implementations to simplify some aspects, but those could be culled out. I have added a sub topic to the TCK agenda item for next week's call on this topic.

Specific to JWT, I would expect part of the TCK actually running in a web server, so it makes sense.  You need to process an HTTP request and mocking out a body isn't useful.

 

 

On the other hand, it's an interesting idea to use profiles to implicitly certify an implementation - however, I see some problems here.

First, implementations are created very often after the final version of a feature is released. Profiles would have to be added after the release, which in fact means they would be added to the next snapshot release - at least this is how we work in the config repo. A solution would be to have a separate "TCK runner" repository, which would be independent from feature lifecycles and would contain configuration for every MP version and profiles for every submitted and accepted implementation.

The TCK should have minor releases often and in between the spec and api releases because it should be an evolving entity with unit tests added and corrected as implementations go through passing it, and spec meanings are clarified. The implementation profiles still could be placed into another repository that has even more frequent releases and pulled in as a submodule as desired. Maybe the TCK release should be completely independent of the api and spec releases. I'll get back to this below.

Agreed on the first part.  This is actually what I'm pushing for within Config, we have 2 defects as I think of it (actually Mark S seemed to think one of them should have worked, and may have been a G specific issue, but I don't want to speak for him).  At the same time, there are minor spec changes so I would expect a new spec version gets shipped.

CDI follows a pattern of TCK releasing independent of spec.  It works OK - its not communicated when a new version is released AFAIK.  Hopefully the number of TCK specific issues should be low, and part of the release process should be various spec memebers testing the spec against their impl(s) to make sure it makes sense.

 

 

Second issue is the question who and how we would ensure that the implementation passes the TCK before we add the profile for it? This can be solved to, e.g. by a Jenkins job, but we need to consider it too.

 

In my mind this is trivially handled with a pull request that includes the profile AND the TCK test summary.txt/xml file. There is no need to actually verify that the profile produces the results claimed. This is in fact more than is required for Java EE certification, at least when I was doing it as you simply indicated that your implementation passed. Sun did reserve the right to audit your results, but that is a given here as anyone is free to take the profile and run it against the available TCK.

I want to make sure one of the points I made previously is clear.  Suppose I'm IBM.  Websphere is not open source, its not available in maven central.  How would we test this impl?

[sst...@redhat.com]

 

In my mind this is trivially handled with a pull request that includes the profile AND the TCK test summary.txt/xml file. There is no need to actually verify that the profile produces the results claimed. This is in fact more than is required for Java EE certification, at least when I was doing it as you simply indicated that your implementation passed. Sun did reserve the right to audit your results, but that is a given here as anyone is free to take the profile and run it against the available TCK.

I want to make sure one of the points I made previously is clear.  Suppose I'm IBM.  Websphere is not open source, its not available in maven central.  How would we test this impl?

I don't see the MP community spending any CI compute cycles on testing any implementation. If there is no reference implementation, then it is up to implementors to test and self certify. It still is useful to have the profile available in the MP repository so that user's of that container can verify for themselves how it performs.

The initial development of the TCK would have to be based on whatever implementations the MP feature community is interested in developing and certifying in the initial post api and spec release time box. Beyond that it would evolve based on usage of implementations and how questions around behavior filter back into issues in the spec and TCK projects.

[sst...@redhat.com]

So I took a look at what the Eclipse Link project is doing, and in their JPA tests repository: http://git.eclipse.org/c/eclipselink/eclipselink.runtime.git/tree/jpa/eclipselink.jpa.test

and essentially they have the equivalent of vendor specific profiles in that there are vendor specific xml, properties and execution logic in the ant build files. Not all of the files in that directory have license headers, but those that do are using the project license terms. So at least in terms of incorporating vendor specific configuration and libraries by reference, there should be no problem with violating Eclipse IP policy rules.

[Ondro Mihályi]

I suggest doing anything that works, even if we don't have a unified opinion. We can improve later if we have issues with any approach. There's no point in discussing too much about it.

For Configuration, we tried to be vendor-agnostic and it worked for us, although it's not very intuitive. Maybe your approach would work better and we may conclude that it also fits better in case of Configuration and other specs.

--Ondro

[sst...@redhat.com]

I have come up with a way for the MP-JWT TCK to be in charge of the tests that are run while providing an implementation agnostic mechanism that is somewhere in between what the Config TCK does and explicitly having implementation profiles included in the TCK pom directly.

Take a look at the prototype that has been pushed into the outstanding microprofile-jwt-auth pull request:

https://github.com/MicroProfileJWT/microprofile-jwt-auth/tree/tck_update

I need to go through getting a container other than Wildly-Swarm working with this approach to verify that there is nothing implementation specific hidden in getting this to work generically, as well as

flesh out the https://github.com/MicroProfileJWT/wfswarm-jwt-auth-tck-viabase example TCK harness artifact that illustrates the use of the MP-JWT TCK base extension classes, but I think this is a promising approach that I can live with.

There still is the question of how we track implementations of a specification, but a list of references to TCK harness artifact projects for passing implementations could be one way.

[John D. Ament]

I just pushed up an updated impl of JWTPrincipal to hammock.  I also added security context support, but not for the contents of javax.annotation.  If you want to see if your approach works, feel free to use ws.ament.hammock:dist-microprofile (CXF/Weld/Undertow/Johnzon/GConfig) or ws.ament.hammock:dist-microprofile-cochise (CXF/OWB/Tomcat/Johnzon/GConfig)

[Werner Keil]

Isn't Hammock a particular vendor already?;-)

While Weld is probably pretty common, Johnzon is a lot less common on the JON-P side (maybe TomEE does, Payara I would be very surprised if it used something other than the Glassfish RI) 

[sst...@redhat.com]

In the current 1.5 release I'm not seeing any JWTPrincipal implementation. Is release pending?

[John D. Ament]

Source code - https://github.com/hammock-project/hammock - but JWTPrincipal is there in 1.5, in security-jose (not part of the MP dist)

[John D. Ament]

Ok, cool.

I just pushed up a few more things - I now process RolesAllowed in addition to Hammock specific annotations.  If you're not logged in, and attempt to access a resource requiring logged in you get a 401 w/ WWW-Authenticate: Bearer response.  If you're missing roles you get a 403.

As far as APIs go, Hammock supports both Weld and OWB, and has integrations for Jersey, RestEasy and CXF, so that would be 6 different profiles (In travis I use a matrix build to test them all).  But for the scope of this testing, Weld+CXF and OWB+CXF should be enough.

So here's the disconnect.  I'm working on Hammock changes, but to test them I need to run a built of MP JWT.  It would be better if a change to Hammock ran the tests.  Hence why I added the MP Config TCK to the MP Dists.

On Monday, August 7, 2017 at 10:04:16 PM UTC-4, sst...@redhat.com wrote:

I created the following prototype TCK harness project for Hammock:

https://github.com/MicroProfileJWT/hammock-jwt-auth-tck

It is running with the new tokens-se profile (using my latest microprofile-jwt-auth, for which there is a new pull request (https://github.com/eclipse/microprofile-jwt-auth/pull/17) to bring the eclipse/microprofile-jwt-auth in sync:

[tck 1130]$ mvn -Ptokens-se -Dtck.container.groupId=ws.ament.hammock -Dtck.container.artifactId=jwt-auth-tck -Dtck.container.version=1.0-SNAPSHOT test

...

[INFO] -------------------------------------------------------

[INFO]  T E S T S

[INFO] -------------------------------------------------------

[INFO] Running org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest

Registered WFSwarmWarArchiveProcessor

Failed as expected with cause: Signed JWT rejected: No JWS key selector is configured

Failed as expected with cause: Signed JWT rejected: No JWS key selector is configured

jwt: eyJraWQiOiJcL3ByaXZhdGVLZXkucGVtIiwidHlwIjoiSldUIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiIyNDQwMDMyMCIsImF1ZCI6InM2QmhkUmtxdDMiLCJ1cG4iOiJqZG9lQGV4YW1wbGUuY29tIiwicmVzb3VyY2VfYWNjZXNzIjp7Im15LXNlcnZpY2UiOnsic2VydmljZS1CIjp7Imdyb3VwcyI6WyJyb2xlLWluLUIiXX0sInNlcnZpY2UtQyI6eyJncm91cHMiOlsiZ3JvdXBDIiwid2ViLXRpZXIiXX0sImdyb3VwcyI6WyJncm91cDEiLCJncm91cDIiLCJyb2xlLWluLW15LXNlcnZpY2UiXX19LCJhdXRoX3RpbWUiOjE1MDIxNTY0NjMsImlzcyI6Imh0dHBzOlwvXC9zZXJ2ZXIuZXhhbXBsZS5jb20iLCJncm91cHMiOlsiZ3JvdXAxIiwiZ3JvdXAyIiwicm9sZS1pbi1yZWFsbSIsInVzZXIiLCJtYW5hZ2VyIl0sInByZWZlcnJlZF91c2VybmFtZSI6Impkb2UiLCJleHAiOjE1MDIxNTY3NjMsImlhdCI6MTUwMjE1NjQ2MywianRpIjoiYS0xMjMifQ.gRHvydFAMQ8jwtXZSPNrooMLFmkyfxS4BdLaJPbBukdsgDBRdZvW3NssMoFlVR9V5SP-ZPjgo095fQW3wMlALCdXQYwudULbvG_gbdGPhABpBSIBp_Jr8vNYl-SM6r71L7iQ-3KpyAt8hwDFHFSxAxs2I8GbIEVv_UVXrUWGpQTJk74qhWMO8tp3u8c1DlJWGtz5l0g7HpxdsYEpFhpVr5RX5_SQEpqWI-rb3YHCAmU79MhKaoGCz4LSwEJEjje9Hpda2MTEhah8mR8yNkrBiNmuvxMIqXH2Ns13ojD8nMNHKMWDMeC34zAba1hS9jdGKVPhOm7aiwkCB1ThK_dcLA

Failed as expected with cause: Signed JWT rejected: No JWS key selector is configured

[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 0.948 s <<< FAILURE! - in org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest

[ERROR] testRIJWTCallerPrincipal(org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest)  Time elapsed: 0.019 s  <<< ERROR!

ws.ament.hammock.jwt.JWTException: Unable to parse jwt

at org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest.testRIJWTCallerPrincipal(TokenValidationTest.java:77)

Caused by: com.nimbusds.jose.proc.BadJOSEException: Signed JWT rejected: No JWS key selector is configured

at org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest.testBadSigner(TokenValidationTest.java:176)

[INFO] 

[INFO] Results:

[INFO] 

[ERROR] Errors: 

[ERROR]   TokenValidationTest.testRIJWTCallerPrincipal:77 » JWT Unable to parse jwt

[INFO] 

[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 1

[INFO]

The failures are due to not being able to set a JWSKeySelector to use the ITokenParser#parse() publicKey argument on the ws.ament.hammock.jwt.processor.DefaultValidatingJWTProcessor as it only uses injection of the JWTConfiguration, and this test is just a simple Java SE based test as the weld dependency has been removed.

I created stubs for the HammockMPJwtTckExtension and HammockArchiveProcessor that will be needed to run the full tests in the container profile, but I don't know what JAX-RS client implementation you might need nor what arquillian container integration, so I left TODOs in the pom.xml to fill in.

[John D. Ament]

Can you change ITokenParser to be looked up via CDI, rather than via ServiceLoader?

If not, I would at least use this impl:

CDI.current().select(DefaultValidatingJWTProcessor.class).get()

The configuration class should be used.  Unless I'm misunderstanding something.

[sst...@redhat.com]

The issue is that there is no CDI implementation for the basic parsing test. Previously this was running under weld-se, but I dropped that. If desired, we can go back to running that test under weld-se and use CDI rather than a ServiceLoader.

[sst...@redhat.com]

I did just try to see if the hammock-dist included a CDI provider via CDI.select()..., and while it does, it appears to depend on CDI 2.0, and so the test is failing to find the SeContainer class:

[ERROR] testRIJWTCallerPrincipal(org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest)  Time elapsed: 0.016 s  <<< ERROR!

java.lang.NoClassDefFoundError: javax/enterprise/inject/se/SeContainer

at org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest.testRIJWTCallerPrincipal(TokenValidationTest.java:77)

Caused by: java.lang.ClassNotFoundException: javax.enterprise.inject.se.SeContainer

at org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest.testRIJWTCallerPrincipal(TokenValidationTest.java:77)

[John D. Ament]

Are you overriding the cdi-api? Mark yours as provided, so that mine will come through.  Yes, I'm exclusively using CDI 2.0 to make use of SE Bootstrapping.

[sst...@redhat.com]

It was not marked as provided, but changing that did not fix the issue. Looking at the 

[tck 1172]$ mvn -Ptokens-se -Dtck.container.groupId=ws.ament.hammock -Dtck.container.artifactId=jwt-auth-tck -Dtck.container.version=1.0-SNAPSHOT dependency:tree

[INFO] Scanning for projects...

[INFO] 

[INFO] ------------------------------------------------------------------------

[INFO] Building MicroProfile JWT Auth TCK Harness 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO] 

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ microprofile-jwt-auth-tck ---

[INFO] org.eclipse.microprofile.jwt:microprofile-jwt-auth-tck:jar:1.0-SNAPSHOT

[INFO] +- org.eclipse.microprofile.jwt:microprofile-jwt-auth:jar:1.0-SNAPSHOT:compile

[INFO] +- javax.enterprise:cdi-api:jar:1.2:provided

[INFO] |  +- javax.el:javax.el-api:jar:3.0.0:provided

[INFO] |  +- javax.interceptor:javax.interceptor-api:jar:1.2:provided

[INFO] |  \- javax.inject:javax.inject:jar:1:provided

[INFO] +- javax.ws.rs:javax.ws.rs-api:jar:2.0:provided

[INFO] +- javax.annotation:javax.annotation-api:jar:1.2:provided

[INFO] +- com.nimbusds:nimbus-jose-jwt:jar:4.23:compile

[INFO] |  +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile

[INFO] |  +- net.minidev:json-smart:jar:1.3.1:compile

[INFO] |  \- commons-io:commons-io:jar:2.5:compile (version selected from constraint [2.4,))

[INFO] +- org.jboss.shrinkwrap:shrinkwrap-api:jar:1.2.6:compile

[INFO] +- junit:junit:jar:4.12:test

[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test

[INFO] +- org.jboss.arquillian.container:arquillian-container-spi:jar:1.1.13.Final:compile

[INFO] |  +- org.jboss.arquillian.core:arquillian-core-spi:jar:1.1.13.Final:compile

[INFO] |  |  \- org.jboss.arquillian.core:arquillian-core-api:jar:1.1.13.Final:compile

[INFO] |  +- org.jboss.arquillian.config:arquillian-config-api:jar:1.1.13.Final:compile

[INFO] |  +- org.jboss.arquillian.config:arquillian-config-impl-base:jar:1.1.13.Final:compile

[INFO] |  |  \- org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-spi:jar:2.0.0-alpha-10:compile

[INFO] |  \- org.jboss.shrinkwrap.descriptors:shrinkwrap-descriptors-api-base:jar:2.0.0-alpha-10:compile

[INFO] +- org.jboss.arquillian.container:arquillian-container-test-spi:jar:1.1.13.Final:compile

[INFO] |  +- org.jboss.arquillian.test:arquillian-test-spi:jar:1.1.13.Final:compile

[INFO] |  \- org.jboss.arquillian.container:arquillian-container-test-api:jar:1.1.13.Final:compile

[INFO] +- org.jboss.arquillian.junit:arquillian-junit-container:jar:1.1.13.Final:test

[INFO] |  +- org.jboss.arquillian.junit:arquillian-junit-core:jar:1.1.13.Final:test

[INFO] |  +- org.jboss.arquillian.test:arquillian-test-api:jar:1.1.13.Final:compile

[INFO] |  +- org.jboss.arquillian.core:arquillian-core-impl-base:jar:1.1.13.Final:test

[INFO] |  +- org.jboss.arquillian.test:arquillian-test-impl-base:jar:1.1.13.Final:test

[INFO] |  +- org.jboss.arquillian.container:arquillian-container-impl-base:jar:1.1.13.Final:test

[INFO] |  +- org.jboss.arquillian.container:arquillian-container-test-impl-base:jar:1.1.13.Final:test

[INFO] |  \- org.jboss.shrinkwrap:shrinkwrap-impl-base:jar:1.2.6:test

[INFO] |     \- org.jboss.shrinkwrap:shrinkwrap-spi:jar:1.2.6:test

[INFO] +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-depchain:pom:2.2.6:test

[INFO] |  +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-api:jar:2.2.4:test

[INFO] |  +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-spi:jar:2.2.4:test

[INFO] |  +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-api-maven:jar:2.2.4:test

[INFO] |  +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-spi-maven:jar:2.2.4:test

[INFO] |  +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-api-maven-archive:jar:2.2.4:test

[INFO] |  +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-impl-maven:jar:2.2.4:test

[INFO] |  |  +- org.eclipse.aether:aether-api:jar:1.0.0.v20140518:test

[INFO] |  |  +- org.eclipse.aether:aether-impl:jar:1.0.0.v20140518:test

[INFO] |  |  +- org.eclipse.aether:aether-spi:jar:1.0.0.v20140518:test

[INFO] |  |  +- org.eclipse.aether:aether-util:jar:1.0.0.v20140518:test

[INFO] |  |  +- org.eclipse.aether:aether-connector-basic:jar:1.0.0.v20140518:test

[INFO] |  |  +- org.eclipse.aether:aether-transport-wagon:jar:1.0.0.v20140518:test

[INFO] |  |  +- org.apache.maven:maven-aether-provider:jar:3.2.5:test

[INFO] |  |  +- org.apache.maven:maven-model:jar:3.2.5:test

[INFO] |  |  +- org.apache.maven:maven-model-builder:jar:3.2.5:test

[INFO] |  |  |  \- org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:test

[INFO] |  |  +- org.apache.maven:maven-repository-metadata:jar:3.2.5:test

[INFO] |  |  +- org.apache.maven:maven-settings:jar:3.2.5:test

[INFO] |  |  +- org.apache.maven:maven-settings-builder:jar:3.2.5:test

[INFO] |  |  +- com.google.guava:guava:jar:18.0:test

[INFO] |  |  +- org.codehaus.plexus:plexus-interpolation:jar:1.21:test

[INFO] |  |  +- org.codehaus.plexus:plexus-utils:jar:3.0.20:test

[INFO] |  |  +- org.sonatype.plexus:plexus-sec-dispatcher:jar:1.3:test

[INFO] |  |  |  \- org.sonatype.plexus:plexus-cipher:jar:1.4:test

[INFO] |  |  +- org.apache.maven.wagon:wagon-provider-api:jar:2.6:test

[INFO] |  |  +- org.apache.maven.wagon:wagon-file:jar:2.6:test

[INFO] |  |  |  \- commons-lang:commons-lang:jar:2.6:test

[INFO] |  |  \- org.apache.maven.wagon:wagon-http-lightweight:jar:2.6:test

[INFO] |  |     \- org.apache.maven.wagon:wagon-http-shared:jar:2.6:test

[INFO] |  |        \- org.jsoup:jsoup:jar:1.7.2:test

[INFO] |  \- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-impl-maven-archive:jar:2.2.4:test

[INFO] |     +- org.jboss.shrinkwrap.resolver:shrinkwrap-resolver-spi-maven-archive:jar:2.2.4:test

[INFO] |     +- org.eclipse.sisu:org.eclipse.sisu.plexus:jar:0.3.0.M1:test

[INFO] |     |  \- org.eclipse.sisu:org.eclipse.sisu.inject:jar:0.3.0.M1:test

[INFO] |     \- org.codehaus.plexus:plexus-compiler-javac:jar:2.3:test

[INFO] |        \- org.codehaus.plexus:plexus-compiler-api:jar:2.3:test

[INFO] \- ws.ament.hammock:jwt-auth-tck:jar:1.0-SNAPSHOT:test

[INFO]    +- org.eclipse.microprofile.jwt:microprofile-jwt-auth-tck:jar:1.0-SNAPSHOT:test

[INFO]    \- ws.ament.hammock:dist-microprofile:jar:2.0-SNAPSHOT:test

[INFO]       +- ws.ament.hammock:bootstrap-weld3:jar:2.0-SNAPSHOT:test

[INFO]       |  +- org.jboss.logging:jboss-logging:jar:3.3.1.Final:test

[INFO]       |  +- javax.servlet:javax.servlet-api:jar:3.1.0:test

[INFO]       |  +- org.jboss.weld.se:weld-se-core:jar:3.0.0.Final:test

[INFO]       |  |  +- org.jboss.weld.environment:weld-environment-common:jar:3.0.0.Final:test

[INFO]       |  |  |  \- org.jboss.weld:weld-core-impl:jar:3.0.0.Final:test

[INFO]       |  |  |     +- org.jboss.classfilewriter:jboss-classfilewriter:jar:1.1.2.Final:test

[INFO]       |  |  |     \- org.jboss.spec.javax.interceptor:jboss-interceptors-api_1.2_spec:jar:1.0.0.Final:test

[INFO]       |  |  \- org.jboss.weld.probe:weld-probe-core:jar:3.0.0.Final:test

[INFO]       |  \- org.jboss.weld.servlet:weld-servlet-core:jar:3.0.0.Final:test

[INFO]       |     +- org.jboss.weld:weld-spi:jar:3.0.Final:test

[INFO]       |     |  \- org.jboss.weld:weld-api:jar:3.0.Final:test

[INFO]       |     +- org.jboss.weld.module:weld-jsf:jar:3.0.0.Final:test

[INFO]       |     +- org.jboss.weld.module:weld-web:jar:3.0.0.Final:test

[INFO]       |     \- org.jboss.spec.javax.el:jboss-el-api_3.0_spec:jar:1.0.7.Final:test

[INFO]       +- ws.ament.hammock:rest-cxf:jar:2.0-SNAPSHOT:test

[INFO]       |  +- org.apache.cxf:cxf-integration-cdi:jar:3.1.12:test

[INFO]       |  |  +- org.apache.cxf:cxf-core:jar:3.1.12:test

[INFO]       |  |  |  +- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:test

[INFO]       |  |  |  |  \- org.codehaus.woodstox:stax2-api:jar:3.1.4:test

[INFO]       |  |  |  \- org.apache.ws.xmlschema:xmlschema-core:jar:2.2.2:test

[INFO]       |  |  \- org.apache.cxf:cxf-rt-frontend-jaxrs:jar:3.1.12:test

[INFO]       |  +- org.apache.cxf:cxf-rt-rs-client:jar:3.1.12:test

[INFO]       |  +- org.apache.cxf:cxf-rt-transports-http:jar:3.1.12:test

[INFO]       |  \- ws.ament.hammock:web-spi:jar:2.0-SNAPSHOT:test

[INFO]       |     \- org.slf4j:slf4j-api:jar:1.7.25:test

[INFO]       +- ws.ament.hammock:web-undertow:jar:2.0-SNAPSHOT:test

[INFO]       |  +- io.undertow:undertow-servlet:jar:1.4.18.Final:test

[INFO]       |  |  +- io.undertow:undertow-core:jar:1.4.18.Final:test

[INFO]       |  |  |  +- org.jboss.xnio:xnio-api:jar:3.3.8.Final:test

[INFO]       |  |  |  \- org.jboss.xnio:xnio-nio:jar:3.3.8.Final:test

[INFO]       |  |  +- org.jboss.spec.javax.servlet:jboss-servlet-api_3.1_spec:jar:1.0.0.Final:test

[INFO]       |  |  \- org.jboss.spec.javax.annotation:jboss-annotations-api_1.2_spec:jar:1.0.0.Final:test

[INFO]       |  \- io.undertow:undertow-websockets-jsr:jar:1.4.18.Final:test

[INFO]       |     \- org.jboss.spec.javax.websocket:jboss-websocket-api_1.1_spec:jar:1.1.0.Final:test

[INFO]       +- ws.ament.hammock:hammock-core:jar:2.0-SNAPSHOT:test

[INFO]       |  +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.8.2:test

[INFO]       |  |  \- org.apache.logging.log4j:log4j-api:jar:2.8.2:test

[INFO]       |  +- org.apache.logging.log4j:log4j-core:jar:2.8.2:test

[INFO]       |  \- org.apache.logging.log4j:log4j-jul:jar:2.8.2:test

[INFO]       +- org.apache.johnzon:johnzon-jaxrs:jar:1.1.1:test

[INFO]       |  \- org.apache.johnzon:johnzon-mapper:jar:1.1.1:test

[INFO]       |     \- org.apache.johnzon:johnzon-core:jar:1.1.1:test

[INFO]       +- org.apache.geronimo.specs:geronimo-json_1.1_spec:jar:1.0:test

[INFO]       +- org.apache.geronimo.config:geronimo-config-impl:jar:1.1-SNAPSHOT:test

[INFO]       |  \- org.eclipse.microprofile.config:microprofile-config-api:jar:1.1-SNAPSHOT:test

[INFO]       \- ws.ament.hammock:security-jose:jar:2.0-SNAPSHOT:test

[INFO]          \- ws.ament.hammock:security-spi:jar:2.0-SNAPSHOT:test

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 1.424 s

[INFO] Finished at: 2017-08-08T10:29:27-07:00

[INFO] Final Memory: 22M/619M

[INFO] ------------------------------------------------------------------------

[tck 1173]$ 

[ERROR] testBadIssuer(org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest)  Time elapsed: 0.013 s  <<< ERROR!

java.lang.NoClassDefFoundError: javax/enterprise/inject/se/SeContainer

at org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest.testBadIssuer(TokenValidationTest.java:161)

Caused by: java.lang.ClassNotFoundException: javax.enterprise.inject.se.SeContainer

at org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest.testBadIssuer(TokenValidationTest.java:161)

Something must be pulling in 1.2 at runtime before the hammock TCK harness is loaded.

[John D. Ament]

That's not readable.  Can you just gist it?

[Werner Keil]

Do you expect the JWT functionality only to be ready once a Java EE equivalent stack was also used by MicroProfile?

At the moment CDI 2 is not the yet used.

[Werner Keil]

I mean Java EE 8 equivalent;-)

[sst...@redhat.com]

Without formatting. Something is pulling in the 1.2 CDI classes even tough it is marked as provided:

[tck 1182]$ mvn -Ptokens-se -Dtck.container.groupId=ws.ament.hammock -Dtck.container.artifactId=jwt-auth-tck -Dtck.container.version=1.0-SNAPSHOT dependency:tree

mvn -Ptokens-se -Dtck.container.groupId=ws.ament.hammock -Dtck.container.artifactId=jwt-auth-tck -Dtck.container.version=1.0-SNAPSHOT test

[ERROR] testBadIssuer(org.eclipse.microprofile.jwt.tck.parsing.TokenValidationTest)  Time elapsed: 0.014 s  <<< ERROR!

[sst...@redhat.com]

No, the 1.0 only requires EE 7.

[Werner Keil]

Judging from those logs, Hammoock must clearly be ahead of that, a Snapshot version of MP Config 1.1 as well as

 org.apache.geronimo.specs:geronimo-json_1.1_spec

I interpret that as the Geronimo flavor of JSON-P 1.1 which is also part of Java EE 8, not 7.

[John D. Ament]

You need to mark it as excluded in your pom, not provided.

[Mark Struberg]

The testng XML setup of mp-config is really similar to the one of Bean Validation, JBatch and CDI.
You need to run all the tests as shown in the docs.

[sst...@redhat.com]

I thought on today's mp hangout call you were saying that there were implementation being tested when one builds the current config tck. I'm not seeing that happen and don't see how it could with the current tck pom.xml. What were you mentioning on the call about MP Config TCK examples?

[John D. Ament]

Scott,

I think what Mark was referring to is how the Geronimo Config build works in ASF Jenkins

https://builds.apache.org/view/E-G/view/Geronimo%20Components%20and%20Specs/job/GeronimoConfig/org.apache.geronimo.config$geronimo-config-impl/90/testReport/

If you look we build against the config TCK on each of our commits.  This is typical for vendors, running some kind of test suite (the config TCK is so short it makes sense to run per commit, but other projects I would expect to run less frequently).

John

[sst...@redhat.com]

Ok, I see. So can we include a Examples/Samples section in the tck/running_the_tck.asciidoc that links to one or more implementation's setup such as https://svn.apache.org/repos/asf/geronimo/components/config/trunk/impl/ for running the current TCK?

All I want is for a user or potential implementor to be able to go run an existing implementation against the TCK so that they can have that as a starting point for switching to their preferred implementation or container.

[John D. Ament]

The pom contents to run the TCK from config are taken directly from https://github.com/eclipse/microprofile-config/blob/master/tck/running_the_tck.asciidoc so I'm not sure what you're aiming to do at this point.

John

[Scott Stark]

There there is a link to one or more working implementation such as https://svn.apache.org/repos/asf/geronimo/components/config/trunk/impl/ so that one can clearly see what the doc is describing.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/DAZzVp-YsXc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/4de7f7c4-000c-4d71-8d32-3a36920c66c4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gordon Hutchison]

I as part of developing implementations and tck tests I run bits from the Config and/or FaultTolerant TCKs many times a day. 

I tend to clone the MP project locally, then mvn clean install it into my local repo. As recommended I then run the TCK from

a separate project that references it in the POM.

(in my runner projects ./tck/pom.xml I currently use 

...

<dependency>

    <groupId>org.eclipse.microprofile.fault-tolerance</groupId>

    <artifactId>microprofile-fault-tolerance-api</artifactId>

    <version>1.0-SNAPSHOT</version>

</dependency>

<dependency>

    <groupId>org.eclipse.microprofile.fault-tolerance</groupId>

    <artifactId>microprofile-fault-tolerance-tck</artifactId>

    <version>1.0-SNAPSHOT</version>

    <scope>test</scope>

</dependency>

)

Another hack I have over https://github.com/eclipse/microprofile-config/blob/master/tck/running_the_tck.asciidoc

is I have: 

<plugin>

   <groupId>org.apache.maven.plugins</groupId>

      <artifactId>maven-surefire-plugin</artifactId>

         <version>2.17</version>

         <configuration>

            <suiteXmlFiles>

               <suiteXmlFile>${suiteXmlFile}</suiteXmlFile>

            </suiteXmlFiles>

            <testSourceDirectory>${basedir}/src/main/java/

            </testSourceDirectory>

         </configuration>

</plugin> 

Then I run specific tests with

 mvn test -DsuiteXmlFile=interestingTests.xml

Where these interestingTests.xml files can include/exclude packages/methods/classes like this:

<suite name="bulkhead-suite" verbose="2" configfailurepolicy="continue" >

<test name="bulkhead-suite">

<classes>

<class name="org.eclipse.microprofile.fault.tolerance.tck.bulkhead.BulkheadAsynchRetryTest">

   <methods>

                                        <include name="testBulkheadRetryClassAsynchronous55" />

                                </methods>

</class>

</classes>

</test>

</suite>

The other bits of the runner projects ./tck/pom.xml target local versions of our implementation (in Websphere Liberty)

and a slightly hacked https://github.com/arquillian/arquillian-container-was/tree/master/wlp-managed-8.5 that handles

the deployment exception tests from the Config TCK a bit better.

If anyone has any problems running the TCK, then feel free to post and I will help if I can.

Gordon.

[Werner Keil]

This sounds slightly similar to what was done for the JSR 363 TCK:

https://github.com/unitsofmeasurement/unit-tck

The only difference there is, abstracting the suite by offering a profile like "FULL" or "MINIMAL".

If customizing MicroProfile for different cases requires more than those 7 profiles, maybe a set of suites would also be fine.

Werner