Conformance test failures on Existing authenicator product

[Jay Huang]

Hello all,

In order to make sure I am using conformance tool properly, I uses Yubikey Bio to run FIDO Conformance Tools v1.7.7 || CTAP2.1 - MDS3.

In theory, Yubikey is certified with FIDO 2.1 so I think it should pass however I still found some failure items.

For example: for "Metadata tests", I have P-32 item failed.

TypeError: parseUVModality is not a function at n.eval (eval at compileCode (js/sandbox.js:25:26), <anonymous>:2510:37)

The metadata is derived from FIDO MDS Explorer and I put it at attachment for reference.

My guess is the version of test tool is already different but unsure. Any comment is appreicated :)

[nuno sung]

https://github.com/fido-alliance/conformance-test-tools-resources/issues/628#issuecomment-1017462352

Jay Huang 在 2023年3月9日 星期四下午1:53:32 [UTC+8] 的信中寫道：

[Jay Huang]

Thanks Nuno, looks like it is a bug still.

On the other hand, I test Trasports and Generic and has this failure

TypeError: Cannot read property '4' of undefined at n.eval (eval at compileCode (js/sandbox.js:25:26), <anonymous>:6464:28)

Looks like test tool doesn't receive property '4', which is the options. But I looked into the output from yubikey bio, there is the options data at getInfo() response, I put the wireshark log for reference.

I wonder if there is some misunderstanding?

[nuno sung]

I don't see this issue with tool v1.7.9 on Win10

Jay Huang 在 2023年3月9日 星期四下午5:02:17 [UTC+8] 的信中寫道：

[Jay Huang]

Hi Nuno,
Thanks! I have updated the tool with v1.7.9 and that failure no longer occur.

However when I continue the testing with MakeCredential Request, I found out multiple IPC Timeout. Does this happen at your side or do you know why this occurs?

My Yubikey bio F/W is 5.5.6

[nuno sung]

Hi Jay,

Some test items of MakeCredential request UP(User Presence) test, in this case you need to apply a finger touch on the sensor when seeing blinking led.

Jay Huang 在 2023年3月10日 星期五下午4:01:27 [UTC+8] 的信中寫道：

[Jay Huang]

Thanks Nuno,

Understood and I have passed most of the test items. There is one last failure item puzzles me.

At HMAC test, it expects CTAP1_ERR_SUCCESS after test tool sends MakeCredential command with hmac extensions. However yubikey bio reply with CTAP2_ERR_PUAT_REQUIRED.

After some digging into CTAP 2.1 spec, there are multiple possiblites for this error code, including 

1. pinUvAuthToken is false or not present. However it is set to true in metadata
   
2. If the authenticator is not protected by some form of user verification, but since pinUvAuthToken is true, then it should be protected.
   

I wonder if this failure also happen at your side? It is almost the last step for this topic and all comments are appreicated..

[nuno sung]

https://github.com/fido-alliance/conformance-test-tools-resources/issues/667

It's due to the key you uses has options.alwaysUv=true by default and I think it was test tool's compliant issue.

Jay Huang 在 2023年3月16日 星期四上午9:57:10 [UTC+8] 的信中寫道：

[John Bradley]

The bio key you have has alwaysUV true defaulting to true after a reset.  

The test tool was turning that off after each reset.  

There must have been a regression in the last year where that step got missed for some tests. 

The test tool developer has one of those keys so should have caught it. 

It is a bug in the current tool release. 

John B. 

Sent from my iPhone

On Mar 16, 2023, at 12:19 AM, nuno sung <nuno...@gmail.com> wrote:

https://github.com/fido-alliance/conformance-test-tools-resources/issues/667

--
You received this message because you are subscribed to the Google Groups "FIDO Dev (fido-dev)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fido-dev+u...@fidoalliance.org.
To view this discussion on the web visit https://groups.google.com/a/fidoalliance.org/d/msgid/fido-dev/27157151-b246-4697-9250-6b313e5c4191n%40fidoalliance.org.

[Jay Huang]

Thumbs up John and Nuno,

I have reported it and hopefully it will be fixed. https://github.com/fido-alliance/conformance-test-tools-resources/issues/702