When will test vectors for FIPS 203-205 be published?

[Scott Fluhrer (sfluhrer)]

In the past, NIST has been quite good at publishing extensive test vectors for approved algorithms.

 

Now, for FIPS 203, 204, 205, they have implemented ACVP, which automatically generates (and verifies) test vectors.

 

That is nice, however it would still be nice to have static test vectors:

 

• For preliminary testing of an implementation (before we hand it off to ACVP for formal testing)
• For generating the Known Answer Tests

 

Now, NIST has mentioned in their 8/14 announcement that:

 

“While test vectors will not be included in the three PQC FIPS, test vectors will be available on NIST's website.”

 

Do we have a timeframe for when they will be available?

[Celi, Christopher T. (Fed)]

Hi Scott,

 

Static vectors are posted to https://github.com/usnistgov/ACVP-Server/tree/master/gen-val/json-files for every algorithm ACVP supports. I’ll be working to make this link more apparent on the CSRC CAVP webpage.

 

Thanks,

Chris

 

--
You received this message because you are subscribed to the Google Groups "pqc-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.
To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CH0PR11MB5444D00819FC76B98A950470C18C2%40CH0PR11MB5444.namprd11.prod.outlook.com.

[Celi, Christopher T. (Fed)]

Right now, most of the ACVP test vectors are randomly generated. The code is open source in that repository, but a bit dense to see exactly how test cases are generated.

 

The CAVP has plans on improving testing for these algorithms over time, ideally before the end of the calendar year. We talk about the test procedures in our protocol specifications, https://pages.nist.gov/ACVP/draft-celi-acvp-ml-dsa.html#name-test-types-and-test-coverag.

 

Thanks,

Chris

 

From: Filippo Valsorda <fil...@ml.filippo.io>
Date: Tuesday, August 20, 2024 at 8:55 AM
To: Celi, Christopher T. (Fed) <christop...@nist.gov>, Scott Fluhrer (sfluhrer) <sflu...@cisco.com>, pqc-forum <pqc-...@list.nist.gov>
Subject: Re: [pqc-forum] Re: When will test vectors for FIPS 203-205 be published?

Hi Chris,

 

Thank you for publishing test vectors along with the final specs. Can you confirm whether the ACVP vectors are randomly generated, or whether they test dedicated edge cases? In other words, should I expect an equivalent level of test coverage if I use a different, equally large set of random vectors?

 

Thank you,

Filippo

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB7591F7479CA16AC0298121A7F08C2%40CO6PR09MB7591.namprd09.prod.outlook.com.

 

[Filippo Valsorda]

Hi Chris,

Thank you for publishing test vectors along with the final specs. Can you confirm whether the ACVP vectors are randomly generated, or whether they test dedicated edge cases? In other words, should I expect an equivalent level of test coverage if I use a different, equally large set of random vectors?

Thank you,

Filippo

2024-08-19 19:19 GMT+02:00 'Celi, Christopher T. (Fed)' via pqc-forum <pqc-...@list.nist.gov>:

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB7591F7479CA16AC0298121A7F08C2%40CO6PR09MB7591.namprd09.prod.outlook.com.

[Celi, Christopher T. (Fed)]

Yes, adding external interfaces with prehash/pure is on our agenda thanks to the feedback of the community. We’re not a large team within the CAVP. I’d like to get it out the door around October.

 

Thanks,

Chris

 

From: Robin Larrieu <robin....@cryptonext-security.com>
Date: Wednesday, August 21, 2024 at 5:25 AM
To: Celi, Christopher T. (Fed) <christop...@nist.gov>
Cc: pqc-forum <pqc-...@list.nist.gov>
Subject: Re: [pqc-forum] Re: When will test vectors for FIPS 203-205 be published?

Hi Chris,

Thank you again for the publication of ACVP test vectors.
I think the question has already been asked, but if there has been an answer I missed it, so let me reiterate: Is there any plan in the near future to handle Pure and PreHash modes within ACVP ?
It seems important that the test vectors exercise the different cases Pure/PreHash, with/without a context string (ideally with different lengths), to check that the "internal message" M' is constructed correctly in these different scenarii.

Currently, the ACVP test vectors for both ML-DSA and SLH-DSA are meant to use the Sign_internal/Verify_internal function. Because of this, implementations that choose not to expose this function (typically to prevent user errors / intentional misbehavior) cannot run ACVP test vectors. Moreover, since the input messages are entirely random, even implementations that do expose this function cannot run ACVP test vectors it they perform some format checking on their input (again to prevent user errors / intentional misbehavior).

At the very least, I think the messages used in the test vectors should correspond to valid "internal messages" M', for the different test cases. This way,
- implementations that perform input checking in Sign_internal/Verify_internal would work
- implementations that only expose the "external" functions can use a wrapper to undo the transformation (parse the message M' to split it into the context string, hash identifier (if any), and input message/digest) and call the appropriate external function
This solution has the advantage that it does not change the format of the JSON files, so it can be implemented pretty quickly as a workaround until an extension is designed for compatibility with external functions (if deemed relevant).

Best regards,
Robin Larrieu

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB75918636A51C4C232D0294B1F08D2%40CO6PR09MB7591.namprd09.prod.outlook.com.

 

[Markku-Juhani O. Saarinen]

Hi,

These were my findings after looking at the test vectors over last few days. I'm not sure if this is explained in NIST documentation:

I was pleasantly surprised to find that the test vectors seem to increase coverage by including e.g. malformed signatures for FIPS 204 and FIPS 205 that have been corrupted in various ways (there is a "reason" field in the internalProjection.json files explaining what is being tested, e.g. "too many hints" or "z too large"). Similarly there are "expect-fail" tests for FIPS 203 to exercise the Fujisaki-Okamoto implicit rejection logic. So the coverage is a bit better than with straightforward randomized test vectors. What is limiting the coverage right now is that only "*_internal" functions are being tested for 204 and 205; the padding logic for the actual signing and verification functions is not tested. In a CMUF meeting today NIST said that they're working on it.

I added Python implementations of FIPS 203,204,205 and a simple json parsing code here, so it's easy enough to explore the coverage in more detail: https://github.com/mjosaarinen/py-acvp-pqc

Cheers,

-markku

[Robin Larrieu]

Hi Chris,

Thank you again for the publication of ACVP test vectors.
I think the question has already been asked, but if there has been an answer I missed it, so let me reiterate: Is there any plan in the near future to handle Pure and PreHash modes within ACVP ?
It seems important that the test vectors exercise the different cases Pure/PreHash, with/without a context string (ideally with different lengths), to check that the "internal message" M' is constructed correctly in these different scenarii.

Currently, the ACVP test vectors for both ML-DSA and SLH-DSA are meant to use the Sign_internal/Verify_internal function. Because of this, implementations that choose not to expose this function (typically to prevent user errors / intentional misbehavior) cannot run ACVP test vectors. Moreover, since the input messages are entirely random, even implementations that do expose this function cannot run ACVP test vectors it they perform some format checking on their input (again to prevent user errors / intentional misbehavior).

At the very least, I think the messages used in the test vectors should correspond to valid "internal messages" M', for the different test cases. This way,
- implementations that perform input checking in Sign_internal/Verify_internal would work
- implementations that only expose the "external" functions can use a wrapper to undo the transformation (parse the message M' to split it into the context string, hash identifier (if any), and input message/digest) and call the appropriate external function
This solution has the advantage that it does not change the format of the JSON files, so it can be implemented pretty quickly as a workaround until an extension is designed for compatibility with external functions (if deemed relevant).

Best regards,
Robin Larrieu

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB75918636A51C4C232D0294B1F08D2%40CO6PR09MB7591.namprd09.prod.outlook.com.

[Ben Livelsberger]

I wanted to add to what Chris said.

• CAVP plans to add testing for the ML-DSA and SLH-DSA external sign and verify interfaces, but not for the external key generation interfaces.
  
• For the case of testing the non-deterministic variants of the ML-DSA.sign(), slh_sign() and hash_slh_sign() external interfaces, the CAVP tests will supply the additional randomness values. I.e., algorithm implementations will need to be able to accept values for rnd (ML-DSA.sign()) and addrnd (slh_sign() and hash_slh_sign()) provided by the CAVP tests.

-Ben

--

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+unsubscribe@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CH0PR11MB5444D00819FC76B98A950470C18C2%40CH0PR11MB5444.namprd11.prod.outlook.com.

 

--

You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+unsubscribe@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB7591F7479CA16AC0298121A7F08C2%40CO6PR09MB7591.namprd09.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+unsubscribe@list.nist.gov.

[Phillip Hallam-Baker]

On Mon, Aug 26, 2024 at 5:03 PM 'Ben Livelsberger' via pqc-forum <pqc-...@list.nist.gov> wrote:

I wanted to add to what Chris said.

• CAVP plans to add testing for the ML-DSA and SLH-DSA external sign and verify interfaces, but not for the external key generation interfaces.
  
• For the case of testing the non-deterministic variants of the ML-DSA.sign(), slh_sign() and hash_slh_sign() external interfaces, the CAVP tests will supply the additional randomness values. I.e., algorithm implementations will need to be able to accept values for rnd (ML-DSA.sign()) and addrnd (slh_sign() and hash_slh_sign()) provided by the CAVP tests.

-Ben

On Wednesday, August 21, 2024 at 11:12:18 AM UTC-4 Celi, Christopher T. (Fed) wrote:

Yes, adding external interfaces with prehash/pure is on our agenda thanks to the feedback of the community. We’re not a large team within the CAVP. I’d like to get it out the door around October.

 

Thanks,

Chris

That would be very useful.  

But not half as useful to me as finding that there is a large, unrestricted implementation of every algorithm I could expect to need in C#!

If I was starting today, I might think about using Rust but I started a decade ago when C# was the only real game in town for a modern object oriented language with strong typing and without IPR encumbrances. And it is pretty good except for having to write my own crypto algorithms most of the time because .NET crypto tends to lag behind.

Only thing I have to do now is upgrade to .Net 8.0 and extract the algorithms I need.

[Ben Livelsberger]

Enjoy! :)

[COSTA Graham]

THALES GROUP LIMITED DISTRIBUTION to email recipients

 

When CAVP adds the testing for the external sign and verify interfaces, will the standalone testing of the internal functions become optional or withdrawn?

 

i.e. Can we confirm that this update once implemented will remove the need for vendors to expose the internal functions for CAVP test purposes rather than simply being an addition?

 

Thanks,

 

Graham.

 

From: 'Ben Livelsberger' via pqc-forum <pqc-...@list.nist.gov>
Sent: Monday, August 26, 2024 10:03 PM
To: pqc-forum <pqc-...@list.nist.gov>
Cc: Celi, Christopher T. (Fed) <christop...@nist.gov>; pqc-forum <pqc-...@list.nist.gov>; Robin Larrieu <robin....@cryptonext-security.com>
Subject: Re: [pqc-forum] Re: When will test vectors for FIPS 203-205 be published?

 

I wanted to add to what Chris said.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CH0PR11MB5444D00819FC76B98A950470C18C2%40CH0PR11MB5444.namprd11.prod.outlook.com.

 

--

You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB7591F7479CA16AC0298121A7F08C2%40CO6PR09MB7591.namprd09.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB75918636A51C4C232D0294B1F08D2%40CO6PR09MB7591.namprd09.prod.outlook.com.

 

--

You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.
To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/51f5e9b8-f1a4-4341-bb4a-6bfb5f0c6e1dn%40list.nist.gov.

[Ben Livelsberger]

Hi Graham,

Chris and I discussed this yesterday. Our plan is to 

• offer testing for both internal and external ML-DSA and SLH-DSA sign and verify interfaces. 
  
• a lab/vendor may choose to test against one interface or the other
• CAVEAT: If an implementation supports the non-deterministic variants of the external ML-DSA or SLH-DSA sign interfaces, the implementation MUST be able to accept/use additional randomness values supplied by the CAVP tests in order to use the external sign interface testing. If the implementation cannot, the internal sign interface testing must be run.

Thanks,

Ben

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+unsubscribe@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CH0PR11MB5444D00819FC76B98A950470C18C2%40CH0PR11MB5444.namprd11.prod.outlook.com.

 

--

You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+unsubscribe@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB7591F7479CA16AC0298121A7F08C2%40CO6PR09MB7591.namprd09.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+unsubscribe@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB75918636A51C4C232D0294B1F08D2%40CO6PR09MB7591.namprd09.prod.outlook.com.

 

--
You received this message because you are subscribed to the Google Groups "pqc-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+unsubscribe@list.nist.gov.

[niux_d...@icloud.com]

Hi all!

I'd like to report that I've independently verified the current interim test vectors (from json files in the NIST ACVP-Server repo) for the internal routines. I've not committed and pushed the changes to GitHub yet since I don't want to put overly big files in my repo that's going to be replaced very soon. I intend to release my code when we're able to test publicly-exposed APIs.

Hope you find this helpful.
Pax~ DannyNiu/NJF.

> 2024年9月5日 22:05，'Ben Livelsberger' via pqc-forum <pqc-...@list.nist.gov> 写道：
>
> Hi Graham,
>
> Chris and I discussed this yesterday. Our plan is to
> •
> offer testing for both internal and external ML-DSA and SLH-DSA sign and verify interfaces.
> • a lab/vendor may choose to test against one interface or the other
> • CAVEAT: If an implementation supports the non-deterministic variants of the external ML-DSA or SLH-DSA sign interfaces, the implementation MUST be able to accept/use additional randomness values supplied by the CAVP tests in order to use the external sign interface testing. If the implementation cannot, the internal sign interface testing must be run.
> Thanks,
>
> Ben
>
> On Friday, August 30, 2024 at 9:31:36 AM UTC-4 COSTA Graham wrote:
> THALES GROUP LIMITED DISTRIBUTION to email recipients
> When CAVP adds the testing for the external sign and verify interfaces, will the standalone testing of the internal functions become optional or withdrawn?
> i.e. Can we confirm that this update once implemented will remove the need for vendors to expose the internal functions for CAVP test purposes rather than simply being an addition?
> Thanks,
> Graham.
> From: 'Ben Livelsberger' via pqc-forum <pqc-...@list.nist.gov>
> Sent: Monday, August 26, 2024 10:03 PM
> To: pqc-forum <pqc-...@list.nist.gov>

> Cc: Celi, Christopher T. (Fed) <christop...@nist.gov>; pqc-forum <pqc-...@list.nist.gov>; Robin Larrieu <robin....@cryptonext-security.com>

> Subject: Re: [pqc-forum] Re: When will test vectors for FIPS 203-205 be published?
> I wanted to add to what Chris said.
>
> • CAVP plans to add testing for the ML-DSA and SLH-DSA external sign and verify interfaces, but not for the external key generation interfaces.
> • For the case of testing the non-deterministic variants of the ML-DSA.sign(), slh_sign() and hash_slh_sign() external interfaces, the CAVP tests will supply the additional randomness values. I.e., algorithm implementations will need to be able to accept values for rnd (ML-DSA.sign()) and addrnd (slh_sign() and hash_slh_sign()) provided by the CAVP tests.
> -Ben
> On Wednesday, August 21, 2024 at 11:12:18 AM UTC-4 Celi, Christopher T. (Fed) wrote:
> Yes, adding external interfaces with prehash/pure is on our agenda thanks to the feedback of the community. We’re not a large team within the CAVP. I’d like to get it out the door around October. Thanks,
> Chris

> From: Robin Larrieu <robin....@cryptonext-security.com>

> To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

> To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CH0PR11MB5444D00819FC76B98A950470C18C2%40CH0PR11MB5444.namprd11.prod.outlook.com.
> --
> You received this message because you are subscribed to the Google Groups "pqc-forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

> To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB7591F7479CA16AC0298121A7F08C2%40CO6PR09MB7591.namprd09.prod.outlook.com.
> --
> You received this message because you are subscribed to the Google Groups "pqc-forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

> To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB75918636A51C4C232D0294B1F08D2%40CO6PR09MB7591.namprd09.prod.outlook.com.
> --
> You received this message because you are subscribed to the Google Groups "pqc-forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

> To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/51f5e9b8-f1a4-4341-bb4a-6bfb5f0c6e1dn%40list.nist.gov.
>

> --
> You received this message because you are subscribed to the Google Groups "pqc-forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.
> To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/b5775527-451c-4fed-b4e7-f091cc5d881bn%40list.nist.gov.

[niux_d...@icloud.com]

It's almost a month now, do we have any update?

[Livelsberger, Benjamin R. (Fed)]

Sure. CAVP is actively working on implementing this. We plan to have this testing available on our ACVTS Demo server and the sample json test vectors posted to https://github.com/usnistgov/ACVP-Server/tree/master/gen-val/json-files within the next month. 

Ben

[niux_d...@icloud.com]

I noticed SLH-DSA test vectors were updated "3 days ago" at the GitHub repo. Looks like we're making progress huh?

--
You received this message because you are subscribed to the Google Groups "pqc-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/E68F4B26-AEB7-44D0-BDF4-73F37869A62E%40nist.gov.

[Stephan Mueller]

Am Freitag, 8. November 2024, 14:22:37 Mitteleuropäische Normalzeit schrieb
niux_dannyniu via pqc-forum:

Hi niux_dannyniu,

> I noticed SLH-DSA test vectors were updated "3 days ago" at the GitHub repo.
> Looks like we're making progress huh?

To what I see it just got coverage of more hash / shake permutations. But it
does not yet cover the external functions.

Ciao
Stephan

[niux_d...@icloud.com]

I've been busy waiting like a spinlock for official test vectors for external routines for PQ-DSAs. Are we almost there now?

2024年10月22日 01:02，'Livelsberger, Benjamin R. (Fed)' via pqc-forum <pqc-...@list.nist.gov> 写道：

--
You received this message because you are subscribed to the Google Groups "pqc-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pqc-forum+...@list.nist.gov.

To view this discussion on the web visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/E68F4B26-AEB7-44D0-BDF4-73F37869A62E%40nist.gov.

[Celi, Christopher T. (Fed)]

Hi,

 

I’m still working on it. I’m adding several things at once here to our ML-DSA testing, so it has taken a bit longer than expected. I am hopeful to post to CSRC and this group early next week.

 

Thanks,

Chris Celi

[niux_d...@icloud.com]

ML-DSA, interesting. And SLH-DSA?

To view this discussion visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB75919635391B35A906958CF5F0212%40CO6PR09MB7591.namprd09.prod.outlook.com.

[Celi, Christopher T. (Fed)]

Because I’ll also be working on SLH-DSA it will be after I’m done with ML-DSA. FYI done with ML-DSA, I’ll be printing out some test vectors and sharing them here later today.

 

Thanks,

Chris

[Scott Fluhrer (sfluhrer)]

I just looked at the test vectors in https://github.com/usnistgov/ACVP-Server/tree/master/gen-val/json-files/SLH-DSA-sigGen-FIPS205 and there appears to be something missing:

 

• SLH-DSA takes a ‘context’ parameter – I don’t see where these test vectors include one.  Surely you want to test implementations with a non-empty context.
• Hash-SLH-DSA – I don’t see any test vectors for that

To view this discussion visit https://groups.google.com/a/list.nist.gov/d/msgid/pqc-forum/CO6PR09MB7591202DEE9CB8090111814CF02E2%40CO6PR09MB7591.namprd09.prod.outlook.com.

[Livelsberger, Benjamin R. (Fed)]

This is on the way. Updated test vectors should be available later this month. 

In case you missed it, this thread picks up where the current thread left off.

Ben