Hi,
I'd like to raise awareness of two bugs I found in the AVX2-optimized ML-DSA signature verifier in the official Dilithium repository. Both bugs have now been fixed.
1. crypto_sign_verify_internal() in avx2/sign.c did not check that the absolute value of each coefficient in the vector z is less than GAMMA1 - BETA, as required by line 13 of FIPS 204 Algorithm 8, ML-DSA.Verify_internal. This caused the verifier to accept some invalid signatures. This bug was fixed by
https://github.com/pq-crystals/dilithium/commit/d35ba3fe5449bee3e6d43e1f296c3ca818bd36be2. use_hint_avx() in avx2/rounding.c did not correctly implement line 4 of FIPS 204 Algorithm 40, UseHint. It checked r0 < 0 instead of r0 <= 0. This caused the verifier to accept some invalid signatures. This bug was fixed by
https://github.com/pq-crystals/dilithium/commit/bba9534a3c5cbef66481cd6518d786e23ae601b5Projects that have copied this AVX2-optimized code should apply these fixes. The portable code in the ref/ directory is not affected.
I recommend that ML-DSA signature verifier implementations include tests for edge cases, including these. Wycheproof's ML-DSA test vectors already cover these edge cases. UseHint can also be unit-tested against all allowed (h, r) combinations, which would have detected the second bug.
It can also be effective to give an LLM two implementations (such as the ref/ and avx2/ implementations in this case) and ask it to review them for differences in behavior.
- Eric