XTS does not get the expected result for non-byte-aligned NIST vectors

[yu li]

I intend to use Crypto++  XTS-AES for encrypting data.  I tested XTS-AES with the following NIST vector and got an unexpected CT= 54a8629d76db46d0c516fca52c9c903baa3a635ddd56f09760f63252c8b46140 .   

I call the following function:

XTS_Mode< AES >::Encryption e;

e.SetKeyWithIV(in_key, KeySize, in_iv, IVSize);

    StringSource(in_plain_txt,svSize(plain_txt, 1),true, 

new StreamTransformationFilter(e,

new StringSink(cipher),

                StreamTransformationFilter::NO_PADDING

) // StreamTransformationFilter

); // StringSource

My questions:

• Does  Crypto++  XTS-AES  not support non-byte aligned data encryption and decryption?
  

NIST vector :

COUNT = 301

DataUnitLen = 250

Key = d6b58f6638f64c3692a5b53671c5d51e0abf5169437aacb363fbcfcb91f9f6b0f536d0e12dd662d3151a77524eae5ad5e1b9cf860199981edd09ccf99f86fcd2

i = f5bf2260198883d03b133810fafec16c

PT = 8b81371f87661e49b904c8272def606717c3050c125905d23a2f0c142132ee00

CT = 84a7f42be588fa36442d33f098ec56e554a8629d76db46d0c516fca52c9c9000

[Jeffrey Walton]

On Wednesday, May 19, 2021 at 11:53:54 PM UTC-4 yu...@nvidia.com wrote:

I intend to use Crypto++  XTS-AES for encrypting data.  I tested XTS-AES with the following NIST vector and got an unexpected CT= 54a8629d76db46d0c516fca52c9c903baa3a635ddd56f09760f63252c8b46140 .   

I call the following function:

XTS_Mode< AES >::Encryption e;

e.SetKeyWithIV(in_key, KeySize, in_iv, IVSize);

    StringSource(in_plain_txt,svSize(plain_txt, 1),true, 

new StreamTransformationFilter(e,

new StringSink(cipher),

                StreamTransformationFilter::NO_PADDING

) // StreamTransformationFilter

); // StringSource

My questions:

• Does  Crypto++  XTS-AES  not support non-byte aligned data encryption and decryption
  

XTS mode should support non-aligned data. For SSE, it uses _mm_loadu_si128 and _mm_storeu_si128, which are unaligned loads and stores. The class also uses 'GetWord<word64>(false, LITTLE_ENDIAN_ORDER, ...)' and 'PutWord<word64>(false, LITTLE_ENDIAN_ORDER, ...)'. The 'false' says the data is not aligned, so a memcpy is used. Also see https://github.com/weidai11/cryptopp/blob/master/xts.cpp.

We've seen some trouble with bad code generation when two pointers happen to be the same in strcipher.cpp. But I don't believe XTS uses it. Also see https://github.com/weidai11/cryptopp/issues/1010.

Do you have a reproducer?

Jeff

[Jeffrey Walton]

Something else that might be a factor... The head notes in xts.h (https://github.com/weidai11/cryptopp/blob/master/xts.h) says:

    /// \details XTS mode is a wide block mode defined by IEEE P1619-2008. NIST
    /// SP-800-38E approves the mode for storage devices citing IEEE 1619-2007.
    /// IEEE 1619-2007 provides both a reference implementation and test vectors.
    /// The IEEE reference implementation fails to arrive at the expected result

    /// for some test vectors.

That's a bad sign :(

I keep the programs that are used to generate test vectors so we can always establish provenance if needed. The program is located in one of my testing GitHubs. For XTS, it is located at https://github.com/noloader/cryptopp-test.

The test vectors we use are located at https://github.com/weidai11/cryptopp/blob/master/TestVectors/xts.txt. Looking through the test vector, it looks like we use both the XTS test vectors, the XTS reference implementation and Botan. Botan is Jack Lloyd's Botan (https://github.com/randombit/botan), and we use to to add additional test vectors when we feel there are gaps.

Jeff

[yu li]

Thank you for your reply.

All test vector in https://github.com/weidai11/cryptopp/blob/master/TestVectors/xts.txt.  are byte-aligned . I've tested these vectors locally, and they're all pass. But it doesn't have a non-byte-aligned vector like datalen=250bit in https://github.com/weidai11/cryptopp/blob/master/TestVectors/xts.txt.  We can found non-byte-aligned vector in https://csrc.nist.gov/Projects/Cryptographic-Algorithm-Validation-Program/CAVP-TESTING-BLOCK-CIPHER-MODES#XTS .  All byte-aligned vector in https://csrc.nist.gov/Projects/Cryptographic-Algorithm-Validation-Program/CAVP-TESTING-BLOCK-CIPHER-MODES#XTS can pass， but all non-byte-aligned vector are fail.

[Jeffrey Walton]

On Thu, May 20, 2021 at 4:02 AM yu li <yu...@nvidia.com> wrote:
>
> Thank you for your reply.
>
> All test vector in https://github.com/weidai11/cryptopp/blob/master/TestVectors/xts.txt. are byte-aligned . I've tested these vectors locally, and they're all pass. But it doesn't have a non-byte-aligned vector like datalen=250bit in https://github.com/weidai11/cryptopp/blob/master/TestVectors/xts.txt. We can found non-byte-aligned vector in https://csrc.nist.gov/Projects/Cryptographic-Algorithm-Validation-Program/CAVP-TESTING-BLOCK-CIPHER-MODES#XTS . All byte-aligned vector in https://csrc.nist.gov/Projects/Cryptographic-Algorithm-Validation-Program/CAVP-TESTING-BLOCK-CIPHER-MODES#XTS can pass， but all non-byte-aligned vector are fail.

My bad Yu. I took alignment to mean type alignment. Sorry about that.

XTS mode in Crypto++ is byte oriented, not bit oriented. We could
possibly change that.

I'm looking at https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/aes/XTSVS.pdf,
and it says (from p. 6):

An implementation may support a data unit length that is not a
multiple of 8 bits. In this case, the plaintext (PT) and ciphertext
(CT) will be represented in the request, sample, and response files by
a bit string padded with zeros on the right to the next byte boundary,
in hexadecimal. For example, suppose an implementation supports a 137
bit data unit. The first 128-bit block consists of the first (i.e.,
leftmost) 128 bits. If the second, nine-bit partial block is
011011011, then in the request and sample files it will be padded with
seven zeros on the right – 0110 1101 1000 0000 – and represented as
6d80 (hex). Response files values should be formatted the same way.

The values seem non-sensical to me. Where did 137-bit data unit come
from? That's going to be a sector size in real life, so its going to
be 512, 2048, 4096, 64k, etc.

Jeff

[yu li]

 Will you support bit-oriented XTS in the next release？

The values seem non-sensical to me. Where did 137-bit data unit come
from? That's going to be a sector size in real life, so its going to
be 512, 2048, 4096, 64k, etc.

>>> In practice, there seems to be no such use case.