Compatibility Issues with OpenSSL

[Siddharth Dash]

Hello All,

We are currently integrating OpenSSL with Windows Schannel SSPI in a bidirectional setup—where OpenSSL acts both as a client and a server, and vice versa.

Our observations are as follows:

• When using the SCHANNEL_CREDENTIALS structure with Schannel, the implementation works as expected—successfully performing the TLS handshake, data encryption, and decryption without issues.

• However, when switching to the SCH_CREDENTIALS structure, we encounter an unexpected behavior:

  • The TLS handshake completes successfully, indicating that the initial authentication works.

  • Yet, data encryption and decryption operations fail, preventing secure communication post-handshake.

This discrepancy suggests that while SCH_CREDENTIALS supports the initial authentication phase, it might lack certain configurations required for subsequent cryptographic operations.

Request for Assistance:
Has anyone encountered a similar issue when using SCH_CREDENTIALS with OpenSSL? If so, could you provide insights or recommendations on how to resolve this? Possible considerations include:

• Required flags or settings in SCH_CREDENTIALS for enabling encryption/decryption.

• Differences in behavior between SCHANNEL_CREDENTIALS and SCH_CREDENTIALS in this context.

• Any known compatibility issues between Schannel’s credential structures and OpenSSL.

We appreciate any guidance or references to documentation that could help resolve this problem.

--

Regards,
Siddharth

[Viktor Dukhovni]

On Sat, Jul 12, 2025 at 01:24:39PM +0530, Siddharth Dash wrote:

> We are currently integrating *OpenSSL* with *Windows Schannel SSPI* in a
> bidirectional setup—where OpenSSL acts both as a *client* and a *server*,
> and vice versa.

By "SCHANNEL credentials", are you referring to the private key and
certificate used by either end of the connection, or to some other form
of credentials used as a PSK? OpenSSL has no code that deals with
SCHANNEL. Since this is the OpenSSL users group, your request for help
is best framed in terms of objects actually supported by the OpenSSL
API, or you at least need to explain clearly how you're adapting
Schannel objects for use in OpenSSL.

> When using the *SCHANNEL_CREDENTIALS* structure with Schannel, the

> implementation works as expected—successfully performing the TLS handshake,
> data encryption, and decryption without issues.

Since OpenSSL has no support for SCHANNEL_CREDENTIALS, you need to
explain what you mean.

> However, when switching to the *SCH_CREDENTIALS* structure, we encounter
> an unexpected behavior:
> -

Since OpenSSL has no support for SCH_CREDENTIALS, you need to explain
what you mean.

> The *TLS handshake completes successfully*, indicating that the
> initial authentication works.
> -
>
> Yet, *data encryption and decryption operations fail*, preventing
> secure communication post-handshake.

OpenSSL supports configuring a client or server SSL_CTX (context) or SSL
(connection) handle with one or more keypairs (private key and
certificate chain). If you're actually using OpenSSL to implement a TLS
connection, how onbtained the key material is immaterial, Schannel is
not actually used by OpenSSL, even if it was the source of the key
material. If you're using Schannel API to complete the TLS handshake,
then you're not using OpenSSL.

--
Viktor.

[Siddharth Dash]

Hello Community Users,

In our secure communication setup between a Windows Server 2022 client using SCHANNEL and a Red Hat Enterprise Linux 8 server running OpenSSL 1.1, the TLS handshake was successfully completed, indicating that both systems initially agreed on encryption parameters. Following the handshake, the Windows client encrypted and transmitted a message, which the Linux server successfully decrypted. The Linux server then responded with its own encrypted message. However, upon receiving this response, the Windows client encountered a decryption failure, returning the error: DecryptMessage: SEC_E_MESSAGE_ALTERED. Simultaneously, the Linux server logged an OpenSSL error: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:ssl/record/ssl3_record.c:677:.

Regards,

Siddharth

--
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-user...@openssl.org.
To view this discussion visit https://groups.google.com/a/openssl.org/d/msgid/openssl-users/aHI_L6iXYclu64T2%40chardros.imrryr.org.

--

Regards,
Siddharth