ruby-openssl + provider interface: generating keys with provider

[Michael Richardson]

I want to access providers from ruby (and ultimately, rails)
(I care about using an IDevID provisioned into a TPM2 device to sign
certain artifacts/claims. The details are not that important, unicast if you
like)
I expect the private key to remain in the TPM, rather than be loaded
from an encrypted storage, but that could change in the future.

ruby-openssl now works with openssl3, although I have CMS patches I am trying
to upstream. I've been looking at how it loads private keys.
Neither ruby-openssl, nor I think many of the openssl "apps" actually have
been updated to use the openssl 3.x OSSL_STORE*, so they are not actually
able to create or use EVP_PKEY whose keys never leave a secure element.

ruby-openssl now has an OpenSSL::Provider interface, and it can load the
tpm2-openssl provider. But that's about all.

I added an OpenSSL::PKEY.load_from_handle function that calls
OSSL_STORE_open, OSSL_STORE_load, and OSSL_STORE_INFO_get1_PKEY(store).
It seems to work. It would be nice if the man page for OSSL_STORE_open had
more information on ui_method, and post_process.
I concluded I didn't need them, and set them to NULL.
I also found code at:
https://github.com/latchset/pkcs11-provider/blob/f9b04f9e7e200be5e82cf8283a7fb9c4ead0dd37/tests/tcmpkeys.c#L40
that does this (all NULL), but it could be wrong too.

After some more futzing around, fixing things where openssl was not involved,
I was able to *use* a key stored in my [test] (sw)tpm.

(I would suggest that maybe the documentation aimed at provider authors, such
as provider(7), be more clearly separated from provider users. Not sure how
to do that)

Now, I had to create the keys using tpm2_createak and tpm2_evictcontrol.
I don't really like that. I think that the provider interface supports
generating keys, but I don't know how to get it to do that!

I feel that perhaps ruby's generate_parameters, which calls
EVP_PKEY_CTX_new_from_name might let me set the provider explicitely, if I
knew what string to use. Getting the right things out to specify RSA/EC,
curves, bit sizes, etc. was not well explained, and I have patches to
ruby-openssl to update the documentation. That it has to reference the help
page from "openssl genpkey" rather than a somepage(3) seems a documentation bug.

And then there is: EVP_PKEY_CTX *EVP_PKEY_CTX_new_id(int id, ENGINE *e);
but, it mentions ENGINE, not Provider. Not sure what to think. It's not
marked obsolete either.

new_from_{dispatch,name}() takes OSSL_LIB_CTX, but I'm rather unclear if it's for me.
It says:
OSSL_LIB_CTX_new_from_dispatch() creates a new OpenSSL library context
initialised to use callbacks from the OSSL_DISPATCH structure. This is
**PRIMARILY** USEFUL FOR PROVIDER AUTHORS. The handle and dispatch

Maybe secondarily used by others?
Unclear.

Then, we have:
OSSL_PROVIDER_get0_provider_ctx() returns the provider context for the
given provider. The provider context is an opaque handle set by the
provider itself and is passed back to the provider by libcrypto in
various function calls.

but *it* returns a void*, not an OSSL_LIB_CTX. Maybe it's a typo?
A void* could point at an OSSL_LIB_CTX, but it would be wrong that wasn't expected.

From a ruby point of view, what would make sense to me is to load a provider,
and then from the provider object, ask it to produce an EVP_PKEY_CTX.
I'd ask then ask that object to generate an appropriate key, returning an EVP_PKEY.
(Since EVP_PKEY's are supposed to be immutable)
Right now, there is a generate_parameters which returns an **EVP_PKEY**,
[not a new object], and then that is used to generate a key, or it can all be
one step. It is kinda confused, because it tries to handle 1.x and 3.x flow.

I have confirmed that nothing magically uses the tpm2 provider for
generation:
The key I loaded from the TPM handle:
#<OpenSSL::PKey::RSA:0x00007fbe6986ede8 oid=rsaEncryption type_name=RSA provider=tpm2>

The key I generated:
#<OpenSSL::PKey::EC:0x00007fbe6986ea78 oid=id-ecPublicKey type_name=EC provider=default>

(I know one is RSA and one is EC. Just a test. TPM emulator should support
both)

--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | IoT architect [
] m...@sandelman.ca http://www.sandelman.ca/ | ruby on rails [

[Michael Richardson]

Hi, the ruby-openssl code has a check:
EVP_PKEY *
GetPrivPKeyPtr(VALUE obj)

which is used in the .sign function.
It throws an exception if one tries to do a signature without a private key.
Alas, it checks a bunch of internal things, but tries things like
RSA_get0_key() if it's an RSA key, etc. (Since ruby has OpenSSL::PKey::{RSA,EC,..})
This clearly is WRONG in a provider world where a private key might be kept
safely away from muggles like me.

Looking for man pages, and then evp.h, and store.h, to try to learn if there
is a way to understand if an EVP_PKEY has a private key associated with it.
I found nothing, but I could perhaps see if it was loaded via a provider, but
I think that a truism now, because the default provider is still a provider.

Okay, step back.
What we really want to know is: can this EVP_PKEY perform a signature operation.
It seems that perhaps I should just skip the tests that GetPrivPKeyPtr does,
and just depend upon EVP_DigestSignInit() failing if it can't be done?

EVP_DigestSignInit() still has ENGINE* argument.
While EVP_DigestSignInit_ex() has OSSL_LIB_CTX.
I find this confusing!

[Viktor Dukhovni]

On Tue, Aug 12, 2025 at 09:15:08PM -0400, Michael Richardson wrote:

> I want to access providers from ruby (and ultimately, rails) (I care
> about using an IDevID provisioned into a TPM2 device to sign certain
> artifacts/claims. The details are not that important, unicast if you
> like) I expect the private key to remain in the TPM, rather than be
> loaded from an encrypted storage, but that could change in the future.

I am not convinced you really **want** to explicitly "access providers".
Rather, you likely want to have some operations (key generation, loading
a key, signing, ...) performed by a chosen provider. And for that,
appropriate arguments to mainstream EVP APIs should generally be sufficient,
if the desired provider is loaded.

Some relevant text from the openssl-glossary(7) manpage:

Property Query String
A property query string is a string containing a sequence of
properties that can be used to select an algorithm implementation.
For example the query string "provider=example,foo=bar" will select
algorithms from the "example" provider that have a "foo" property
defined for them with a value of "bar".

Property Query Strings are used during fetching. See Fetching.

property(7)

with more detailed text in property(7):

DESCRIPTION
As of OpenSSL 3.0, a new method has been introduced to decide which of
multiple implementations of an algorithm will be used. The method is
centered around the concept of properties. Each implementation defines a
number of properties and when an algorithm is being selected, filters
based on these properties can be used to choose the most appropriate
implementation of the algorithm.

...

> I also found code at:
> https://github.com/latchset/pkcs11-provider/blob/f9b04f9e7e200be5e82cf8283a7fb9c4ead0dd37/tests/tcmpkeys.c#L40
> that does this (all NULL), but it could be wrong too.

A PKCS#11 provider may well be a appropriate interface to your TPM, but
if you have an adequate dedicated provider that should be fine too. The
main different will be in how keys are named.

> I feel that perhaps ruby's generate_parameters, which calls
> EVP_PKEY_CTX_new_from_name might let me set the provider explicitely, if I
> knew what string to use.

I'd start with:

const char *propq = "provider=tpm2";
const char *key_algor = "EC";
OSSL_LIB_CTX *dflt_lctx = NULL;
EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new_from_name(dflt_lctx, key_algor, propq);
EVP_PKEY *pkey = NULL;
OSSL_PARAM params[...];

/*
* Set any required algorithm and/or provider-specific required
* keygen parameters. For EC you'd need at least the curve (a.k.a.
* group) name. For RSA, you'd need the modulus bit length, ...
*/
params[0] = OSSL_PARAM_construct_...(...);
...
params[...] = OSSL_PARAM_construct_...(...);
params[...] = OSSL_PARAM_construct_end();

if (ctx == NULL) {
... error handling ...;
}
if (EVP_PKEY_keygen_init(ctx) <= 0) {
... error handling ...;
}
if (EVP_PKEY_CTX_set_params(ctx, params) != 1) {
... error handling ...;
}
if (EVP_PKEY_generate(ctx, &pkey) <= 0) {
... error handling ...;

}

> Getting the right things out to specify RSA/EC,
> curves, bit sizes, etc. was not well explained, and I have patches to
> ruby-openssl to update the documentation.

The relevant parameters should be specified in the algorithm and/or
provider docs. If the provider emulates an equivalent "default"
provider interface, than the algorithm docs should suffice, but
otherwise, the provider docs need to explain what the provider expects.

> I have confirmed that nothing magically uses the tpm2 provider for
> generation:
> The key I loaded from the TPM handle:
> #<OpenSSL::PKey::RSA:0x00007fbe6986ede8 oid=rsaEncryption type_name=RSA provider=tpm2>

> [...]

> The key I generated:
> #<OpenSSL::PKey::EC:0x00007fbe6986ea78 oid=id-ecPublicKey type_name=EC provider=default>

It seems your propq should be at least "provider=tpm2".

For relevant key generation parameters see:

https://github.com/tpm2-software/tpm2-openssl/blob/master/docs/keys.md

On Fri, Aug 22, 2025 at 05:07:29PM -0400, Michael Richardson wrote:

> Looking for man pages, and then evp.h, and store.h, to try to learn if there
> is a way to understand if an EVP_PKEY has a private key associated with it.
> I found nothing, but I could perhaps see if it was loaded via a provider, but
> I think that a truism now, because the default provider is still a provider.

Use of existing keys in a hardware device typically requires accessing a
named object in a logical key container associated with that device. So
typically some sort of store URI. This is where PKCS#11 can help
abstract device-specifics, but you can also use the "tpm2" store support

The below CLI example from
<https://github.com/tpm2-software/tpm2-openssl/blob/master/docs/keys.md>:

openssl rsa -provider tpm2 -modulus -noout -in handle:0x81000000

corresponds to loading the "tpm2" "handle" scheme as described in
OSSL_STORE_open_ex(3):

const OSSL_PARAM params[1] = { OSSL_PARAM_END }; /* If none needed */
const char *propq = "provider=tpm2";
const char *key_algor = "EC";
OSSL_LIB_CTX *dflt_lctx = NULL;
OSSL_STORE_CTX *store_ctx =
OSSL_STORE_open_ex("handle:0x81000000, dflt_ctx, propq,
NULL, NULL, /* assuming no passwd required */
params, NULL, NULL);

then use the store to iterate through the returned objects.

--
Viktor. 🇺🇦 Слава Україні!

[Tomas Mraz]

There is a EVP_PKEY_can_sign(const EVP_PKEY *pkey) function which will
give you answer, whether the key type can be used for signing. However
it won't check whether there actually is a private key held by the pkey
object.

We do have a provider API call (the has() function) to discover that
but that is not directly exposed via EVP API. Furthermore there might
be other reasons (in 3rd party providers) why a key cannot be used for
signing such as access rights or similar.

You can use EVP_PKEY_can_sign() as a sort of pre-check but then you
still need to check the result of EVP_DigestSignInit() which can only
give you the definitive answer.

> EVP_DigestSignInit() still has ENGINE* argument.
> While EVP_DigestSignInit_ex() has OSSL_LIB_CTX.
> I find this confusing!

The EVP_DigestSignInit() function pre-dates the providers and library
contexts, but it will still work with provider-based keys (with the
default library context).

EVP_DigestSignInit_ex() has to be used if you need non-default library
context.

--
Tomáš Mráz, Public Support and Security Manager, OpenSSL Foundation
Join the Code Protectors or support us on Github Sponsors
https://openssl-foundation.org/donate/

[Viktor Dukhovni]

On Fri, Aug 22, 2025 at 05:07:29PM -0400, Michael Richardson wrote:

> Looking for man pages, and then evp.h, and store.h, to try to learn if there
> is a way to understand if an EVP_PKEY has a private key associated with it.
> I found nothing, but I could perhaps see if it was loaded via a provider, but
> I think that a truism now, because the default provider is still a provider.

The question of whether the algorithm of given key's type supports
creating and verifying signatures is answered by EVP_PKEY_can_sign(3):

EVP_PKEY_can_sign() checks if the functionality for the key type of
*pkey* supports signing. No other check is done, such as whether *pkey*
contains a private key.

But it should be noted that this will return true regardless of whether
the key handle has any of, either or both a private and public
component, so you are expected to already know whether what you have is
or is not a private key. Whatever method you used to construct the key
would typically have implied or revealed the key type.

There should perhaps be a public API around the internal
evp_pkey_util_has() function:

crypto/evp/keymgmt_lib.c:int evp_keymgmt_util_has(EVP_PKEY *pk, int selection)
crypto/evp/keymgmt_lib.c-{
crypto/evp/keymgmt_lib.c- /* Check if key is even assigned */
crypto/evp/keymgmt_lib.c- if (pk->keymgmt == NULL)
crypto/evp/keymgmt_lib.c- return 0;
crypto/evp/keymgmt_lib.c-
crypto/evp/keymgmt_lib.c: return evp_keymgmt_has(pk->keymgmt, pk->keydata, selection);
crypto/evp/keymgmt_lib.c-}

--
Viktor. 🇺🇦 Слава Україні!

[Michael Richardson]

Tomas Mraz <to...@openssl.org> wrote:
>> Okay, step back. What we really want to know is: can this EVP_PKEY
>> perform a signature operation. It seems that perhaps I should just
>> skip the tests that GetPrivPKeyPtr does, and just depend upon
>> EVP_DigestSignInit() failing if it can't be done?

> There is a EVP_PKEY_can_sign(const EVP_PKEY *pkey) function which will
> give you answer, whether the key type can be used for signing. However
> it won't check whether there actually is a private key held by the pkey
> object.

I don't care if there is a literal private key.
In fact, it's exactly the opposite, since a tpm, secure-element, HSM,
etc. won't have that, as the key is elsewhere... but it can sign.

> We do have a provider API call (the has() function) to discover that
> but that is not directly exposed via EVP API. Furthermore there might
> be other reasons (in 3rd party providers) why a key cannot be used for
> signing such as access rights or similar.

Yes.

> You can use EVP_PKEY_can_sign() as a sort of pre-check but then you
> still need to check the result of EVP_DigestSignInit() which can only
> give you the definitive answer.

Thank you, I think that's what I'm looking for, but reading the man page, I'm

still a bit unclear. It says:
EVP_PKEY_can_sign() checks if the functionality for the key type of

pkey supports signing. No other check is done, such as whether pkey
contains a private key.

which suggests to me that it would work for any RSA key, but fail for any ECDH
key: just based upon the type of the key.

>> EVP_DigestSignInit() still has ENGINE* argument. While
>> EVP_DigestSignInit_ex() has OSSL_LIB_CTX. I find this confusing!

> The EVP_DigestSignInit() function pre-dates the providers and library
> contexts, but it will still work with provider-based keys (with the
> default library context).

> EVP_DigestSignInit_ex() has to be used if you need non-default library
> context.

I really would prefer that the EVP_PKEY provided the context.
I don't see why my application/library should need to know this.
From what I can tell from reading code, it seems like it's the MD's context
that wins.

[Tomas Mraz]

On Tue, 2025-08-26 at 18:32 -0400, Michael Richardson wrote:
>     >> EVP_DigestSignInit() still has ENGINE* argument.  While
>     >> EVP_DigestSignInit_ex() has OSSL_LIB_CTX.  I find this
> confusing!
>
>     > The EVP_DigestSignInit() function pre-dates the providers and
> library
>     > contexts, but it will still work with provider-based keys (with
> the
>     > default library context).
>
>     > EVP_DigestSignInit_ex() has to be used if you need non-default
> library
>     > context.
>
> I really would prefer that the EVP_PKEY provided the context.
> I don't see why my application/library should need to know this.
> From what I can tell from reading code, it seems like it's the MD's
> context
> that wins.

This is a misunderstanding. The concept of the OSSL library context is
inherently something that the applications or libraries should care
about as this provides the necessary separation of unrelated uses of
OpenSSL within the whole application.

Let's imagine your application wants to use OpenSSL to handle CMS
messages. You want to imply some particular settings to which providers
are loaded, what algorithms are allowed, etc. However to transport
these messages the application also uses libcurl with its own set of
constraints on OpenSSL usage and you do not want your settings for
handling the CMS to affect the libcurl operation.

For that reason we've introduced OSSL_LIB_CTX so libcurl can use one
context and the application another and they won't inadvertently affect
operation of each other.

[Michael Richardson]

Viktor Dukhovni <openss...@dukhovni.org> wrote:
>> I want to access providers from ruby (and ultimately, rails) (I care
>> about using an IDevID provisioned into a TPM2 device to sign certain
>> artifacts/claims. The details are not that important, unicast if you
>> like) I expect the private key to remain in the TPM, rather than be
>> loaded from an encrypted storage, but that could change in the future.

> I am not convinced you really **want** to explicitly "access
> providers". Rather, you likely want to have some operations (key
> generation, loading a key, signing, ...) performed by a chosen
> provider.

I don't know if there is semantic subtlety you are trying to get at, but I've
missed it. This seems to be semantic pedantry to me. I'm happy to be wrong.

I think in 60% of the cases, people building bespoke systems in C, Rust, ruby,
python, Perl, ... know exactly what they want, and they are mostly targetting
a specific provider.
Yes, they might want some agility in moving to a different provider, but they
will need to test against those anyway. Their "customers", if they are
external, will be told which ones work, and that others are "who knows" until
tested. For me, this query string is not only unwanted, but could even be dangerous.

In the other 40% of cases, the systems have been built for others to use, and
the query string has some significant value because, the systems are not open
source (or rather, then people installing them are not installing from
source), so a way for the operator to specify the specific provider, and
maybe even some parameters to the provider, such as an IP address (in the
case of an big-iron HSM) to connect to.

I'm working on a bespoke system written RubyOnRails and Rust (different pieces),
and to do that I'm trying to make sure that the ruby-openssl library can do
what I want. I'd like to benefit as many people as possible, but the query
string business is not particularly useful to me.

> Property Query String A property query string is a string
> containing a sequence of properties that can be used to select an
> algorithm implementation. For example the query string
> "provider=example,foo=bar" will select algorithms from the "example"
> provider that have a "foo" property defined for them with a value of
> "bar".

Yes, but I shouldn't have to throw these strings all over the place.
Once is enough.

I've traced into crypto/evp/m_sigver.c, to:
static int do_sigver_init(EVP_MD_CTX *ctx, EVP_PKEY_CTX **pctx,

It's a very complex function. It really deserves some unit tests.
I found that the loop gets it wrong. An EVP_MD_CTX has been created a few
layers up based upon the EVP_PKEY(_CTX), which has a sensible keymgmt
pointing at the provider which has access to the private key.
Picking ANY OTHER provider at this point would be stupid, yet the loop
manages to do exactly that.
The loop starting after the comments at line 116:
for (iter = 1, provkey = NULL; iter < 3 && provkey == NULL; iter++) {

only runs once. I don't think it should exit the loop when provkey!=NULL.
Rather, I think it should run all possibilities, and then it should pick the
*best* one, which is not necessarily the first thing. It says:

* We then try to fetch the keymgmt from the same provider as the
* signature, and try to export |ctx->pkey| to that keymgmt (when
* this keymgmt happens to be the same as |ctx->keymgmt|, the export
* is a no-op, but we call it anyway to not complicate the code even
* more).

but, it doesn't do that. Instead, on the first iter=1, it does:
signature = EVP_SIGNATURE_fetch(locpctx->libctx, supported_sig,
locpctx->propquery);

despite libctx and propquery being NULL, so of course, it finds the default
provider. I added:

if(locpctx->libctx || locpctx->propquery) {
}

around that call, which allowed the second iteration to run, which then did
what the comment above promised.
The correct signature routine was called, and the key was available.

I doubt my fix is correct, but without some unit test cases that actually
had complete code coverage, I won't know what the full set of behaviour is.
It might be that swapping case 1:/case 2: would work.
Or maybe it's the call to:
if (tmp_keymgmt != NULL)
provkey = evp_pkey_export_to_provider(locpctx->pkey, locpctx->libctx,
&tmp_keymgmt, locpctx->propquery);

that is wrong. Maybe that should occur only after the loop finishes.
(Again, not ending prematurely)

In my case, I've created an X509 object that is trying to self-sign the CA
public key. You might say that I need to throw propquery strings into that,
but that seems very wrong: that part of the code shouldn't need to know where the
EVP_PKEY is from, it should just use it.

[Viktor Dukhovni]

On Wed, Aug 27, 2025 at 07:02:45PM -0400, Michael Richardson wrote:

> > I am not convinced you really **want** to explicitly "access
> > providers". Rather, you likely want to have some operations (key
> > generation, loading a key, signing, ...) performed by a chosen
> > provider.
>
> I don't know if there is semantic subtlety you are trying to get at, but I've
> missed it. This seems to be semantic pedantry to me. I'm happy to be wrong.
>
> I think in 60% of the cases, people building bespoke systems in C, Rust, ruby,
> python, Perl, ... know exactly what they want, and they are mostly targetting
> a specific provider.

The distinction is real, I am not in the habit of linguistic pedantry.
There's a difference between using the EVP API with a properties(7)
value that fetches algorithm handles from a selected provider, and using
the **provider API** to directly call the backend provider interfaces.

What I'm suggesting is that I'm not seeing your so far stated requirements as
cause to dive into the backend provider API. You should be able to do
everything you set out to do with the EVP API and judicious use of
"provider=tpm2" in the `propq` argument of various "fetch" calls.

> Yes, they might want some agility in moving to a different provider, but they
> will need to test against those anyway. Their "customers", if they are
> external, will be told which ones work, and that others are "who knows" until
> tested. For me, this query string is not only unwanted, but could even be dangerous.
>
> In the other 40% of cases, the systems have been built for others to use, and
> the query string has some significant value because, the systems are not open
> source (or rather, then people installing them are not installing from
> source), so a way for the operator to specify the specific provider, and
> maybe even some parameters to the provider, such as an IP address (in the
> case of an big-iron HSM) to connect to.

The IP address of a network HSM would likely be part of the provider
configuration (load parameters), and would then be implicit in
subsequent calls to the provider. I would not expect it to have
be specified when fetching algorithms, signing, verifying, ...

> I'm working on a bespoke system written RubyOnRails and Rust
> (different pieces), and to do that I'm trying to make sure that the
> ruby-openssl library can do what I want. I'd like to benefit as many
> people as possible, but the query string business is not particularly
> useful to me.

Your desire to bypass the EVP API is perplexing. The backend provider
API primarily there to enable the EVP layer to support a mix of
providers and to allow some providers to "wrap" (be a front ends to)
other providers. It is not expected that end-applications will directly
interface with provider backends.

> > Property Query String A property query string is a string
> > containing a sequence of properties that can be used to select an
> > algorithm implementation. For example the query string
> > "provider=example,foo=bar" will select algorithms from the "example"
> > provider that have a "foo" property defined for them with a value of
> > "bar".
>
> Yes, but I shouldn't have to throw these strings all over the place.
> Once is enough.

If you're sufficiently disinclined to pass explicit `propq` values
to the relevant API calls, you can, if you prefer, use:

EVP_set_default_properties(3)

to set a default. Whichever libctx (or NULL for the default) is
the one the provider is loaded into, can be set up to make that
provider be target of all fetch operations. You can then opt
out of that default with a "local" property of "-provider".

To better understand the mismatch between behaviour and expectations, a
reproducer is needed. Please share minimal demo code that does not
do what you expect. If you're somewhat confident the EVP implementation
is not correct, a new issue on Github would be helpful, but a bit of
demo code in this thread might be a reasonable way to rule out any basic
misunderstanding.

> In my case, I've created an X509 object that is trying to self-sign
> the CA public key. You might say that I need to throw propquery
> strings into that, but that seems very wrong: that part of the code
> shouldn't need to know where the EVP_PKEY is from, it should just use
> it.

Sure, there's no explicit propq in either X509_sign(3) or
X509_sign_ctx(3), so any implied provider would be associated
with the key or EVP_MD_CTX. Which of these two functions is
the one you're using?

--
Viktor. 🇺🇦 Слава Україні!

[Michael Richardson]

Thank you very much for the reply.
(Please excuse me if I seem grumpy: there is other unrelated shit)

Viktor Dukhovni <openss...@dukhovni.org> wrote:
> The distinction is real, I am not in the habit of linguistic pedantry.
> There's a difference between using the EVP API with a properties(7)
> value that fetches algorithm handles from a selected provider, and
> using the **provider API** to directly call the backend provider
> interfaces.

I never had any interest in calling the provider *API* directly.
I guess the word "interface" is the source of confusion.
I mean: the openssl library calls with OSSL_PROVIDER* in them.

Whereas, you understood me to be referring to the interface *from* openssl to
the provider. Which arguably is the _provider interface_

(we call all these things APIs now, often forgetting what the A means,
and openssl is not the application, and probably there are no better TLAs anymore)

I don't think that I ever said I wanted to do that.
I'm sorry I mis-explained.

I've deleted the rest of your EVP_KEY comments, because clearly they do not apply.
I'm saying that the EVP API does not seem capable, and I'm asking for
clarifications, fixes, and only if necessary, amendments.

> What I'm suggesting is that I'm not seeing your so far stated
> requirements as cause to dive into the backend provider API. You
> should be able to do everything you set out to do with the EVP API and
> judicious use of "provider=tpm2" in the `propq` argument of various
> "fetch" calls.

a. It's not enough, because many places do not have propq arguments, or
require propq arguments in places where that code should not care.
b. It's not actually working.

>> In the other 40% of cases, the systems have been built for others to
>> use, and the query string has some significant value because, the
>> systems are not open source (or rather, then people installing them
>> are not installing from source), so a way for the operator to specify
>> the specific provider, and maybe even some parameters to the provider,
>> such as an IP address (in the case of an big-iron HSM) to connect to.

> The IP address of a network HSM would likely be part of the provider
> configuration (load parameters), and would then be implicit in
> subsequent calls to the provider. I would not expect it to have be
> specified when fetching algorithms, signing, verifying, ...

I should be able to load a provider library, then initialize it multiple
times to talk to multiple HSMs. I don't see a place to configure IP
addresses with OSSL_Provider_load, and anyway, that is a handle on the
library, and not the specific configuration. For awhile, I thought
OSSL_LIB_CTX provided that, but I now understand that providers are inside
libctx, not the other way around.
Once configured, I would expect to get a handle that I can now apply to
whatever structures I need (via set() method of course) to do things.
It being NULL would get whatever the system thinks is sane.

>> Yes, but I shouldn't have to throw these strings all over the place.
>> Once is enough.

> If you're sufficiently disinclined to pass explicit `propq` values to

You are asking me to carry around an attribute in my structures which are
already duplicated in the structures I am creating/using. That is, despite
all the variously arcane mechanisms and legacy mecahnism to make providers
"slip in" without changing code, I'm actually supposed to rework everything?
AFAIK, I can't find anything around X509_sign_ctx that takes a propq.
Nor should I: the PKEY already was loaded correctly.

> the relevant API calls, you can, if you prefer, use:

> EVP_set_default_properties(3)

> to set a default. Whichever libctx (or NULL for the default) is the
> one the provider is loaded into, can be set up to make that provider be
> target of all fetch operations. You can then opt out of that default
> with a "local" property of "-provider".

Yeah, that might solve my immediate problem, but it won't solve it for someone
who had two HSMs at different IP addresses using the same provider.

>> around that call, which allowed the second iteration to run, which
>> then did what the comment above promised. The correct signature
>> routine was called, and the key was available.

> To better understand the mismatch between behaviour and expectations, a
> reproducer is needed. Please share minimal demo code that does not do
> what you expect. If you're somewhat confident the EVP implementation
> is not correct, a new issue on Github would be helpful, but a bit of
> demo code in this thread might be a reasonable way to rule out any
> basic misunderstanding.

Sure, I have a short unit test case in my code base.
I've been up-arrow-return on it all week.
But, it's written in ruby, and uses ruby-openssl.
To see it run, it has to initialize ("manufacture") a swtpm, start it, etc.
Which I do in ruby, as part of the test setup/teardown.
But, I can put that all into a shell script.

I can't generate the keys through openssl yet, as I didn't figure out how to make
that work, and I moved on, although I didn't change the subject line enough.
So, to run the test, you'll need the tpm2_* utilities too.
(pause)

I think that the propquery argument to EVP_PKEY_CTX_new_from_name() probably
does the right thing, and I think I can make that work, but last week I didn't see that.
I can probably create an optional propq argument, but it has a smell.
Or a new API.

I think it would be better to allow EVP_PKEY_CTX_ctrl_str() to accept a "provider" control string.
That's what I initially tried based upon my understanding at the time.

I would prefer a EVP_PKEY_CTX_new_from_provider(OSSL_PROVIDER *), or
EVP_PKEY_new_from_provider(OSSL_PROVIDER *). If I have to change my code,
then I'd rather change it to be a bit more object-oriented.
Ruby,Python,Perl wrappers can be more fully OO here.

Again, my earlier comment that much of the man pages, when they discuss
providers, lead rapidly to the reader to wandering into the provider-API documentation,
the stuff aimed at people writing providers.
Which is almost never what the reader wanted.
(As you said: I really don't want to call any of that directly, nor should anyone)

>> In my case, I've created an X509 object that is trying to self-sign
>> the CA public key. You might say that I need to throw propquery
>> strings into that, but that seems very wrong: that part of the code
>> shouldn't need to know where the EVP_PKEY is from, it should just use
>> it.

> Sure, there's no explicit propq in either X509_sign(3) or
> X509_sign_ctx(3), so any implied provider would be associated with the
> key or EVP_MD_CTX. Which of these two functions is the one you're
> using?

The code current uses X509_sign(), which leads to ASN1_item_sign_ex,
which calls evp_md_ctx_new_ex() make an MD_CTX, and the calls sign_ctx().
I see that X509_sign_ctx() has me make the MD_CTX, and the leads to
ASN1_item_sign_ctx() as well.
I will see if I can create an acceptable EVP_MD_CTX.
evp_md_ctx_new_ex() does a lot of things, but it's internal.

[Viktor Dukhovni]

On Thu, Aug 28, 2025 at 03:03:08PM -0400, Michael Richardson wrote:

> Thank you very much for the reply.
> (Please excuse me if I seem grumpy: there is other unrelated shit)

You're welcome, understood, I'll keep that in mind. :-)

> Viktor Dukhovni <openss...@dukhovni.org> wrote:
> > The distinction is real, I am not in the habit of linguistic pedantry.
> > There's a difference between using the EVP API with a properties(7)
> > value that fetches algorithm handles from a selected provider, and
> > using the **provider API** to directly call the backend provider
> > interfaces.
>
> I never had any interest in calling the provider *API* directly.
> I guess the word "interface" is the source of confusion.
> I mean: the openssl library calls with OSSL_PROVIDER* in them.

Actually, I really am saying that you should not need to use any of the
functions mentioning `OSSL_PROVIDER *`, except perhaps in so far as your
application is explicitly loading some providers at startup. Once all
the relevant providers are loaded, you really SHOULD be able to use just
the EVP API, with any explicit provider preferences indicated through
properties (`propq` arguments), to set a preferred non-mandatory
provider, you'd use "?provider=tpm2" or similar so that the "tpm2"
provider is preferred, but algorithms not found there are still found in
the "default" provider.

If you're having to reach into the OSSL_PROVIDER API in order to
generate or load keys, sign messages or verify signatures, something
is wrong. Perhaps the documentation is poor, or you're not find the
right documentation (hard to find may be a variant of "poor"). Perhaps
you've talked yourself into a non-productive approach. Or perhaps the
EVP API is actually materially deficient. It now remains to figure out
which problem needs solving.


> I've deleted the rest of your EVP_[P]KEY comments, because clearly
> they do not apply.

Well, you keep saying that, but the EVP API is what the expected API
you're expected to use. So if that fails to meet your needs, we need to
fix that. In some cases there are additional providers that facilitate
that use, if, for example, a PKCS#11 provider had good "tpm2" support,
you might find that it meshes more naturally with the EVP layer and is
easier to use, but I don't have anything specific in mind at this time.

> I'm saying that the EVP API does not seem capable, and I'm asking for
> clarifications, fixes, and only if necessary, amendments.

Well, you haven't really provided solid evidence to back up that claim.
I'm willing to accept that there is room for improvement, and open
issues to fill any gaps, but these gaps are yet to be identified.

Let's first identify a simple clear task that isn't working for you.
Is it signing with an existing key that is stored in the TPM? You
have a "message" to sign and a TPM key URI? Or otherwise explain,
and provide some demo code.

--
Viktor.