Using PKINIT with ECC

[Goetz Golla]

Hello there,

in our organisation we are successfully using PKINIT with RSA 2048
client certificates for many years. We are now trying to move to ECC
certificates with the curve secp384r1.

All attempts have been unsuccessful yet.

We are using the Kerberos Version 1.17.x that comes with Ubuntu 20.04,
and we are living in a linux only datacenter.

Tracing kinit gives the following anonymized result.

[11061] 1700039487.898263: Getting initial credentials for user@REALM
[11061] 1700039487.898265: Sending unauthenticated request
[11061] 1700039487.898266: Sending request (187 bytes) to REALM
[11061] 1700039487.898267: Sending initial UDP request to dgram ip:88
[11061] 1700039487.898268: Received answer (323 bytes) from dgram ip:88
[11061] 1700039487.898269: Response was from master KDC
[11061] 1700039487.898270: Received error from KDC:
-1765328359/Additional pre-authentication required
[11061] 1700039487.898273: Preauthenticating using KDC method data
[11061] 1700039487.898274: Processing preauth types: PA-PK-AS-REQ (16),
PA-PK-AS-REP_OLD (15), PA-PK-AS-REQ_OLD (14), PA-FX-FAST (136),
PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147), PA-ENC-TIMESTAMP (2),
PA_AS_FRESHNESS (150), PA-FX-COOKIE (133)
[11061] 1700039487.898275: Selected etype info: etype aes256-cts, salt
"REALMuser", params ""
[11061] 1700039487.898276: Received cookie: MIT
[11061] 1700039488.76900: Preauth module pkinit (147) (info) returned:
0/Success
[11061] 1700039488.76901: PKINIT client received freshness token from KDC
[11061] 1700039488.76902: Preauth module pkinit (150) (info) returned:
0/Success
[11061] 1700039491.317246: PKINIT loading CA certs and CRLs from DIR
[11061] 1700039491.317247: PKINIT client computed kdc-req-body checksum
9/CE1E7C16A71A8DC285A707FB43CB8DEF7EC184F0
[11061] 1700039491.317249: PKINIT client making DH request
[11061] 1700039491.317250: Preauth module pkinit (16) (real) returned:
-1765328360/Preauthentication failed
[11061] 1700039491.317251: PKINIT client ignoring draft 9 offer from RFC
4556 KDC
[11061] 1700039491.317252: Preauth module pkinit (15) (real) returned:
-1765328360/Preauthentication failed
[11061] 1700039491.317253: PKINIT client ignoring draft 9 offer from RFC
4556 KDC
[11061] 1700039491.317254: Preauth module pkinit (14) (real) returned:
-1765328360/Preauthentication failed
[11061] 1700039492.894873: AS key obtained for encrypted timestamp:
aes256-cts/CAD7
[11061] 1700039492.894875: Encrypted timestamp (for 1700039492.190872):
plain 301AA011180F32303233313131353039313133325AA105020302E998,
encrypted
930F652AFE54AD71886006DD4D8AE2E016970562C3772564DD1F6B70245B331CD06CBCAD3A6FB4B9CC577A070A099B3863C8A93FA02EA74C
[11061] 1700039492.894876: Preauth module encrypted_timestamp (2) (real)
returned: 0/Success

[11061] 1700039492.894877: Produced preauth for next request:
PA-FX-COOKIE (133), PA-ENC-TIMESTAMP (2)

For the following questions we did not find any answers in the MIT
Kerberos documentation:

* Does MIT Kerberos support PKINIT with Elliptic Curves as described
in RFC 5349 ?
* Could it be that for ECC client certificates the KDC certificate
also needs the be ECC ?
* If thats true, how can we than migrate from RSA to ECC certificates
for PKINIT ?

Any help is appreciated.

Regards,

Goetz

[Ken Hornstein]

>in our organisation we are successfully using PKINIT with RSA 2048
>client certificates for many years. We are now trying to move to ECC
>certificates with the curve secp384r1.
>
>All attempts have been unsuccessful yet.

My reading of the code (I am using a newer version of MIT Kerberos than
you) is that RSA is hard-coded as the signing algorithm. So it looks
like it won't work (I am confident that if I am wrong someone will
correct me). I know that at least at our site we're going to have to
transition to some kind of post-quantum signing algorithm in the future
like many others so I think that eventually this support will be added,
but that doesn't help you now unfortunately.

--Ken

[Greg Hudson]

On 11/15/23 23:22, Goetz Golla wrote:
> * Does MIT Kerberos support PKINIT with Elliptic Curves as described
> in RFC 5349 ?

A P-384 EC client certificate works in my tests, with either krb5-1.17
or the current code, as long as the KDC is also running MIT krb5.

Ken is correct that there is a hardcoded reference to RSA in the source:

p7si->digest_enc_alg->algorithm =
OBJ_nid2obj(NID_sha256WithRSAEncryption);

and this probably means the CMS signature has a piece of incorrect
metadata when an EC certificate is used. But this field is not used
when generating the signature contents and is ignored by OpenSSL when
verifying the signature (when the KDC is running MIT krb5).

> * Could it be that for ECC client certificates the KDC certificate
> also needs the be ECC ?

In my tests the KDC certificate was an RSA cert, so no.

Of course, my experience doesn't match yours. From your trace, I
believe that the failure occurs in the client code, not on the KDC, so
inspecting the KDC logs would not help. But the trace log does not
contain any detailed information about the failure.

You can sometimes improve the diagnostics for PKINIT failures by
removing the long-term keys associated with the principal, so that
authentication does not fall back to encrypted timestamp:

kadmin purgekeys -all user

If that doesn't help, it may be necessary to build the code with
debugging symbols and and step through it to find out where it is failing.

[Ken Hornstein]

>On 11/15/23 23:22, Goetz Golla wrote:
>> * Does MIT Kerberos support PKINIT with Elliptic Curves as described
>> in RFC 5349 ?
>
>A P-384 EC client certificate works in my tests, with either krb5-1.17
>or the current code, as long as the KDC is also running MIT krb5.

We got burnt a while ago with an older PKINIT client-side plugin that
worked fine when the KDC was linked against OpenSSL 1.0.2 but failed
with OpenSSL 1.1 and above (this was fixed in newer MIT code and only
occured when you were using a smartcard). I am wondering if perhaps the
incorrect metadata makes something fail on other versions of OpenSSL?
I know this seems to be a completely client-side problem.

>Of course, my experience doesn't match yours. From your trace, I
>believe that the failure occurs in the client code, not on the KDC, so
>inspecting the KDC logs would not help. But the trace log does not
>contain any detailed information about the failure.

I have mentioned this before, but ... is there any interest in adding
additional trace points for every place where the old "pkiDebug" calls
are made? Hidden errors when doing PKINIT are the bane of my existence
and I feel that I'm not the only one. I understand there are concerns
about making the trace log too verbose but I think every error could
generate a trace message and it wouldn't add too much to the trace output
when everything was working.

--Ken

[Goetz Golla]

I have made some progress locating the problem which I would like to
share here.

When I am using openssl with the extensions file exactly as recommended
to create the client certificate and refer to the file in
/etc/krb5.conf, kinit works flawlessly with Public Key Algorithm
id-ecPublicKey / secp384r1. I can even use ECC for the CA certificate
without problems.

The problem is that we are using Yubikeys with the PIV function as
smartcards. Yubikey supports ECC with secp384r1.

But when I create a new key pair with ECC, sign the public key with
openssl with the same extension file, and try to use it, I get the
following situation

* When I do a kinit, connection is opened to the KDC and it sends the
message "pre-authentication required" back.
* After I enter the yubikey PIN, there is no more communication with
the KDC - so the problem is on the client as Greg has already proposed
* kinit reports that the certificate from the yubikey has been read
and that the CN is OK (I am using pkinit_cert_match...).
* But then, I am getting the following message from opensc:

P:296321; T:0x140609979246400 17:33:26.054 [opensc-pkcs11]
pkcs11-object.c:697:C_SignInit: C_SignInit() = CKR_KEY_HANDLE_INVALID

So there is some problem with opensc-pkcs11. Interestingly I am using
the same Yubikey successfully with pam-pkcs11 to authenticate without
problems.

I have some hope that I am seeing a configuration issue, even though it
is weird that this would depend on the public key algorithm.

On 11/17/23 06:53, Greg Hudson wrote:
> On 11/15/23 23:22, Goetz Golla wrote:
>> * Does MIT Kerberos support PKINIT with Elliptic Curves as described
>> in RFC 5349 ?
>
> A P-384 EC client certificate works in my tests, with either krb5-1.17
> or the current code, as long as the KDC is also running MIT krb5.
>

> Ken is correct that there is a hardcoded reference to RSA in the source:
>
>         p7si->digest_enc_alg->algorithm =
>             OBJ_nid2obj(NID_sha256WithRSAEncryption);
>
> and this probably means the CMS signature has a piece of incorrect
> metadata when an EC certificate is used.  But this field is not used
> when generating the signature contents and is ignored by OpenSSL when
> verifying the signature (when the KDC is running MIT krb5).
>
>> * Could it be that for ECC client certificates the KDC certificate
>> also needs the be ECC ?
>
> In my tests the KDC certificate was an RSA cert, so no.
>

> Of course, my experience doesn't match yours.  From your trace, I
> believe that the failure occurs in the client code, not on the KDC, so
> inspecting the KDC logs would not help.  But the trace log does not
> contain any detailed information about the failure.
>

[Carson Gaspar]

On 11/19/2023 9:00 AM, Ken Hornstein via Kerberos wrote:
> I have mentioned this before, but ... is there any interest in adding
> additional trace points for every place where the old "pkiDebug" calls
> are made? Hidden errors when doing PKINIT are the bane of my existence
> and I feel that I'm not the only one. I understand there are concerns
> about making the trace log too verbose but I think every error could
> generate a trace message and it wouldn't add too much to the trace output
> when everything was working.

Consider this a +1 for some way to enable useful PKINIT debugging
without a recompile. The number of times I've had to install a debug
plugin .so just to figure out basic config issues...

Hell, even just adding an autoconf option to enable it so I didn't have
to hand-edit the include file would be a win... (yeah, I could probably
pass in a custom CPPFLAGS option, but by the time I find the !@#$% macro
I'm already in the include file, so...)

--

Carson

[Ken Hornstein]

>P:296321; T:0x140609979246400 17:33:26.054 [opensc-pkcs11]
>pkcs11-object.c:697:C_SignInit: C_SignInit() = CKR_KEY_HANDLE_INVALID
>
>So there is some problem with opensc-pkcs11. Interestingly I am using
>the same Yubikey successfully with pam-pkcs11 to authenticate without
>problems.

CKR_KEY_HANDLE_INVALID means "The handle passed is not a a valid key".
Which is not exactly helpful ("handles" in PKCS#11 are nonzero integers
and refer to objects on the card). You MIGHT be running into an issue
where there is a bug in the PKINIT code that makes PKCS#11 calls but
that code has been stable for a long time so I would be surprised if the
failure was there (but, I have been surprised before!). I believe there
is some environment variable or other configuration you can set to get
more debugging information out of opensc but I don't recall it right
now.

However, I believe Yubico provides a PKCS#11 module for Yubikeys; have
you tried that? The OpenSC people usually do a good job in terms of
supporting a wide variety of cards but depending on how old the particular
version of OpenSC you are using is you may be running into a compatibility
issue.

--Ken

[Ken Hornstein]

>P:296321; T:0x140609979246400 17:33:26.054 [opensc-pkcs11]
>pkcs11-object.c:697:C_SignInit: C_SignInit() = CKR_KEY_HANDLE_INVALID

As a follow-up to my previous message, I believe the problem is that for
_smartcards_ the PKCS#11 mechanism is hardcoded to CKM_RSA_PKCS:

/*
* We'd like to use CKM_SHA256_RSA_PKCS for signing if it's available, but
* historically many cards seem to be confused about whether they are
* capable of mechanisms or not. The safe thing seems to be to ignore the
* mechanism list, always use CKM_RSA_PKCS and calculate the sha256 digest
* ourselves.
*/
id_cryptoctx->mech = CKM_RSA_PKCS;

There's a CKM_ECDSA_SHA1 mechanism which does the hashing for you, and
there's also a CKM_ECDSA mechanism where you have to calculate the hash
value which is kind of close (but not exactly the same as) to CKM_RSA_PKCS.
So, dang, code is going to have to be written.

--Ken

[Greg Hudson]

On 11/19/23 12:00, Ken Hornstein via Kerberos wrote:
> I have mentioned this before, but ... is there any interest in adding
> additional trace points for every place where the old "pkiDebug" calls
> are made? Hidden errors when doing PKINIT are the bane of my existence
> and I feel that I'm not the only one. I understand there are concerns
> about making the trace log too verbose but I think every error could
> generate a trace message and it wouldn't add too much to the trace output
> when everything was working.

I would be happy to have more trace logging to diagnose PKINIT errors,
but converting every pkiDebug() call probably wouldn't meet the criteria
for good trace logging. We've already made a few passes in this area,
most recently one from you which went into release 1.20 (commit
34625d594c339a077899fa01fc4b5c331a1647d0).

Based on this thread, it is clear that there is still room for
improvement in the handling of PKCS11 errors. While we mostly handle
OpenSSL errors through the oerr() wrapper which trace logs the OpenSSL
error queue and sets an extended error message, we don't have any such
wrapper for PKCS11 errors. In this case, we now know that C_SignInit()
failed; here is the handling for that error:

if ((r = id_cryptoctx->p11->C_SignInit(id_cryptoctx->session, &mech,
obj)) != CKR_OK) {
pkiDebug("C_SignInit: %s\n", pkcs11err(r));
return KRB5KDC_ERR_PREAUTH_FAILED;
}

So only the generic "Preauthentication failed" message shows up in the
trace log (at the libkrb5 level) while the old debugging would have
indicated the failed operation and the PKCS11 error string.

[Ken Hornstein]

>I would be happy to have more trace logging to diagnose PKINIT errors,
>but converting every pkiDebug() call probably wouldn't meet the criteria
>for good trace logging. We've already made a few passes in this area,
>most recently one from you which went into release 1.20 (commit
>34625d594c339a077899fa01fc4b5c331a1647d0).

I guess what I was thinking was maybe not EVERY pkiDebug() call, but
more all of the ones that report errors. E.g:

> if ((r = id_cryptoctx->p11->C_SignInit(id_cryptoctx->session, &mech,
> obj)) != CKR_OK) {
> pkiDebug("C_SignInit: %s\n", pkcs11err(r));
> return KRB5KDC_ERR_PREAUTH_FAILED;
> }

There are others than the PKCS#11 calls, of course. I guess what I'd like
(if possible) was that anytime the plugin returned PREAUTH_FAILED, the
debug trace will explain why.

--Ken

[Greg Hudson]

On 11/20/23 14:09, Ken Hornstein via Kerberos wrote:
> I guess what I was thinking was maybe not EVERY pkiDebug() call, but
> more all of the ones that report errors.

I think we're in agreement. Setting an extended error message should
generally be sufficient, as it should get trace logged by libkrb5 or
included in the KDC log.

[Goetz Golla]

On 11/19/23 18:33, Ken Hornstein wrote:
> However, I believe Yubico provides a PKCS#11 module for Yubikeys; have
> you tried that? The OpenSC people usually do a good job in terms of
> supporting a wide variety of cards but depending on how old the particular
> version of OpenSC you are using is you may be running into a compatibility
> issue.
>
> --Ken

Indeed the module provided by Yubico solved the issue. It is called
ykcs11 and is readily available in the linux package managers.

E.g. using

 kinit -X X509_user_identity='PKCS11:libykcs11.so'

instead of

 kinit -X X509_user_identity='PKCS11:opensc-pkcs11.so'

BUT with ykcs11 I got the following message in the trace

[14174] 1700562344.750583: PKINIT error: There are 3 certs, but there
must be exactly one.
[14174] 1700562344.750584: PKINIT client has no configured identity;
giving up
[14174] 1700562344.750585: Preauth module pkinit (16) (real) returned:
22/Invalid argument

This is hard to understand because there is only one certificate on the
Yubikey.

I solved this with the following line in /etc/krb5.conf

 pkinit_cert_match = &&<SUBJECT>UID=.*CN=.*$<ISSUER>CN=YUBIKEY-CA${code}

The line matches our certificate, so there is only one left and kinit is
working now with ECC certificates.

But I am wondering if using pkinit_cert_match without really
understanding why I need it and what the other two certificates are is
such a good idea ?

[Ken Hornstein]

>> However, I believe Yubico provides a PKCS#11 module for Yubikeys; have
>> you tried that? The OpenSC people usually do a good job in terms of
>> supporting a wide variety of cards but depending on how old the particular
>> version of OpenSC you are using is you may be running into a compatibility
>> issue.
>>
>> --Ken
>
>Indeed the module provided by Yubico solved the issue. It is called
>ykcs11 and is readily available in the linux package managers.

I am a LITTLE surprised it worked! The MIT PKINIT plugin hard-codes
the mechanism in the request; I guess the Yubico library ignores the
mechanism given to it, which seems strange to me.

I have to ask ... are you SURE that it's using ECC? Because the code that
uses the PKCS#11 library is actually generating a PKCS#1 digest. I was
under the impression that ECC signatures are in a different format, so
I am puzzled how it works at all.

>[14174] 1700562344.750583: PKINIT error: There are 3 certs, but there
>must be exactly one.

I also use smartcards with multiple certificates, and ... well, I'm
not sure how the code would get it wrong. I would use some PKCS#11
tools to poke at the Yubico library to see what certificates it
says that it has (the KRB5_TRACE output should give you the subjects
of the certificates that it finds).

--Ken

[Goetz Golla]

On 11/24/23 21:47, Ken Hornstein wrote:
>>> However, I believe Yubico provides a PKCS#11 module for Yubikeys; have
>>> you tried that? The OpenSC people usually do a good job in terms of
>>> supporting a wide variety of cards but depending on how old the particular
>>> version of OpenSC you are using is you may be running into a compatibility
>>> issue.
>>>
>>> --Ken
>> Indeed the module provided by Yubico solved the issue. It is called
>> ykcs11 and is readily available in the linux package managers.
> I am a LITTLE surprised it worked! The MIT PKINIT plugin hard-codes
> the mechanism in the request; I guess the Yubico library ignores the
> mechanism given to it, which seems strange to me.
>
> I have to ask ... are you SURE that it's using ECC? Because the code that
> uses the PKCS#11 library is actually generating a PKCS#1 digest. I was
> under the impression that ECC signatures are in a different format, so
> I am puzzled how it works at all.

We had it working in November with Yubico's libykcs11 in a lab and in
production tested by two independent people. Testing it again this year
it failed. We are in the process of finding out what exactly we have
tested in November.

I am really confused now. I thought that the problem was in the opensc
code and replacing it with Yubico's libykcs11, which officially supports
ECC, should fix it.

Now you seem to suggest that the problem is in the Kerberos code ?

Regards,

Goetz

[Ken Hornstein]

>We had it working in November with Yubico's libykcs11 in a lab and in
>production tested by two independent people. Testing it again this year
>it failed. We are in the process of finding out what exactly we have
>tested in November.
>
>I am really confused now. I thought that the problem was in the opensc
>code and replacing it with Yubico's libykcs11, which officially supports
>ECC, should fix it.
>
>Now you seem to suggest that the problem is in the Kerberos code ?

Well, geez dude, this was back in November and I brought this up then.
But here is some snippets of the PKCS#11 code in MIT Kerberos:

When specifying the search parameters to find the private key:

keytype = CKK_RSA;
attrs[nattrs].type = CKA_KEY_TYPE;
attrs[nattrs].pValue = &keytype;
attrs[nattrs].ulValueLen = sizeof keytype;
nattrs++;

When setting the key signing mechanism:

/*
* We'd like to use CKM_SHA256_RSA_PKCS for signing if it's available, but
* historically many cards seem to be confused about whether they are
* capable of mechanisms or not. The safe thing seems to be to ignore the
* mechanism list, always use CKM_RSA_PKCS and calculate the sha256 digest
* ourselves.
*/
id_cryptoctx->mech = CKM_RSA_PKCS;

Those are all hardcoded use of RSA keys and signing mechanisms and it
doesn't handle ECC at all. So unless the Yubico library ignored the
key type and mechanism (which I think would be extremely unlikely but
not impossible) I suspect you were using RSA back during your original
testing and didn't realize it.

--Ken

[Goetz Golla]

On 1/11/24 15:41, Ken Hornstein wrote:
> But here is some snippets of the PKCS#11 code in MIT Kerberos:
>
> When specifying the search parameters to find the private key:
>
> keytype = CKK_RSA;
> attrs[nattrs].type = CKA_KEY_TYPE;
> attrs[nattrs].pValue = &keytype;
> attrs[nattrs].ulValueLen = sizeof keytype;
> nattrs++;
>
> When setting the key signing mechanism:
>
> /*
> * We'd like to use CKM_SHA256_RSA_PKCS for signing if it's available, but
> * historically many cards seem to be confused about whether they are
> * capable of mechanisms or not. The safe thing seems to be to ignore the
> * mechanism list, always use CKM_RSA_PKCS and calculate the sha256 digest
> * ourselves.
> */
> id_cryptoctx->mech = CKM_RSA_PKCS;
>
> Those are all hardcoded use of RSA keys and signing mechanisms and it
> doesn't handle ECC at all. So unless the Yubico library ignored the
> key type and mechanism (which I think would be extremely unlikely but
> not impossible) I suspect you were using RSA back during your original
> testing and didn't realize it.
>
> --Ken

Its good to know the reason why MIT Kerberos cannot handle EC
certificates right now.

I know that NIST is happy with RSA 2048, but in Europe RSA >= 3072 is
already mandatory, and this key size makes small devices like the
Yubikeys very slow when generating the keys. In fact, Yubikeys only
support RSA <=2048.

So is there a way to submit a feature request for ECDSA support in MIT
Kerberos ?

-- Goetz

[Ken Hornstein]

>Its good to know the reason why MIT Kerberos cannot handle EC
>certificates right now.

I think it's important to be specific here; the issue is specifically
a PKCS#11 token; Greg has already said that a software ECC certificate & key
work fine.

>So is there a way to submit a feature request for ECDSA support in MIT
>Kerberos ?

I have no inner view to the priorities of the MIT development team, so
I can't answer that. I can say I personally have had success by submitting
pull requests to their github page, which I suppose is a roundabout way
of saying that the best way to make this happen is to do it yourself.

I imagine at some time we will be transitioning to ECDSA certificates
so if no one has implemented support by then I will probably do it.
However, it sounds like you need this more urgently than I so I would
not suggest waiting for me.

--Ken

[Simo Sorce]

On Fri, 2024-01-26 at 08:01 +0100, Goetz Golla wrote:
> On 1/11/24 15:41, Ken Hornstein wrote:
> > But here is some snippets of the PKCS#11 code in MIT Kerberos:
> >
> > When specifying the search parameters to find the private key:
> >
> > keytype = CKK_RSA;
> > attrs[nattrs].type = CKA_KEY_TYPE;
> > attrs[nattrs].pValue = &keytype;
> > attrs[nattrs].ulValueLen = sizeof keytype;
> > nattrs++;
> >
> > When setting the key signing mechanism:
> >
> > /*
> > * We'd like to use CKM_SHA256_RSA_PKCS for signing if it's available, but
> > * historically many cards seem to be confused about whether they are
> > * capable of mechanisms or not. The safe thing seems to be to ignore the
> > * mechanism list, always use CKM_RSA_PKCS and calculate the sha256 digest
> > * ourselves.
> > */
> > id_cryptoctx->mech = CKM_RSA_PKCS;
> >
> > Those are all hardcoded use of RSA keys and signing mechanisms and it
> > doesn't handle ECC at all. So unless the Yubico library ignored the
> > key type and mechanism (which I think would be extremely unlikely but
> > not impossible) I suspect you were using RSA back during your original
> > testing and didn't realize it.
> >
> > --Ken
>

> Its good to know the reason why MIT Kerberos cannot handle EC
> certificates right now.

Whatever shortcomings there are the reason is low demand, or not enough
justification to spend the time on it.

> I know that NIST is happy with RSA 2048, but in Europe RSA >= 3072 is
> already mandatory,
>

Please cite the source of this statement, as far as I know only BSI
requires it for some German government stuff and there is no EU level
agency that requires this anywhere, just like in the USA NIAP requires
them for Common criteria certification.
A desired for 3k keys is understandable but it is unworkable given the
rest of the worldwide PKI infrastructure still relies on Intermediate
CAs that use 2k keys.

> and this key size makes small devices like the
> Yubikeys very slow when generating the keys. In fact, Yubikeys only
> support RSA <=2048.
>

> So is there a way to submit a feature request for ECDSA support in MIT
> Kerberos ?

Ken provided reasonable answers for this part.

Simo.

--
Simo Sorce
Distinguished Engineer
RHEL Crypto Team
Red Hat, Inc