"Algorithm constraints check failed" for JNDI provider due to AggregateTrustManager

[Tyler Camp]

When using LDAPS with the JNDI provider and the correct certs in my cacerts file, I'm getting "Algorithm constraints check failed on signature algorithm: 1.2.840.113549.1.1.10" upon connecting, which I wasn't getting with our previous Shiro-based LDAP implementation. After some debugging I found that the given signing algorithm wasn't "registered" in this case, even though it had worked before without issue. After lots more debugging I figured out that Ldaptive's use of AggregateTrustManager, which implements X509TrustManager but not X509ExtendedTrustManager, is causing it to be wrapped in sun.security.ssl.AbstractTrustManagerWrapper by JNDI automatically and leading to this error.

I was able to fix it by customizing the JndiProviderConfig with a custom SSLSocketFactory and SSLContextInitializer, which takes the AggregateTrustManager from Ldaptive and flattens it back to the original set of trust managers. Everything runs flawlessly with this fix.

The issue is that this works on a regular build, but not in an obfuscated release build. My SSLSocketFactory instance is being given to the config object but is not being instantiated or used afterwards. I assume this has to do with how JNDI uses reflection to get the getDefault method of the SSLSocketFactory class given to it. I'm struggling to figure out why it's behaving this way.

So here's my question: Is there an easier way to get around this "Algorithm constraints" error, and avoid having to override the SSLSocketFactory/ContextInitializers?

Some notes:

- I'd like to keep using the JNDI provider to maximize backwards-compatibility

- The certs are signed via RSASSA-PSS/SHA512

- My test and QA machines are both on Java 1.8.0_191

Thank you for your time

[Tyler Camp]

Here's the relevant code: https://gist.github.com/tylercamp/e0a6498aabf26e716acfca486e2d60a1

[Daniel Fisher]

If you could share a sample certificate that causes the constraints check failed error that will help my investigation. Thanks.

--Daniel Fisher

--
You received this message because you are subscribed to the Google Groups "ldaptive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ldaptive+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tyler Camp]

The problematic cert has been attached here. Thank you for your time.

[Daniel Fisher]

I don't think the solution you posted is correct.

You're only executing the first trust manager when you extract the managers from the aggregate implementation.

(That's the problem it was designed to solve.)

So the reason that works is you're executing a trust manager that doesn't check algorithm constraints.

What are the values of jdk.tls.disabledAlgorithms and jdk.certpath.disabledAlgorithms for your JVM?

The default values are located in $JAVA_HOME/jre/lib/security/java.security

My best guess is that you've got a certificate in your chain (not necessarily the end entity cert) that is failing the algorithm constraints.

Can you post the entire chain?

--Daniel Fisher

On Fri, Dec 7, 2018 at 10:39 AM Tyler Camp <tylermat...@gmail.com> wrote:

--

[Tyler Camp]

Sorry for the delay, and thanks again for your time. The two other certs were exported and attached here. AVI-RootCA -> AVI-CA1 -> TOWNLINE