For once, I'm posting a question I already know the answer to :) It took me days to figure this out and there is precious little information at all on the Internet or in the Undertow docs on this. I was only able to solve it after lots of testing and reading the source code.
Basically, if you are using the client cert authentication mechanism with SSL renegotiation enabled (the default) or just attempting to manually ask for renegotiation during an SSL request in order to get the user's browser to prompt them for a client cert on pages of your choice, it likely won't work. The reason it doesn't work isn't reflected anywhere in the docs, nor in any sort of log messages from Undertow. But the hang up is that HTTP/2 does not support SSL renegotiation AND TLSv1.3 also does not support SSL renegotiation. So, for any server in which you want to have a user hit the home page without being asked for a cert (client auth mode: NOT_REQUESTED), and then ask for renegotiation on the requests where you want to collect a client cert, you will need to disable both HTTP/2 and TLSv1.3 or nothing will happen in the browser and Undertow's authentication will just fail without logging any errors.
I have found IIS appears to do the same thing when configuring the main site to "ignore" client certs and then configuring certain directories or locations in require client certs.
This has some alarming consequences in regards to the fact that I assume this workaround will only work for so long until such a point comes that TLSv1.3 or higher is required by most browsers. Hopefully, the JDK will have support for
TLS's new method of renegotiation which is
not currently supported by then.
Thanks!
~Brad
Developer Advocate
Ortus Solutions, Corp