reverse-proxy() handler won't work with HTTPS endpoints

195 views
Skip to first unread message

Brad Wood

unread,
Nov 11, 2021, 1:54:20 PM11/11/21
to Undertow Dev
I'm fairly sure this is a bug, but I want to confirm prior to ticketing.  It's been pointed out to me that using an HTTPS endpoint with the reverse-proxy() handler always returns a 503 service unavailable.  I'd first like to note that a good job of exposing the issue isn't done here.  The actual error eluded me for a bit as it comes from the io.undertow.request logger instead of the expected io.undertow.proxy logger.  And furthermore, the log level of the message is "DEBUG" and not "ERROR" or at least "WARN".  

[DEBUG] io.undertow.request: Failed to connect
java.io.IOException: UT000065: SSL must be specified to connect to a https URL
        at io.undertow.client.http.HttpClientProvider.connect(HttpClientProvider.java:94)
        at io.undertow.client.UndertowClient.connect(UndertowClient.java:161)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.openConnection(ProxyConnectionPool.java:274)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:550)
        at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:340)
        at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:329)
java.io.IOException: UT000065: SSL must be specified to connect to a https URL
        at io.undertow.client.http.HttpClientProvider.connect(HttpClientProvider.java:94)
        at io.undertow.client.UndertowClient.connect(UndertowClient.java:161)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.openConnection(ProxyConnectionPool.java:274)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:550)
        at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:340)
        at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:329)

The issue appears to be that the ProxyHandlerBuilder class calls 

loadBalancingProxyClient.addHost(url);

for each host provided. The overloaded addHost() method which only accepts a URI sets the SSL flag to null and nowhere does the proxy client seem smart enough to look at the URL and default that setting accordingly. 

I think two changes are in order:
  • The reverse-proxy() handler builder or the proxy client needs to detect HTTPS URIs and set the SSL flag
  • The logging when this happens should be of an ERROR or WARN log level and ideally, present in the proxy logger.

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 

Stuart Douglas

unread,
Nov 11, 2021, 11:52:35 PM11/11/21
to Brad Wood, Undertow Dev
On Fri, 12 Nov 2021 at 05:54, Brad Wood <bdw...@gmail.com> wrote:
I'm fairly sure this is a bug, but I want to confirm prior to ticketing.  It's been pointed out to me that using an HTTPS endpoint with the reverse-proxy() handler always returns a 503 service unavailable.  I'd first like to note that a good job of exposing the issue isn't done here.  The actual error eluded me for a bit as it comes from the io.undertow.request logger instead of the expected io.undertow.proxy logger.  And furthermore, the log level of the message is "DEBUG" and not "ERROR" or at least "WARN".  

[DEBUG] io.undertow.request: Failed to connect
java.io.IOException: UT000065: SSL must be specified to connect to a https URL
        at io.undertow.client.http.HttpClientProvider.connect(HttpClientProvider.java:94)
        at io.undertow.client.UndertowClient.connect(UndertowClient.java:161)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.openConnection(ProxyConnectionPool.java:274)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:550)
        at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:340)
        at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:329)
java.io.IOException: UT000065: SSL must be specified to connect to a https URL
        at io.undertow.client.http.HttpClientProvider.connect(HttpClientProvider.java:94)
        at io.undertow.client.UndertowClient.connect(UndertowClient.java:161)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.openConnection(ProxyConnectionPool.java:274)
        at io.undertow.server.handlers.proxy.ProxyConnectionPool.connect(ProxyConnectionPool.java:550)
        at io.undertow.server.handlers.proxy.LoadBalancingProxyClient.getConnection(LoadBalancingProxyClient.java:340)
        at io.undertow.server.handlers.proxy.ProxyHandler$ProxyClientHandler.run(ProxyHandler.java:329)

The issue appears to be that the ProxyHandlerBuilder class calls 

loadBalancingProxyClient.addHost(url);

for each host provided. The overloaded addHost() method which only accepts a URI sets the SSL flag to null and nowhere does the proxy client seem smart enough to look at the URL and default that setting accordingly. 

I think two changes are in order:
  • The reverse-proxy() handler builder or the proxy client needs to detect HTTPS URIs and set the SSL flag

The issue is that there is no real way to set this from the predicate language, as the SSL object is basically a configured SSLContext, which is not really possible to represent in the predicate language. Maybe this could be expanded to pass in keystore/truststore into the proxy handle and set it up that way.
  • The logging when this happens should be of an ERROR or WARN log level and ideally, present in the proxy logger.

This should be detected at setup time, rather than on connection. The reason why this is logged at debug is because IOException is generally logged at debug (to prevent an attacker flooding the logs by just opening connections and then breaking them), the issue in this case is that we don't really discriminate between a client IO exception (where we have had issues talking to another server), vs a problem with our server connection.

Stuart
 

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 

--
You received this message because you are subscribed to the Google Groups "Undertow Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to undertow-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/undertow-dev/CALbQ1ok%3DjedLhd9kqE4jRbDxDJAzpUYAo_QjvnzY%2B3gMHMXEDw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Brad Wood

unread,
Nov 12, 2021, 12:22:59 AM11/12/21
to Stuart Douglas, Undertow Dev
The issue is that there is no real way to set this from the predicate language, as the SSL object is basically a configured SSLContext, which is not really possible to represent in the predicate language. Maybe this could be expanded to pass in keystore/truststore into the proxy handle and set it up that way.

You've thrown out the baby with the bathwater here by getting too focused on the edge cases.  Just default the SSL context to a generic one using the truststore defaulted in the JVM and boom, you've got 99% of use cases covered, right?  Let's get that in place, and then we can worry about the edge cases if and when they come up.  

This should be detected at setup time...

This shouldn't need to be detected at all because the proxy client should be smart enough to use a default SSL context  

... rather than on connection

I think the only errors we need to actually worry about catching ARE the ones that happen on connection. They are transient, only detectable when making a connection, and are important for being able to debug a non-working proxy.  I'm thinking low level stuff like connection refusal from the endpoint, or inability to negotiate an SSL connection.  

The reason why this is logged at debug is because IOException is generally logged at debug (to prevent an attacker flooding the logs by just opening connections and then breaking them), the issue in this case is that we don't really discriminate between a client IO exception (where we have had issues talking to another server), vs a problem with our server connection.

Isn't this what log rotation is for?  Also, you're still making assumptions about the log levels configured on the server.  If I'm logging DEBUG level logs (which is currently the only way to debug a proxy connection failure) then it's going to fill my logs anyway!  I say Undertow should just log what it needs to log using the correct severity and I'll worry about what I plan to do with those logs and when/how I rotate them. 

If we catch any connection failures in the proxy client, we can log appropriately at that point since we know it's part of the proxy.  But even then, I don't quite understand why the "client" classes are logging to the "request" logger.  To me, that logger is for incoming request plumbing, not outgoing requests intialted from undertow to a remote endpoint.  Are the UndertowClient or HttpClientProvider classes used to process incoming requests to undertow as well or are they using the wrong logger?

I don't mind taking a stab at a pull request for the default SSL context, but I'm not really sure what needs to happen.  I'm not super familiar with how org.xnio.ssl.XnioSsl is used.  I see an example of one being created in the HTTP/2 example, but I'm not sure if that's how you'd do it for real.

Thanks!

~Brad

Developer Advocate
Ortus Solutions, Corp 

ColdBox Platform: http://www.coldbox.org 


Brad Wood

unread,
Nov 17, 2021, 2:15:15 PM11/17/21
to Undertow Dev
I've entered a ticket here:


I'm still interested in the answers to my questions in my previous post such as why the client classes are using the "request" logger and how to best provide a default SSL context.

Flavia Rainone

unread,
Nov 17, 2021, 2:48:01 PM11/17/21
to Undertow Dev
Hi Brad!

I haven't been around for the past days, but now that I am, I saw your ticket and I'm going to have a careful look at what you're suggesting. I'll get back to you soon, but regarding logs we need to be aware that anything that can be used to fill up log output can be a security issue, so we need to thread carefully in this regard.

Also, Stuart's suggestions for properly taking care of this by properly configuring the key store and trust store is something to be taken into consideration. Naturally, that doesn't imply we shouldn't be using the default ssl context as well, in the absence of those configs. I need to think more about this.

I'm busy today taking care of some failures in the CI and doing an Undertow release, but I'll have a look at this one in the sequence and get back to you. Thanks for creating the ticket, and feel free to submit a suggested PR for the default SSLContext if you will.

Talk to you soon,
Flavia
Reply all
Reply to author
Forward
0 new messages