Forwarding traffic over an incoming HTTP/2 connection as a reverse tunnel?

[Craig Ringer]

Hi all

Is it possible to have Envoy expose an _inbound_ HTTP/2 CONNECT or HTTP/3 CONNECT session as a listener over which it can route other proxied traffic?

I know Envoy can do HTTP upgrades and HTTP CONNECT termination when both the tunnelled connections and tunnelling connection are initiated from the same direction, but I want the tunneling connection used as a transport to be initiated from the destination end of the tunnelled connections. So Envoy accepts a remote-initiated connection to use as a reverse tunnel.

The underlying problem is that I have a hub-and-spoke configuration where I want to transit east-west traffic between the hub kube network and select services on the spokes' independent kube networks. The spokes are behind NAT and restrictive firewalls; they are not Internet-reachable and cannot be connected to via a VPN. There is no common private trunk or backbone that can be used to peer them. But the spokes *can* initiate and maintain connections to the hub. I'm trying to expose one specific service on each spoke to the hub so the hub can initiate queries to that specific service on each spoke, and nothing else.

You'd think this would be well-solved problem in the k8s space, but all the multi-network multi-meshpeering and federation options I've looked into (Istio, Consul, etc) require mutual reachability, where the "spokes" are public-routable from the hub. And I've been unable to find any sort of kube-friendly reverse tunneling operator or reverse-tunnel proxy management tool. The underlying tools like "ssh -R" exist and I found a HTTP/2 based reverse tunnel tool https://github.com/hotnops/gtunnel, but they'll all require quite a bit of glue to make them play nicely in kube.

So I was hoping to use my existing Istio and Envoy deployment with some custom Envoy rules/config or even a plugin to do these reverse tunnels. But I can't find any sign it's possible in Envoy. Am I missing something, or can this just not be done?

Related:

* I found one related thread, https://groups.google.com/g/envoy-users/c/9VDLqqWnE5w/m/xalnagmaBgAJ, but it had no replies

* Stack Overflow post https://stackoverflow.com/questions/77277028/can-http-2-connect-be-used-as-a-reverse-tunnel

* Istio specific question on Istio hub/spoke style peering https://discuss.istio.io/t/istio-multi-network-multi-mesh-hub-and-spoke-with-no-listening-ports-on-spokes-passive-reverse-tunnel/16865

[Craig Ringer]

Possibly relevant Envoy github issue: https://github.com/envoyproxy/envoy/issues/22697

Based on that post, this is probably not possible unless there some hard-to-find Envoy extension that implements it.

(I tried to back-link this from my post on the Istio forums, but its extremely aggressive spam filtering thinks that a post of any link is spam, hid my message, and blocked me. Yay.)

[Yan Avlasov]

This is not possible in Envoy. By design Envoy initiates connections to upstream services. I do not even have anything to suggest to solve your problem. Sorry.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/5046b28a-75cf-43cf-87ba-b68b2cb5f575n%40googlegroups.com.

[Craig Ringer]

On Wednesday, October 18, 2023 at 8:39:13 AM UTC+13 Yan Avlasov wrote:

This is not possible in Envoy. By design Envoy initiates connections to upstream services. I do not even have anything to suggest to solve your problem. Sorry.

Thanks. I appreciate the answer. I suspected as much from my reading of the docs, but it's good to have confirmed.

I suspect it might actually be possible to do it with multiple custom plugins but it'd get so complicated that it wouldn't make much sense to use Envoy for the job, rather than have Envoy talk to an external reverse-tunnel manager presented as a regular Service.

[Craig Ringer]

I'd like to share a relevant discussion on the new kube gateway SIG tracker too - https://github.com/kubernetes-sigs/gateway-api/discussions/2490

[Craig Ringer]

This Envoy github issue has a proposal for this functionality with a patch in progress: https://github.com/envoyproxy/envoy/issues/33320

Thanks to SO user @thiagogcm for pointing it out.