crossbar on k8, sidecars and so on

[tobias.o...@gmail.com]

Hi everyone,

we wanted to gather a bit of feedback regarding our upcoming support for running crossbar on k8 (beyond what's possible today already .. eg single-node router infra)

as some of you are aware, we've long announced router-to-router WAMP links, but we are only now ready to make that fly in a sensible and scalable way. this has been delayed because of various other activities/projects/customer stuff, but it is finally coming now. here are some very quick screenshots of WAMP PubSub transparently routed between 5 crossbar nodes: https://crossbario.com/docs/crossbarfx/scaling/mesh_links.html

anyways. wanted to talk about k8;)

so here is an example of a deployment of the current design:

(1) Clients resolve DNS pointing to a layer 4 load balancer (eg AWS ELB).

(2) the load balancer distributes the incoming connections (arbitrarily, but sticky) onto the backend nodes, which are run by k8 as a daemonset (on every node of the k8 cluster)

(3) the k8 Pods contain sidecar routers which connect to the core router (in the daemonset, on the respective localhost)

(4) app containers/microservices connect only to the sidecare routers, which have an uplink to the (node local) core routers

(5) the core routers connect to each other in a full mesh

so in this architecture, the app containers / microservices can talk to each in 1 hop among each other (via the sidecar router), and can talk to app containers / microservices in other Pods via the core routers (in the daemonset), while the incoming connections are all going to the core routers, from where messages are routed to the local sidecar, or another core router and then sidecar router.

all of that routing is fully transparent at the endpoints (app containers / microservices and clients)

===

I simplified above diagram in one respect: all crossbar nodes (edge and core) connect to a crossbar controller, which in turn directly talks to k8 (eg to get notified when the daemonset changes etc)

anyways, we wanted to take the chance to listen to users and gather early feedback on this design. what do you think?

Cheers,

/Tobias