Hosting a ship on Kubernetes: how to make the Ames UDP port accessible?

[lavlyn-litmeg]

TL;DR: Does anyone with experience in hosting ships in Kubernetes have advice how to handle the UDP port mapping?

A little background: I'm working on becoming a Urbit hosting provider. I plan to host the ships on Kubernetes (DigitalOcean managed cluster) and I have most of it working already, like spinning up a new comet or planet and making it accessible via HTTP.

But what gives me trouble is making the Ames UDP port accessible on an external IP.

In the cloud provider I use, I cannot give the Kubernetes Service an ExternalIP directly, which would be the easiest solution, I think. So I'm left with either NodePort or LoadBalancer service types. Even though I can get LoadBalancer working, I don't think this is what I need since each Ship is unique. This leaves me only with NodePort. But here I'm not sure how I can make sure that the ship publishes its correct IP and Ames port to the network when it is bound to the cluserIP, which is internal, and is forwarded traffic from a different external port. Does Ames have the kind of NAT traversal chops for this to work? Also, AFAIK, NodePort opes all bound ports on all nodes. So if I host a lot of ships then that would mean a lot of ports (but this is not my main worry right now :).

Any advice is appreciated,

-luke

[Anton Dyudin]

Per https://urbit.org/using/running/hosting/#setting-up-a-basic-firewall, you probably want to set a "static" port with -p, so the one urbit is told about is the one it's externally accessible at. As far as IPs are concerned, there's certainly considerable NAT traversal built into urbit, though as a host it would probably be polite to produce a direct IP if possible and not put load on the NAT servers - really they're meant more for laptops, phones, and other uses where a public IP was never in the cards to begin with.

Though one existing might be sufficient for urbit to determine it by talking to a star once and then tell peers about it directly, not sure how robust that part of the system is.

As far as MxN ports, keeping your http config as is, it does look like you can configure nodePorts one at a time: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#exposing-the-service

--
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@urbit.org.

[Lukas Bühler]

Thank you for this.

So maybe to rephrase it so I can find the right solution in Kubernetes:

1) how does urbit deal with port forwarding of the UDP port? Does the externally accessible port need to be the same as what is passed in via -p?

2) how does Ames figure out its external IP? Does it have to be the IP of the host OS? Or are there other discovery mechanisms? In other words, can I proxy udp traffic to the host of my ship?

3) is there an easy way to check I the ship’s dojo what IP and port other peers use to connect to it? This would be helpful to figure out if my ship is  a good citizen and does direct p2p or if all traffic is routed through a galaxy (or star?).

[Philip Monk]

Ames doesn't determine its own IP.  Your ship periodically pings your galaxy, and your galaxy stores the route (ip+port) that it received the ping from.  Other ships get that route from your galaxy.

This means you can proxy or port forward as much as you want as long as you can receive packets at the same route as you sent them.  Load balancer setups often send from a different route than they can receive, and that won't work.  Or rather, you'll have connectivity but only via forwarding through your galaxy, not a direct route.  The reason it'll work at all is that you can receive packets forwarded by your galaxy for about 30 seconds after you ping (before the UDP pinhole closes) and we ping every 30 seconds.

If you run |start %dbug in the dojo, you can navigate your browser to /~debug and go to the ames tab.  There, you can see what route you have for each other ship.  You want to make sure that stays "direct" when viewed from another ship, even if you don't send any packets for a few minutes, then run |hi again.

On Fri, Mar 19, 2021 at 10:43 AM, Lukas Bühler <lukas....@gmail.com> wrote:

Thank you for this.

So maybe to rephrase it so I can find the right solution in Kubernetes:

1) how does urbit deal with port forwarding of the UDP port? Does the externally accessible port need to be the same as what is passed in via -p?

2) how does Ames figure out its external IP? Does it have to be the IP of the host OS? Or are there other discovery mechanisms? In other words, can I proxy udp traffic to the host of my ship?

3) is there an easy way to check I the ship’s dojo what IP and port other peers use to connect to it? This would be helpful to figure out if my ship is  a good citizen and does direct p2p or if all traffic is routed through a galaxy (or star?).

On Thu, Mar 18, 2021 at 9:11 PM Anton Dyudin <antechno777@gmail.com> wrote:

Per https://urbit.org/using/running/hosting/#setting-up-a-basic-firewall, you probably want to set a "static" port with -p, so the one urbit is told about is the one it's externally accessible at. As far as IPs are concerned, there's certainly considerable NAT traversal built into urbit, though as a host it would probably be polite to produce a direct IP if possible and not put load on the NAT servers - really they're meant more for laptops, phones, and other uses where a public IP was never in the cards to begin with.

Though one existing might be sufficient for urbit to determine it by talking to a star once and then tell peers about it directly, not sure how robust that part of the system is.

As far as MxN ports, keeping your http config as is, it does look like you can configure nodePorts one at a time: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#exposing-the-service

On Thu, Mar 18, 2021 at 12:53 PM lavlyn-litmeg <lukas.buehler@gmail.com> wrote:

TL;DR: Does anyone with experience in hosting ships in Kubernetes have advice how to handle the UDP port mapping?

A little background: I'm working on becoming a Urbit hosting provider. I plan to host the ships on Kubernetes (DigitalOcean managed cluster) and I have most of it working already, like spinning up a new comet or planet and making it accessible via HTTP.

But what gives me trouble is making the Ames UDP port accessible on an external IP.

In the cloud provider I use, I cannot give the Kubernetes Service an ExternalIP directly, which would be the easiest solution, I think. So I'm left with either NodePort or LoadBalancer service types. Even though I can get LoadBalancer working, I don't think this is what I need since each Ship is unique. This leaves me only with NodePort. But here I'm not sure how I can make sure that the ship publishes its correct IP and Ames port to the network when it is bound to the cluserIP, which is internal, and is forwarded traffic from a different external port. Does Ames have the kind of NAT traversal chops for this to work? Also, AFAIK, NodePort opes all bound ports on all nodes. So if I host a lot of ships then that would mean a lot of ports (but this is not my main worry right now :).

Any advice is appreciated,

-luke

--

To unsubscribe from this group and stop receiving emails from it, send an email to dev+unsubscribe@urbit.org.

[Lukas Bühler]

Ah, perfect! That makes it clear how I can make this work—probability using k8s NodePort Services since the IP and port will stay stable for days or weeks.

I did not know about %dbug; I’m delighted.

On Fri, Mar 19, 2021 at 8:21 PM Philip Monk <phi...@tlon.io> wrote:

Ames doesn't determine its own IP.  Your ship periodically pings your galaxy, and your galaxy stores the route (ip+port) that it received the ping from.  Other ships get that route from your galaxy.

This means you can proxy or port forward as much as you want as long as you can receive packets at the same route as you sent them.  Load balancer setups often send from a different route than they can receive, and that won't work.  Or rather, you'll have connectivity but only via forwarding through your galaxy, not a direct route.  The reason it'll work at all is that you can receive packets forwarded by your galaxy for about 30 seconds after you ping (before the UDP pinhole closes) and we ping every 30 seconds.

If you run |start %dbug in the dojo, you can navigate your browser to /~debug and go to the ames tab.  There, you can see what route you have for each other ship.  You want to make sure that stays "direct" when viewed from another ship, even if you don't send any packets for a few minutes, then run |hi again.

On Fri, Mar 19, 2021 at 10:43 AM, Lukas Bühler <lukas....@gmail.com> wrote:

Thank you for this.

So maybe to rephrase it so I can find the right solution in Kubernetes:

1) how does urbit deal with port forwarding of the UDP port? Does the externally accessible port need to be the same as what is passed in via -p?

2) how does Ames figure out its external IP? Does it have to be the IP of the host OS? Or are there other discovery mechanisms? In other words, can I proxy udp traffic to the host of my ship?

3) is there an easy way to check I the ship’s dojo what IP and port other peers use to connect to it? This would be helpful to figure out if my ship is  a good citizen and does direct p2p or if all traffic is routed through a galaxy (or star?).

On Thu, Mar 18, 2021 at 9:11 PM Anton Dyudin <antec...@gmail.com> wrote:

Per https://urbit.org/using/running/hosting/#setting-up-a-basic-firewall, you probably want to set a "static" port with -p, so the one urbit is told about is the one it's externally accessible at. As far as IPs are concerned, there's certainly considerable NAT traversal built into urbit, though as a host it would probably be polite to produce a direct IP if possible and not put load on the NAT servers - really they're meant more for laptops, phones, and other uses where a public IP was never in the cards to begin with.

Though one existing might be sufficient for urbit to determine it by talking to a star once and then tell peers about it directly, not sure how robust that part of the system is.

As far as MxN ports, keeping your http config as is, it does look like you can configure nodePorts one at a time: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#exposing-the-service

On Thu, Mar 18, 2021 at 12:53 PM lavlyn-litmeg <lukas....@gmail.com> wrote:

TL;DR: Does anyone with experience in hosting ships in Kubernetes have advice how to handle the UDP port mapping?

A little background: I'm working on becoming a Urbit hosting provider. I plan to host the ships on Kubernetes (DigitalOcean managed cluster) and I have most of it working already, like spinning up a new comet or planet and making it accessible via HTTP.

But what gives me trouble is making the Ames UDP port accessible on an external IP.

In the cloud provider I use, I cannot give the Kubernetes Service an ExternalIP directly, which would be the easiest solution, I think. So I'm left with either NodePort or LoadBalancer service types. Even though I can get LoadBalancer working, I don't think this is what I need since each Ship is unique. This leaves me only with NodePort. But here I'm not sure how I can make sure that the ship publishes its correct IP and Ames port to the network when it is bound to the cluserIP, which is internal, and is forwarded traffic from a different external port. Does Ames have the kind of NAT traversal chops for this to work? Also, AFAIK, NodePort opes all bound ports on all nodes. So if I host a lot of ships then that would mean a lot of ports (but this is not my main worry right now :).

Any advice is appreciated,

-luke

--

To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@urbit.org.