Using Minikube in public clouds

[Ted Ross]

For testing purposes, I would like to deploy minikube in a number of public cloud locations rather than paying for a full Kubernetes service.  The problem is that the Skupper connection-token generation doesn't work in this scenario, at least not with any minikube setup that I've tried.

First of all, has anyone found a way to make this work?

If not, would it be possible to add NodePort to the list of supported ingress strategies?  It might be necessary to tell Skupper what the external IP address/hostname is in the skupper init command (or possibly the skupper connection-token command).

I'd be happy to put together a pull request if this is considered a viable solution.

Thanks,

-Ted

[Gordon Sim]

On 01/03/2021 22:19, Ted Ross wrote:
> For testing purposes, I would like to deploy minikube in a number of
> public cloud locations rather than paying for a full Kubernetes
> service.  The problem is that the Skupper connection-token generation
> doesn't work in this scenario, at least not with any minikube setup that
> I've tried.
>
> First of all, has anyone found a way to make this work?
>
> If not, would it be possible to add NodePort to the list of supported
> ingress strategies?

Can you run minikube tunnel? If so, a loadbalancer service will work.
For NodePort you would need to now the IP of the specific pod to forward
the traffic I think(?) whereas with the loadbalancer, you should get a
stable IP for communicating with the service (which I guess you are then
exposing via some other, cloud specific mechanism).

>  It might be necessary to tell Skupper what the
> external IP address/hostname is in the skupper init command (or possibly
> the skupper connection-token command).

I think you have to do it on init. The name needs to be included in the
certificate for the local site, which you don't want to change every
time you generate a connection-token for use at another site.

As an alternative, whatabout using the minikube ingress controller? We
would still need to add support for ingress as a valid value for the
ingress option. That would be a more broadly useful approach and perhaps
even cleaner for your case (possibly at the expense of more effort though).

[Ted Ross]

On Monday, March 1, 2021 at 5:44:09 PM UTC-5 gs...@redhat.com wrote:

On 01/03/2021 22:19, Ted Ross wrote:
> For testing purposes, I would like to deploy minikube in a number of
> public cloud locations rather than paying for a full Kubernetes
> service.  The problem is that the Skupper connection-token generation
> doesn't work in this scenario, at least not with any minikube setup that
> I've tried.
>
> First of all, has anyone found a way to make this work?
>
> If not, would it be possible to add NodePort to the list of supported
> ingress strategies?

Can you run minikube tunnel? If so, a loadbalancer service will work.

I always use minikube tunnel.  I don't think skupper init will complete without the tunnel running.  The tunnel makes it possible to access the loadbalancer services from the same host, but the addresses used are private (in the 10.0.0.0/8 network) and cannot be accessed from off-site.

 

For NodePort you would need to now the IP of the specific pod to forward
the traffic I think(?) whereas with the loadbalancer, you should get a
stable IP for communicating with the service (which I guess you are then
exposing via some other, cloud specific mechanism).

Therein lies the problem.  The public IP must be known by Skupper so it can be put into the certificate and yaml for the connection token.  For NodePort, you have a stable IP address that lasts for the lifetime of the host.  It's just the host's IP address.  The NodePort on the host is mapped to the service in question.

 

>  It might be necessary to tell Skupper what the
> external IP address/hostname is in the skupper init command (or possibly
> the skupper connection-token command).

I think you have to do it on init. The name needs to be included in the
certificate for the local site, which you don't want to change every
time you generate a connection-token for use at another site.

Agreed.

 

As an alternative, whatabout using the minikube ingress controller? We
would still need to add support for ingress as a valid value for the
ingress option. That would be a more broadly useful approach and perhaps
even cleaner for your case (possibly at the expense of more effort though).

I've look at this as well.  It's definitely heavier weight and would likely be quite a bit more effort.

[Gordon Sim]

On 01/03/2021 23:23, Ted Ross wrote:
> I always use minikube tunnel.  I don't think skupper init will complete
> without the tunnel running.

That is true, unless you do --ingress=none (or --cluster-local in the
'old' terminology)

>  The tunnel makes it possible to access the
> loadbalancer services from the same host, but the addresses used are
> private (in the 10.0.0.0/8 network) and cannot be accessed from off-site.

[...]

> Therein lies the problem.  The public IP must be known by Skupper so it
> can be put into the certificate and yaml for the connection token.  For
> NodePort, you have a stable IP address that lasts for the lifetime of
> the host.  It's just the host's IP address.  The NodePort on the host is
> mapped to the service in question.

With NodePort you need the IP of the kubernetes node on which the pod is
running, no? I.e. a VM created by minikube, not the actual host on which
minikube is being run? At least for me a NodePort service on minikube is
not accessible via the actual host IP (unless I am doing something wrong?).

[Ted Ross]

If you are running minikube on a public VM, you would likely use --driver=none to avoid running VMs on VMs.  In this case, the address of the outer (and only) VM works correctly through a NodePort service.

Another option we're looking at is using Metal LB (an addon shipped with minikube) as a solution for LoadBalancer services.  One of the nice things about this is that you don't need to run 'minikube tunnel' to access the services from outside.  The current problem with this is that MetalLB only uses the 'nodePort' port from the LoadBalancer service.  I don't yet understand why an LB service has a nodePort (this is the second port listed after the colon in the PORTS column in 'kubectl get services').

 

[Gordon Sim]

On 02/03/2021 18:02, Ted Ross wrote:
> If you are running minikube on a public VM, you would likely use
> --driver=none to avoid running VMs on VMs.  In this case, the address of
> the outer (and only) VM works correctly through a NodePort service.

Makes sense, I was not aware you could run without a driver.

I have no objections to either a 'nodeport' option for --ingress, or to
the addition of an option that allows the hostname to be supplied.

[Gordon Sim]

Fyi, I posted a PR for this if still of interest:
https://github.com/skupperproject/skupper/pull/497