Is it possible to set up a K8s cluster federation with two local clusters?

[Qian Zhang]

Hi,

I have two local K8s clusters which are running in two different racks, now I'd like to create a single deployment to utilize resources from both of these two clusters, e.g., some pods of the deployment are in one cluster while other pods of the same deployment are in another cluster. This seems what K8s cluster federation can handle, so is it possible to set up a federation via "kubefed" and join these two cluster into the federation?

Regards,

Qian

[Qian Zhang]

And will there be a problem for service discovery (kube-dns) across the two clusters in this case?

[Madhusudan C.S.]

It should be possible to use Cluster Federation it its current state if you just want to create a Deployment resource that spans across two clusters. However, if you want a service that spans two or more clusters and want service discovery across them, there are some caveats:

1. As of Kubernetes v1.5.x, which is the latest stable release right now, Google CloudDNS and AWS Route53 are the only two DNS providers that are supported for cross-cluster service discovery. While `kube-dns` is used for cluster local services, these DNS providers are programmed for federated (cross-cluster) services. It is important to note that it is still possible to use these DNS providers with your local clusters. However, your service DNS entries will be globally visible to all the DNS servers on the public internet. Having said all that, CoreDNS[0] DNS provider support is coming in Kubernetes v1.6. ETA March, 22nd[1]. This will let you run your own DNS server for federated service discovery. 

2. Federated Services today create a separate service shard in each underlying cluster in a federation. And to create DNS records for service discovery, federation depends on the load balancer addresses allocated by the cluster service shards. If you are on an on-prem environment and don't have a load balancer implementation for your provider, federation will probably not be able to create DNS entries for these services.

3. kubefed, as it was launched in v1.5, was alpha and had problems with clusters in on-prem environments. We have fixed almost all those known issues now and kubefed should be able to build a federation control plane using on-prem clusters when it is released as beta in v1.6. However, you should be able to deploy a federation control plane by hand today using Kelsey's tutorial[2]. Just skip the cluster provisioning step in the Bootrapping Tasks list. (Federation The Hard Way ;-))

[0] https://coredns.io/

[1] https://github.com/kubernetes/features/blob/master/release-1.6/release-1.6.md#16-release-schedule

[2] https://github.com/kelseyhightower/kubernetes-cluster-federation

[Qian Zhang]

Thanks Madhusudan for the sharing which is very helpful!!!

And I have a few more questions:

1. I tried to use one of my on-prem clusters as host cluster to deploy federation control plane via "kubefed init", and I found federation-controller-manager failed to start with the error below:

F0302 05:30:19.607781       1 controllermanager.go:160] Cloud provider could not be initialized: could not init DNS provider "google-clouddns": google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

So I guess I need to specify the option "--dns-provider=coredns" to "kubefed init" since I do not have "Google CloudDNS" and "AWS Route53", right? And do you know if there is any doc about how to configure federation to work with CoreDNS? I guess I need to specify another option "--dns-provider-config" to "kubefed init" which was introduced by this PR recently, but I do not know what value of this option should be for CoreDNS.

2. I'd like to know more about how federation-controller-manager works with Google CloudDNS and AWS Route53. Before this PR was merged, federation-controller-manage will be started by "kubefed init" without setting "--dns-provider-config", then how can federation-controller-manage know how to access Google CloudDNS and AWS Route53? I guess at least federation-controller-manager should know their URL so that it can call their API to generate DNS records in them.

3.

If you are on an on-prem environment and don't have a load balancer implementation for your provider, federation will probably not be able to create DNS entries for these services.

So even I use CoreDNS in my on-prem environment, I still need a load balancer in each of my two on-prem environment, right? But as I know usually load balancer is only available in the on-cloud environment like GCE/AWS, so can you please let me know how I can have a load balancer in a on-prem env?

4. Based on the information I got from this doc, it seems after I join my two on-prem clusters into federation, I also need to update the KubeDNS in these two on-prem clusters, right?

5. I see there are two ways to set up federation: kubefed and `federation-up` & `deploy.sh`, are both of them the official way? Which one will be the long term solution in future?

6. How do we handle the HA of federation control plane itself? Since it is running in a host cluster, what if the host cluster is down?