Re: Replacing kube-up clusters with kops e2e clusters

[Wojciech Tyczyński]

+kubernetes-sig-scale 

Did you maybe try also with the informing SIG-scalaibility tests? These might require extensions to kops as we require tuning a bunch of non-obvious flags of different components.

On Fri, Sep 29, 2023 at 5:40 PM Mahamed Ali <c...@borg.dev> wrote:

Hi All

I'm working on replacing kube-up clusters(clusters created from the cluster/* shell scripts in the kubernetes/kubernetes repo) with kops clusters for running Kubernetes e2e tests. The shell scripts that provision kube-up clusters are very fragile and no new features are being added to it.

I created a copy of the following critical release informing/blocking jobs:

• ci-kubernetes-e2e-gci-gce-serial
• I'm still stabilising this test. It was running with the wrong parallel value.
• ci-kubernetes-e2e-gci-gce-slow
• Most of the tests are passing except:
• Kubernetes e2e suite.[It] [sig-cloud-provider-gcp] Addon update should propagate add-on file changes [Slow]
• This test should be deleted, addon-manager is deprecated and kops doesn't use it
• Kubernetes e2e suite.[It] [sig-network] LoadBalancers should be able to create an internal type load balancer [Slow]
• Caused by https://github.com/kubernetes/cloud-provider-gcp/issues/612 
• ci-kubernetes-e2e-gci-gce-alpha-features
• kops & AllAlpha feature gate is broken, I need to investigate this.
• ci-kubernetes-gce-conformance-latest
• The kops equivalent is passing
• ci-kubernetes-e2e-gci-gce
• Most of the tests are passing, except for these:
• Kubernetes e2e suite.[It] [sig-apps] ReplicaSet should serve a basic image on each replica with a private image
• Kubernetes e2e suite.[It] [sig-apps] ReplicationController should serve a basic image on each replica with a private image
• Kubernetes e2e suite.[It] [sig-autoscaling] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios
• Kubernetes e2e suite.[It] [sig-network] Firewall rule should have correct firewall rules for e2e cluster
• Kubernetes e2e suite.[It] [sig-node] NodeProblemDetector should run without error
• The sig-apps failures are happening because of a missing KubeletCredentialProvider. kops fix
• The sig-autoscaling test needs to be rewritten. kops deploys coredns and the kube-dns-autoscaler resources with different names.
• The sig-network test needs to be removed, it tests the existence of firewall rules created by kube-up clusters. kops cluster provisioning business logic is already well tested in kubernetes/kops repo.
• NPD failure needs a closer look.

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeup-to-kops#Summary

I would be grateful if you could:

• review the proposed approach mentioned in the KEP
• help me with removing or reworking e2e tests that don't work with kops. I have plenty of these.

Thanks

Mahamed Ali

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAAusGiNxkjOtSwJsCG67cdL25BBt1rtVEY2Q2%2BwHmhn34-h5iA%40mail.gmail.com.

[Mahamed Ali]