Replacing kube-up clusters with kops e2e clusters

66 views
Skip to first unread message

Mahamed Ali

unread,
Sep 29, 2023, 11:40:07 AM9/29/23
to d...@kubernetes.io, kubernetes-...@googlegroups.com, kubernetes-sig-c...@googlegroups.com
Hi All

I'm working on replacing kube-up clusters(clusters created from the cluster/* shell scripts in the kubernetes/kubernetes repo) with kops clusters for running Kubernetes e2e tests. The shell scripts that provision kube-up clusters are very fragile and no new features are being added to it.

I created a copy of the following critical release informing/blocking jobs:
  • ci-kubernetes-e2e-gci-gce-serial
    • I'm still stabilising this test. It was running with the wrong parallel value.
  • ci-kubernetes-e2e-gci-gce-slow
    • Most of the tests are passing except:
      • Kubernetes e2e suite.[It] [sig-cloud-provider-gcp] Addon update should propagate add-on file changes [Slow]
        • This test should be deleted, addon-manager is deprecated and kops doesn't use it
      • Kubernetes e2e suite.[It] [sig-network] LoadBalancers should be able to create an internal type load balancer [Slow]
  • ci-kubernetes-e2e-gci-gce-alpha-features
    • kops & AllAlpha feature gate is broken, I need to investigate this.
  • ci-kubernetes-gce-conformance-latest
    • The kops equivalent is passing
  • ci-kubernetes-e2e-gci-gce
    • Most of the tests are passing, except for these:
      • Kubernetes e2e suite.[It] [sig-apps] ReplicaSet should serve a basic image on each replica with a private image
      • Kubernetes e2e suite.[It] [sig-apps] ReplicationController should serve a basic image on each replica with a private image
      • Kubernetes e2e suite.[It] [sig-autoscaling] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios
      • Kubernetes e2e suite.[It] [sig-network] Firewall rule should have correct firewall rules for e2e cluster
      • Kubernetes e2e suite.[It] [sig-node] NodeProblemDetector should run without error
    • The sig-apps failures are happening because of a missing KubeletCredentialProvider. kops fix
    • The sig-autoscaling test needs to be rewritten. kops deploys coredns and the kube-dns-autoscaler resources with different names.
    • The sig-network test needs to be removed, it tests the existence of firewall rules created by kube-up clusters. kops cluster provisioning business logic is already well tested in kubernetes/kops repo.
    • NPD failure needs a closer look.

I would be grateful if you could:
  • review the proposed approach mentioned in the KEP
  • help me with removing or reworking e2e tests that don't work with kops. I have plenty of these.
Thanks

Mahamed Ali

Wojciech Tyczyński

unread,
Oct 2, 2023, 8:23:11 AM10/2/23
to c...@borg.dev, kubernetes-sig-scale, d...@kubernetes.io, kubernetes-...@googlegroups.com, kubernetes-sig-c...@googlegroups.com
+kubernetes-sig-scale 

Did you maybe try also with the informing SIG-scalaibility tests? These might require extensions to kops as we require tuning a bunch of non-obvious flags of different components.

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAAusGiNxkjOtSwJsCG67cdL25BBt1rtVEY2Q2%2BwHmhn34-h5iA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages