Replacing kube-up clusters with kops e2e clusters

Skip to first unread message

Mahamed Ali

Sep 29, 2023, 11:40:07 AM9/29/23
Hi All

I'm working on replacing kube-up clusters(clusters created from the cluster/* shell scripts in the kubernetes/kubernetes repo) with kops clusters for running Kubernetes e2e tests. The shell scripts that provision kube-up clusters are very fragile and no new features are being added to it.

I created a copy of the following critical release informing/blocking jobs:
  • ci-kubernetes-e2e-gci-gce-serial
    • I'm still stabilising this test. It was running with the wrong parallel value.
  • ci-kubernetes-e2e-gci-gce-slow
    • Most of the tests are passing except:
      • Kubernetes e2e suite.[It] [sig-cloud-provider-gcp] Addon update should propagate add-on file changes [Slow]
        • This test should be deleted, addon-manager is deprecated and kops doesn't use it
      • Kubernetes e2e suite.[It] [sig-network] LoadBalancers should be able to create an internal type load balancer [Slow]
  • ci-kubernetes-e2e-gci-gce-alpha-features
    • kops & AllAlpha feature gate is broken, I need to investigate this.
  • ci-kubernetes-gce-conformance-latest
    • The kops equivalent is passing
  • ci-kubernetes-e2e-gci-gce
    • Most of the tests are passing, except for these:
      • Kubernetes e2e suite.[It] [sig-apps] ReplicaSet should serve a basic image on each replica with a private image
      • Kubernetes e2e suite.[It] [sig-apps] ReplicationController should serve a basic image on each replica with a private image
      • Kubernetes e2e suite.[It] [sig-autoscaling] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios
      • Kubernetes e2e suite.[It] [sig-network] Firewall rule should have correct firewall rules for e2e cluster
      • Kubernetes e2e suite.[It] [sig-node] NodeProblemDetector should run without error
    • The sig-apps failures are happening because of a missing KubeletCredentialProvider. kops fix
    • The sig-autoscaling test needs to be rewritten. kops deploys coredns and the kube-dns-autoscaler resources with different names.
    • The sig-network test needs to be removed, it tests the existence of firewall rules created by kube-up clusters. kops cluster provisioning business logic is already well tested in kubernetes/kops repo.
    • NPD failure needs a closer look.

I would be grateful if you could:
  • review the proposed approach mentioned in the KEP
  • help me with removing or reworking e2e tests that don't work with kops. I have plenty of these.

Mahamed Ali

Wojciech Tyczyński

Oct 2, 2023, 8:23:11 AM10/2/23
to, kubernetes-sig-scale,,,

Did you maybe try also with the informing SIG-scalaibility tests? These might require extensions to kops as we require tuning a bunch of non-obvious flags of different components.

You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit
Reply all
Reply to author
0 new messages