Where is the testcases for "Feature:Performance"?

Dave Chen

unread,

Dec 9, 2019, 3:17:35 AM12/9/19

to kubernetes-sig-scale

Hi,

Lots of posts tell me to run e2e performance test with Ginkgo focus specified to "Feature:Performance", but I have no idea which testcases actually run with the commands like below,

/usr/bin/ginkgo /usr/bin/e2e.test -- --kubeconfig=../resources/kubeconfig.kubemark --ginkgo.flakeAttempts=1 --host=http://127.0.0.1:8080 --repo-root=/home/dave/Project/kubernetes --prefix=e2e --network=e2e --provider=kubemark --cluster-monitoring-mode=standalone --dns-domain=cluster.local --ginkgo.slowSpecThreshold=300 --master-os-distro=ubuntu --node-os-distro=debian --num-nodes=2 --report-dir=/var/log/kubernetes/e2e --e2e-verify-service-account=false --dump-logs-on-failure=false --delete-namespace=false --ginkgo.focus=\[Feature:Performance\]

I grepped the whole source and unable to find any tests associated with regex as "[Feature:Performance]"， and when I run the command above, seem like all of the e2e testcases are executed one by one, and lots of failure found finally.

What I am want to do is using kubemark as the tool to test the performance of my local cluster, after digging into the source, it goes to the above command to run the performance test, could anyone kindly let me:

- which kinds of testcases are mapped to "Feature:Performance]" regex? what's the rule?

- Is kubemark still the right tool to test the performance of local cluster?

Any comments is appreciated! Many many thanks!

Best Regards,

Dave

Wojciech Tyczynski

unread,

Dec 9, 2019, 4:10:37 AM12/9/19

to Dave Chen, kubernetes-sig-scale

On Mon, Dec 9, 2019 at 9:17 AM Dave Chen <dave.j...@gmail.com> wrote:

Hi,

Lots of posts tell me to run e2e performance test with Ginkgo focus specified to "Feature:Performance", but I have no idea which testcases actually run with the commands like below,

/usr/bin/ginkgo /usr/bin/e2e.test -- --kubeconfig=../resources/kubeconfig.kubemark --ginkgo.flakeAttempts=1 --host=http://127.0.0.1:8080 --repo-root=/home/dave/Project/kubernetes --prefix=e2e --network=e2e --provider=kubemark --cluster-monitoring-mode=standalone --dns-domain=cluster.local --ginkgo.slowSpecThreshold=300 --master-os-distro=ubuntu --node-os-distro=debian --num-nodes=2 --report-dir=/var/log/kubernetes/e2e --e2e-verify-service-account=false --dump-logs-on-failure=false --delete-namespace=false --ginkgo.focus=\[Feature:Performance\]

I grepped the whole source and unable to find any tests associated with regex as "[Feature:Performance]"， and when I run the command above, seem like all of the e2e testcases are executed one by one, and lots of failure found finally.

What I am want to do is using kubemark as the tool to test the performance of my local cluster, after digging into the source, it goes to the above command to run the performance test, could anyone kindly let me:
- which kinds of testcases are mapped to "Feature:Performance]" regex? what's the rule?

These tests were removed some time ago:

https://github.com/kubernetes/kubernetes/pull/83322

We have already migrated the tests to use ClusterLoader framework:

https://github.com/kubernetes/perf-tests/tree/master/clusterloader2/

You can look how we run scalability tests in the default config:

https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-scalability/sig-scalability-release-blocking-jobs.yaml

- Is kubemark still the right tool to test the performance of local cluster?

Kubemark is still the right tool, and we use it all the time, e.g:

https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml#L200

What do you mean by "local cluster" in this?

Any comments is appreciated! Many many thanks!

Best Regards,
Dave

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/82a3d2eb-d69a-4167-83e5-46460bf1a37f%40googlegroups.com.

Dave Chen

unread,

Dec 9, 2019, 4:24:47 AM12/9/19

to kubernetes-sig-scale

Thank you Wojciech Tyczynski!

local cluster is the cluster setup locally and want to test the performance of the cluster, instead of using the provider like GCE, I saw the the script is using the GCE by default.

So, is that possible to test the performance of my cluster using kubemark?

and what's the below command trying to run if the testcases has been migrated to another place?

usr/bin/ginkgo /usr/bin/e2e.test -- --kubeconfig=../resources/kubeconfig.kubemark --ginkgo.flakeAttempts=1 --host=http://127.0.0.1:8080 --repo-root=/home/dave/Project/kubernetes --prefix=e2e --network=e2e --provider=kubemark --cluster-monitoring-mode=standalone --dns-domain=cluster.local --ginkgo.slowSpecThreshold=300 --master-os-distro=ubuntu --node-os-distro=debian --num-nodes=2 --report-dir=/var/log/kubernetes/e2e --e2e-verify-service-account=false --dump-logs-on-failure=false --delete-namespace=false --ginkgo.focus=\[Feature:Performance\]

On Monday, December 9, 2019 at 5:10:37 PM UTC+8, Wojciech Tyczynski wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-scale+unsub...@googlegroups.com.

Dave Chen

unread,

Dec 9, 2019, 4:31:58 AM12/9/19

to kubernetes-sig-scale

I still don't how the Ginkgo focus string "[Feature:Performance]" mapping to the the scalability test that is removed,

https://github.com/kubernetes/kubernetes/pull/83322/files

Could you pls let me the rule pls? I cannot find the string in the removed testcases either.

Thank you a lot!

Wojciech Tyczynski

unread,

Dec 9, 2019, 7:16:59 AM12/9/19

to Dave Chen, kubernetes-sig-scale

answers lnline

On Mon, Dec 9, 2019 at 10:31 AM Dave Chen <dave.j...@gmail.com> wrote:

I still don't how the Ginkgo focus string "[Feature:Performance]" mapping to the the scalability test that is removed,
https://github.com/kubernetes/kubernetes/pull/83322/files

It's not - Feature:Performance currently doesn't map to anything - it's something that wasn't properly cleaned up.

Could you pls let me the rule pls? I cannot find the string in the removed testcases either.

Thank you a lot!

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/CANS-GzAedd-v6mWX_Kerd4w1unyfMVyV7Hn57fq-bD4xo-9eYQ%40mail.gmail.com.

Wojciech Tyczynski

unread,

Dec 9, 2019, 7:20:52 AM12/9/19

to Dave Chen, kubernetes-sig-scale

On Mon, Dec 9, 2019 at 10:24 AM Dave Chen <dave.j...@gmail.com> wrote:

Thank you Wojciech Tyczynski!

local cluster is the cluster setup locally and want to test the performance of the cluster, instead of using the provider like GCE, I saw the the script is using the GCE by default.
So, is that possible to test the performance of my cluster using kubemark?

I think you're mixing couple things. Kubemark is not something that tests performance.

Kubemark is a tool/framework to simulate real cluster with smaller compute power.

We create what we call "hollow nodes" that simulate node components (Kubelet, KubeProxy, ...) and run them as pods with a bunch of subcomponents faked (e.g. runtime, iptables).

and what's the below command trying to run if the testcases has been migrated to another place?

I don't know the commands - you can probably reverse-engineer them from what our tests are doing, e.g.:

https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-scalability/sig-scalability-release-blocking-jobs.yaml#L126

usr/bin/ginkgo /usr/bin/e2e.test -- --kubeconfig=../resources/kubeconfig.kubemark --ginkgo.flakeAttempts=1 --host=http://127.0.0.1:8080 --repo-root=/home/dave/Project/kubernetes --prefix=e2e --network=e2e --provider=kubemark --cluster-monitoring-mode=standalone --dns-domain=cluster.local --ginkgo.slowSpecThreshold=300 --master-os-distro=ubuntu --node-os-distro=debian --num-nodes=2 --report-dir=/var/log/kubernetes/e2e --e2e-verify-service-account=false --dump-logs-on-failure=false --delete-namespace=false --ginkgo.focus=\[Feature:Performance\]

On Monday, December 9, 2019 at 5:10:37 PM UTC+8, Wojciech Tyczynski wrote:

On Mon, Dec 9, 2019 at 9:17 AM Dave Chen <dave....@gmail.com> wrote:
Hi,

Lots of posts tell me to run e2e performance test with Ginkgo focus specified to "Feature:Performance", but I have no idea which testcases actually run with the commands like below,

/usr/bin/ginkgo /usr/bin/e2e.test -- --kubeconfig=../resources/kubeconfig.kubemark --ginkgo.flakeAttempts=1 --host=http://127.0.0.1:8080 --repo-root=/home/dave/Project/kubernetes --prefix=e2e --network=e2e --provider=kubemark --cluster-monitoring-mode=standalone --dns-domain=cluster.local --ginkgo.slowSpecThreshold=300 --master-os-distro=ubuntu --node-os-distro=debian --num-nodes=2 --report-dir=/var/log/kubernetes/e2e --e2e-verify-service-account=false --dump-logs-on-failure=false --delete-namespace=false --ginkgo.focus=\[Feature:Performance\]

I grepped the whole source and unable to find any tests associated with regex as "[Feature:Performance]"， and when I run the command above, seem like all of the e2e testcases are executed one by one, and lots of failure found finally.

What I am want to do is using kubemark as the tool to test the performance of my local cluster, after digging into the source, it goes to the above command to run the performance test, could anyone kindly let me:
- which kinds of testcases are mapped to "Feature:Performance]" regex? what's the rule?

These tests were removed some time ago:
https://github.com/kubernetes/kubernetes/pull/83322

We have already migrated the tests to use ClusterLoader framework:
https://github.com/kubernetes/perf-tests/tree/master/clusterloader2/

You can look how we run scalability tests in the default config:
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-scalability/sig-scalability-release-blocking-jobs.yaml

- Is kubemark still the right tool to test the performance of local cluster?

Kubemark is still the right tool, and we use it all the time, e.g:
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml#L200

What do you mean by "local cluster" in this?

Any comments is appreciated! Many many thanks!

Best Regards,
Dave

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-s...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/82a3d2eb-d69a-4167-83e5-46460bf1a37f%40googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/bbec377a-8e30-40ee-a560-c7608b643e28%40googlegroups.com.

Dave Chen

unread,

Dec 9, 2019, 9:03:38 AM12/9/19

to kubernetes-sig-scale

On Monday, December 9, 2019 at 8:20:52 PM UTC+8, Wojciech Tyczynski wrote:

On Mon, Dec 9, 2019 at 10:24 AM Dave Chen <dave....@gmail.com> wrote:
Thank you Wojciech Tyczynski!

local cluster is the cluster setup locally and want to test the performance of the cluster, instead of using the provider like GCE, I saw the the script is using the GCE by default.
So, is that possible to test the performance of my cluster using kubemark?

I think you're mixing couple things. Kubemark is not something that tests performance.
Kubemark is a tool/framework to simulate real cluster with smaller compute power.
We create what we call "hollow nodes" that simulate node components (Kubelet, KubeProxy, ...) and run them as pods with a bunch of subcomponents faked (e.g. runtime, iptables).

Got it, so that the new framework introduced below still works with kubemark provider, just need to specify "--provider=kubemark" like it was with "e2e.test", right?

https://github.com/kubernetes/perf-tests/tree/master/clusterloader2/

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-scale+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/82a3d2eb-d69a-4167-83e5-46460bf1a37f%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-scale+unsub...@googlegroups.com.

Wojciech Tyczynski

unread,

Dec 9, 2019, 9:08:31 AM12/9/19

to Dave Chen, kubernetes-sig-scale

On Mon, Dec 9, 2019 at 3:03 PM Dave Chen <dave.j...@gmail.com> wrote:

On Monday, December 9, 2019 at 8:20:52 PM UTC+8, Wojciech Tyczynski wrote:

On Mon, Dec 9, 2019 at 10:24 AM Dave Chen <dave....@gmail.com> wrote:
Thank you Wojciech Tyczynski!

local cluster is the cluster setup locally and want to test the performance of the cluster, instead of using the provider like GCE, I saw the the script is using the GCE by default.
So, is that possible to test the performance of my cluster using kubemark?

I think you're mixing couple things. Kubemark is not something that tests performance.
Kubemark is a tool/framework to simulate real cluster with smaller compute power.
We create what we call "hollow nodes" that simulate node components (Kubelet, KubeProxy, ...) and run them as pods with a bunch of subcomponents faked (e.g. runtime, iptables).

Got it, so that the new framework introduced below still works with kubemark provider, just need to specify "--provider=kubemark" like it was with "e2e.test", right?

https://github.com/kubernetes/perf-tests/tree/master/clusterloader2/

correct

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-s...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/82a3d2eb-d69a-4167-83e5-46460bf1a37f%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-s...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/bbec377a-8e30-40ee-a560-c7608b643e28%40googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-scale/9ba7afd6-3df4-4479-afd8-af192a4179d6%40googlegroups.com.

Dave Chen

unread,

Dec 9, 2019, 9:09:29 AM12/9/19

to kubernetes-sig-scale

On Monday, December 9, 2019 at 8:16:59 PM UTC+8, Wojciech Tyczynski wrote:

answers lnline

On Mon, Dec 9, 2019 at 10:31 AM Dave Chen <dave....@gmail.com> wrote:
I still don't how the Ginkgo focus string "[Feature:Performance]" mapping to the the scalability test that is removed,
https://github.com/kubernetes/kubernetes/pull/83322/files

It's not - Feature:Performance currently doesn't map to anything - it's something that wasn't properly cleaned up.

this is the reason why all the e2e testing got executed when I run e2e.test with the "Feature:Performance", right? I properly need to look into the e2e framework to figure out how it organized.

and yes, I saw many docs and shell scripts still use this way to run the scalability test, which is a little bit misleading, many code / doc need to be cleaned up.

Could you pls let me the rule pls? I cannot find the string in the removed testcases either.

Thank you a lot!

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-scale" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-scale+unsub...@googlegroups.com.

Reply all

Reply to author

Forward