[kubevirt-dev] Conformance Tests Failures Due To Docker Image Pull Limits
56 views
Skip to first unread message
Or Mergi
Jun 22, 2021, 3:03:18 AM6/22/21
to kubevi...@googlegroups.com
Hi All,
I noticed that check-up-k8s-1.20 lane fails from time to time.
Looking into this I saw that it fails during conformance tests (kubernetes-e2e plugin) due to docker image pulling limits.
Here is an example of the error message we got on one of the failing jobs:
...
(x3 over 9m7s) kubelet Failed to pull image \"docker.io/library/httpd:2.4.39-alpine\": rpc error: code = Unknown desc = Error reading manifest 2.4.39-alpine in docker.io/library/httpd: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 6m45s (x4 over 9m56s) kubelet Pulling image \"docker.io/library/httpd:2.4.39-alpine\"
Warning Failed 5m59s (x4 over 9m7s) kubelet Error: ErrImagePull
Warning Failed 5m59s kubelet Failed to pull image \"docker.io/library/httpd:2.4.39-alpine\": rpc error: code = Unknown desc = Error loading manifest for target platform: Error reading manifest sha256:dfb792aa3fed0694d8361ec066c592381e9bdff508292eceedd1ce28fb020f71 in docker.io/library/httpd: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
...
Image that fails to pull are:
docker.io/library/httpd, failing jobs [1] [2]docker.io/library/nginx:1.14-alpine, failing job [3]
This issue is probably not exclusive for check-up-k8s-1.20 lane only
and we could encounter this on other lanes that runs conformance tests.
I would suggest to mirror these images to kubevirtci registry and configure sonobuoy to use them, but I guess any other image that is being used on conformance tests could have hit the pulling limit.
(pick one of the falling tests and press on "open stdout")
Regards,
-- Or Mergi
Associate Software Engineer
Openshift Virtualization - Network
 |
Daniel Hiller
Jun 22, 2021, 3:43:41 AM6/22/21
to Or Mergi, kubevirt-dev
Hey Or,
thanks for spotting this! IIUC the sonobuoy tests are executed within our kubevirtci cluster, so adding a pre pull of these images during cluster provision might help, no? Does anyone have insight what images are being pulled from the conformance tests so that we can put a list together?
Regards,--Or Mergi
Associate Software Engineer
Openshift Virtualization - Network
--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAJ38DpD4Rdm6P6LaENPYE%2BXWQyaqKMPU-j7t2Mji9acEBgR%3D4Q%40mail.gmail.com.
--
Kind regards,
Daniel Hiller
He / Him / His
Senior Software Engineer, OpenShift Virtualization
|
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
Petr Horacek
Jun 22, 2021, 3:48:10 AM6/22/21
to Daniel Hiller, Or Mergi, kubevirt-dev
Hey Or,On Tue, Jun 22, 2021 at 9:03 AM Or Mergi <ome...@redhat.com> wrote:Hi All,While working on https://github.com/kubevirt/kubevirtci/pull/621
I noticed that check-up-k8s-1.20 lane fails from time to time.Looking into this I saw that it fails during conformance tests (kubernetes-e2e plugin) due to docker image pulling limits.Here is an example of the error message we got on one of the failing jobs:...
(x3 over 9m7s) kubelet Failed to pull image \"docker.io/library/httpd:2.4.39-alpine\": rpc error: code = Unknown desc = Error reading manifest 2.4.39-alpine in docker.io/library/httpd: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 6m45s (x4 over 9m56s) kubelet Pulling image \"docker.io/library/httpd:2.4.39-alpine\"
Warning Failed 5m59s (x4 over 9m7s) kubelet Error: ErrImagePull
Warning Failed 5m59s kubelet Failed to pull image \"docker.io/library/httpd:2.4.39-alpine\": rpc error: code = Unknown desc = Error loading manifest for target platform: Error reading manifest sha256:dfb792aa3fed0694d8361ec066c592381e9bdff508292eceedd1ce28fb020f71 in docker.io/library/httpd: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
...Image that fails to pull are:docker.io/library/httpd, failing jobs [1] [2]docker.io/library/nginx:1.14-alpine, failing job [3]This issue is probably not exclusive for check-up-k8s-1.20 lane onlyand we could encounter this on other lanes that runs conformance tests.I would suggest to mirror these images to kubevirtci registry and configure sonobuoy to use them, but I guess any other image that is being used on conformance tests could have hit the pulling limit.(pick one of the falling tests and press on "open stdout")thanks for spotting this! IIUC the sonobuoy tests are executed within our kubevirtci cluster, so adding a pre pull of these images during cluster provision might help, no? Does anyone have insight what images are being pulled from the conformance tests so that we can put a list together?
In this case, it is important to move them out of Docker Hub. Conformance tests are meant to be run against arbitrary user clusters. I've already saw one occurence of it failing in somebody's cluster due to pull limit (although it was using older version on the suite and failed pulling our VM images that used to be on Docker).
----Regards,--Or Mergi
Associate Software Engineer
Openshift Virtualization - Network
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAJ38DpD4Rdm6P6LaENPYE%2BXWQyaqKMPU-j7t2Mji9acEBgR%3D4Q%40mail.gmail.com.
--Kind regards,
Daniel Hiller
He / Him / His
Senior Software Engineer, OpenShift Virtualization
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAK%2BeyL48qtB-yr%3DW8T2N1ArjwThNxVE%2BGi1-ChnbAzb0F8jqnA%40mail.gmail.com.
Or Mergi
Jun 22, 2021, 5:04:20 AM6/22/21
to Daniel Hiller, Petr Horacek, kubevirt-dev
Hey, I found this workaround https://github.com/vmware-tanzu/sonobuoy/pull/1194
I have created a PR to use it on kubevirtci https://github.com/kubevirt/kubevirtci/pull/624
I will appreciate your reviews :)
I will appreciate your reviews :)
Or Mergi
Daniel Hiller
Jun 22, 2021, 5:05:44 AM6/22/21
to Petr Horacek, Or Mergi, kubevirt-dev
Hey,
But in order for our registry to be visible at the kubevirtci cluster we would have to inject our proxy registry configuration, wouldn't we? If we didn't do that first, the mirroring would not help? Or do you mean by what you said modifying/forking the conformance tests and generating own manifests?
Petr Horacek
Jun 22, 2021, 5:11:06 AM6/22/21
to Or Mergi, Daniel Hiller, kubevirt-dev
Hey, I found this workaround https://github.com/vmware-tanzu/sonobuoy/pull/1194
AFAIK this only helps to get Sonobuoy image and Kubernetes e2e test images from a mirror. It does not help us with any images that are specific to our test suite.
Petr Horacek
Jun 22, 2021, 5:14:54 AM6/22/21
to Or Mergi, Or Shoval, Daniel Hiller, kubevirt-dev
Oh, my bad @Or Shoval @Daniel Hiller. I thought we were talking about KubeVirt conformance, while this is about Kubernetes conformance. Disregard my comments. The Sonobuoy workaround should help in this case.
út 22. 6. 2021 v 11:10 odesílatel Petr Horacek <phor...@redhat.com> napsal:
Reply all
Reply to author
Forward
0 new messages