Succession of timeouts during Docker actions on ci.jenkins.io

[Chris Kilding]

Hi,

Timeouts related to the Docker Maven Plugin are causing some plugin builds to fail on ci.jenkins.io. The error message is usually along the lines of:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.fabric8:docker-maven-plugin:0.38.0:start (start-moto) on project <plugin>: I/O Error

Some examples:

- https://ci.jenkins.io/job/Plugins/job/aws-secrets-manager-secret-source-plugin/job/PR-33/1/console
- https://ci.jenkins.io/job/Plugins/job/aws-secrets-manager-secret-source-plugin/job/PR-36/1/console
- https://ci.jenkins.io/job/Plugins/job/aws-secrets-manager-secret-source-plugin/job/PR-34/1/console
- https://ci.jenkins.io/job/Plugins/job/aws-secrets-manager-credentials-provider-plugin/job/PR-166/1/console

I don't get this error running these builds on my laptop, and this error only seems to have started happening in the past month or so. Did anything about Docker change on the ci.jenkins.io build agents recently?

Regards,

Chris

[Mark Waite]

On Tuesday, January 4, 2022 at 5:57:04 AM UTC-7 Chris Kilding wrote:

Hi,

Timeouts related to the Docker Maven Plugin are causing some plugin builds to fail on ci.jenkins.io. The error message is usually along the lines of:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.fabric8:docker-maven-plugin:0.38.0:start (start-moto) on project <plugin>: I/O Error

Yes.  The Azure Kubernetes Service made a change in December that amplified some existing issues in the Jenkins public kubernetes cluster.  The infra team is actively working to resolve the issue.

It was discussed in today's infra team meeting.  A recording of the infra team meeting should be available on community.jenkins.io within the next 24-36 hours.

Mark Waite

[Tim Jacomb]

Hmm this isn’t what we were discussing though, this is using ci.Jenkins.io which uses a different cluster as far as I know

Cc infra

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5cfb6653-2b8c-4449-b2d5-ccae2e5a1a3cn%40googlegroups.com.

[Jesse Glick]

[Somehow did not get sent to the list. Reposting:]

On Tue, Jan 4, 2022 at 7:57 AM Chris Kilding <chris+...@chriskilding.com> wrote:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.fabric8:docker-maven-plugin:0.38.0:start (start-moto) on project <plugin>: I/O Error

This is rather opaque. The console log suggests some error during container start but does not give details.

No other plugin I am aware of uses `docker-maven-plugin` for ITs this way. The normal idiom is to use Testcontainers, which (unlike `AWSSecretsManagerRule`) will automatically start the test service as part of the test itself (no need for `maven-failsafe-plugin`) on any free server port. May be worth experimenting with other ways of running Docker from your tests, in case the problem is specific to `docker-maven-plugin`.

[Chris Kilding]

Mark - good to know that there has been an AKS change, I'll stay tuned for updates from the infra team.

Jesse - Thanks for the Testcontainers suggestion. The way the Docker containers are spun up in the test suite hasn't changed in a long time, so I'll give this a go and see if it helps.

Chris

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3f9fxROvx5vQhK5e%3DhDZouTAUDR%3DJBawjfPf4nz2xuBw%40mail.gmail.com.

[Damien Duportal]

Thanks for reporting and pinging us!

I confirm that this timeout error is not related to the Jenkins Kubernetes plugin issue that we face on others Jenkins instances (it’s a VM agent with a Docker Engine in this case).

(The "fabric8" component from the error stack was misleading @Mark ;) ).

We (infra-team) started to check this issue earlier today, and we are on the process to reproduce it.

@Chris, If you have any way to configure the maven-docker plugin to report the equivalent of the commands `docker ps`, `docker logs <moto container ID>` and `docker inspect <moto container ID>` that would be awesome.

Also, if you can try to pull the latest moto image (which is the tag used) and see if it continues working on your environment?

=> On our side we will continue diagnostics tomorrow, by executing the commands on an ephemeral VM based on the same template as ci.jenkins.io to see if we can reproduce the issue with the exact same tools (same kernel, same Docker, same Maven, same JDK). We’ll keep you updated.

Cheers

Damien

You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/CAH-3Bie4Gd2ueR7Hma9%2ByGrnCN-abUABcwTQbCgzjkkXqydf8Q%40mail.gmail.com.

[Chris Kilding]

I tried migrating from the docker-maven-plugin to testcontainers. The resulting code is indeed a little cleaner, and it's nice to not have to run an out-of-band step before running the test suite. Unfortunately the build still fails on ci.jenkins.io, so the underlying Docker issue is probably still there.

Log is here:

https://ci.jenkins.io/job/Plugins/job/aws-secrets-manager-secret-source-plugin/job/PR-37/2/console

Chris

[Jesse Glick]

On Wed, Jan 5, 2022 at 7:41 AM Chris Kilding <chris+...@chriskilding.com> wrote:

Log is here:

More useful is the test stderr https://ci.jenkins.io/job/Plugins/job/aws-secrets-manager-secret-source-plugin/job/PR-37/2/testReport/io.jenkins.plugins.casc.secretsmanager/SecretSourceIT/linux_11_2_263_4___Build__linux_11_2_263_4____shouldRevealSecret/ which says

  File "/usr/local/lib/python3.7/site-packages/moto/ec2/models.py", line 1831, in get_zone_by_name
    for zone in self.zones[self.region_name]:
KeyError: 'ap-southeast-3'

Sounds like a problem in your container perhaps.

[Chris Kilding]

The Moto KeyError seemed to be a one-off flake; I haven't seen it happen since then. Testcontainers seems to be working reliably enough over subsequent builds, so counting this as fixed :)

kubernetes-pipeline-plugin and gitlab-plugin appear to be the 2 other plugins under jenkinsci that use docker-maven-plugin. Is it worth advising them to switch to Testcontainers at some point as well?

Chris

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr35bheD0O3QpUDgR3OTN8ydUY15nR0wRBJ8ivFZ9pT2RA%40mail.gmail.com.

[Jesse Glick]

On Mon, Jan 10, 2022 at 6:19 AM Chris Kilding <chris+...@chriskilding.com> wrote:

kubernetes-pipeline-plugin and gitlab-plugin appear to be the 2 other plugins under jenkinsci that use docker-maven-plugin. Is it worth advising them to switch to Testcontainers at some point as well?

Would not hurt to file issues for them suggesting it, especially given your experience doing such a switch.

Note that a bunch of plugins still use the `jenkinsci/docker-fixtures` library, which is deprecated in favor of Testcontainers but has broadly similar functionality.