How to keep a running docker container after a build fails?

[pisymbol]

I have Googled this and still can't find as simple answer.

Is there a way to have Jenkins not clean up the container after a build? I am having a build failure and I don't quite understand why (a file doesn't exist when it should). But I have to replay the entire build which takes about 40 minutes before I get to the point of failure (ouch!). I wan to preserve the running container so I can bash into it and investigate, etc.

-aps

[Victor Martinez]

Hi,

How do you launch your builds? Pipeline/Freestyle?

Cheers

[pisymbol]

withDockerContainer(image: "centos:7", args: "--privileged") {

            stage ('Negotiate World Peace') {}

            stage ("Cure Cancer") {}

etc.

-aps

[pisymbol]

Anyone do this? I mean how does one debug build bugs in a docker container other than trial and error?

-aps 

[niristotle okram]

for debugging, 

why not put some more echo/ls commands to check if the file you are looking is there or not from the container. 

The documentation says:

Takes an image ID or symbolic name which must already have been pulled locally and starts a container based on that image. Runs all nested sh steps inside that container. The workspace is mounted read-write into the container.

Did you checked if the workspace have the file?

A really brut way, if you want to really shell into the container, might be to docker run the same image, in the workspace mnaually and try to trigger the steps manually. Else, use try and catch to not keep the container enaged even if the main execution fails. 

i haven't used these DSL, so these are some of my thoughts

HTH, 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a851cbd9-364c-4ce8-bd22-a972debce310%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards
nirish okram

[pisymbol .]

On Wed, Feb 7, 2018 at 2:43 PM, niristotle okram <nirish...@gmail.com> wrote:

for debugging, 

why not put some more echo/ls commands to check if the file you are looking is there or not from the container. 

Sure, but it takes 30+ minutes just to get to this spot.

 

The documentation says:

Takes an image ID or symbolic name which must already have been pulled locally and starts a container based on that image. Runs all nested sh steps inside that container. The workspace is mounted read-write into the container.

Did you checked if the workspace have the file?

A really brut way, if you want to really shell into the container, might be to docker run the same image, in the workspace mnaually and try to trigger the steps manually. Else, use try and catch to not keep the container enaged even if the main execution fails. 

But I can't really reproduce the Jenkins run-time though.

I'm really surprised you can't pause the build or have a flag that says don't kill the container at the end of the build. This should really be a feature for docker based builds.

-aps

[niristotle okram]

Pausing a build comes with an expense of a busy executor. 

Why not file a JIRA ticket to the plugin folks, might help someone someday. 

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPpy_fGNZ90Wa9nOx6n3h2m6N0bsvpCjHhaH0vqdyaHVcM3SBQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards
nirish okram

[Victor Martinez]

Just an idea, you have tried with the docker.inside syntax? (https://jenkins.io/doc/book/pipeline/docker/#running-sidecar-containers) It might give you the chance to debug it somehow.

Cheers

[pisymbol]

On Thursday, February 8, 2018 at 5:37:05 AM UTC-5, Victor Martinez wrote:

Just an idea, you have tried with the docker.inside syntax? (https://jenkins.io/doc/book/pipeline/docker/#running-sidecar-containers) It might give you the chance to debug it somehow.

What's the difference between that and my code snippet above (withDockerContainer)?

Because we use SVN, I am forced to use script initially prototype the build before I can really check in a Jenskin file.

-aps 

[Victor Martinez]

As you mentioned the lifecycle of the docker container with the step 'withDockerContainer' is well defined and doesn't allow to keep it alive. So i was assuming the docker.inside might help you to keep the container alive, but I was wrong: the docker.inside step removes the container while the 'docker.withRun' it just stops it, based on the below refcard:

- https://dzone.com/refcardz/continuous-delivery-with-jenkins-workflow?chapter=12

As I said, it was just an idea, unfortunately I haven't had to solve that use case yet...

Cheers

[Reinhold Füreder]

Hi aps,

 

Just in case the problem is still there and to avoid a misunderstanding:

 

Based on (a) Victor’s hint “tr[y] with the docker.inside syntax […] It might give you the chance to debug it somehow.” and (b) your own comment “I'm really surprised you can't pause the build”:

 

docker.image(…).inside(…) {

  sh ‘<some shell command that shows kind of debug output>’

 

  // And now “workaround” to pause the build to allow connecting to the still running Docker container:

  input(message: "Continue?")

}

 

Due to the “input” step the pipeline is then pausing and allows you to e.g.:

 

...@jenkins:...$ docker ps -a

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

51c00502c727        acme/...:5    "cat"               20 seconds ago      Up 20 seconds                           zealous_thompson

...@jenkins:...$ docker exec -it 51c00502c727 /bin/bash

jenkins@51c00502c727:<jenkins workspace>$ # Now you are in the running Docker container…

 

HTH Reinhold

 

[pisymbol .]

Thanks! That is a neat idea. A bit of a hack, but definitely does the trick. I actually wound up exec'ing into the container as the pipeline was running.

-aps