Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42322

Docker rm/stop/... commands killed by the timeout, failing builds

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi,

      I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

      In 1.10 I have the following error on the container launched with .inside { } method :

      $ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
      $ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
      ERROR: Timeout after 10 seconds
      

      And the job, fail with

      java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
      	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
      	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
      	at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
      	at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
      	at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
      	at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
      	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
      	at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      

      This seems to be related with the
      https://github.com/jenkinsci/docker-workflow-plugin/pull/65
      I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

      Is there a way to disable this option ?

      Thanks,

        Attachments

          Issue Links

            Activity

            Hide
            zoobab Benjamin Henrion added a comment -

            Sorry, I made a mistake, the right command is:

            $ while true; do ./stress --cpu 4 --io 25 --vm 4 --vm-bytes 7G; done

            You can then monitor the response time of the docker daemon with:

            $ while true; do time docker ps 2&>1 ; sleep 1; done

            real 0m0.128s
            user 0m0.006s
            sys 0m0.006s

            real 0m0.068s
            user 0m0.007s
            sys 0m0.005s

            real 0m0.039s
            user 0m0.009s
            sys 0m0.003s

            real 0m1.794s
            user 0m0.009s
            sys 0m0.025s

            real 0m1.326s
            user 0m0.006s
            sys 0m0.063s
            ^C
            real 0m1.325s
            user 0m0.003s
            sys 0m0.044s

            Show
            zoobab Benjamin Henrion added a comment - Sorry, I made a mistake, the right command is: $ while true; do ./stress --cpu 4 --io 25 --vm 4 --vm-bytes 7G; done You can then monitor the response time of the docker daemon with: $ while true; do time docker ps 2&>1 ; sleep 1; done real 0m0.128s user 0m0.006s sys 0m0.006s real 0m0.068s user 0m0.007s sys 0m0.005s real 0m0.039s user 0m0.009s sys 0m0.003s real 0m1.794s user 0m0.009s sys 0m0.025s real 0m1.326s user 0m0.006s sys 0m0.063s ^C real 0m1.325s user 0m0.003s sys 0m0.044s
            Hide
            zoobab Benjamin Henrion added a comment -

            This one is better is measure the response time:

            $ while true; do { time docker ps >/dev/null; } 2>&1 | grep real; sleep 2; done

            real 0m0.097s
            real 0m0.121s
            real 0m0.112s
            real 0m0.116s
            real 0m0.276s
            real 0m7.591s
            real 0m7.747s

            Show
            zoobab Benjamin Henrion added a comment - This one is better is measure the response time: $ while true; do { time docker ps >/dev/null; } 2>&1 | grep real; sleep 2; done real 0m0.097s real 0m0.121s real 0m0.112s real 0m0.116s real 0m0.276s real 0m7.591s real 0m7.747s
            Hide
            zoobab Benjamin Henrion added a comment -

            This is my jenkins in a docker-compose that works, and the environment JAVA_OPTS has to be modified to load the params:

                jenkins:
                    image: myregistry.com/jenkins:2.83:latest
                    command: --prefix="/jenkins"
                    user: root
                    ports:
                        - "8081:8080"
                        - "8050:50000"
                    volumes:
                        - jenkins-dv:/var/jenkins_home
                    environment:
                        - JAVA_OPTS=-Dorg.jenkinsci.plugins.docker.workflow.client.DockerClient.CLIENT_TIMEOUT=240

            Show
            zoobab Benjamin Henrion added a comment - This is my jenkins in a docker-compose that works, and the environment JAVA_OPTS has to be modified to load the params:     jenkins:         image: myregistry.com/jenkins:2.83:latest         command: --prefix="/jenkins"         user: root         ports:             - "8081:8080"             - "8050:50000"         volumes:             - jenkins-dv:/var/jenkins_home         environment:             - JAVA_OPTS=-Dorg.jenkinsci.plugins.docker.workflow.client.DockerClient.CLIENT_TIMEOUT=240
            Hide
            gnustavo Gustavo Chaves added a comment -

            I'm commenting to report that we begun facing similar errors last week, after we changed several pipeline scripts, which increased the frequency with which we launch Docker containers.

            We're using Jenkins 2.107.2 and docker-workflow-plugin 1.17.

            I increased the plugin timeout to 250s and decreased the number of executors in our slaves to no avail.

            In despair, I patched the plugin to make it ignore docker-rm errors. Using the patched plugin we were able to finish all of our weekend builds successfully.

            I know this is not a decent solution. I feared that our builds could be delayed waiting for the stuck containers, but I didn't notice this or any other side effect. So, for now, we'll use this patched plugin.

            I don't understand yet what makes the docker-rm command delay. Do you have some hints to share?

            Show
            gnustavo Gustavo Chaves added a comment - I'm commenting to report that we begun facing similar errors last week, after we changed several pipeline scripts, which increased the frequency with which we launch Docker containers. We're using Jenkins 2.107.2 and docker-workflow-plugin 1.17. I increased the plugin timeout to 250s and decreased the number of executors in our slaves to no avail. In despair, I patched the plugin to make it ignore docker-rm errors. Using the patched plugin we were able to finish all of our weekend builds successfully. I know this is not a decent solution. I feared that our builds could be delayed waiting for the stuck containers, but I didn't notice this or any other side effect. So, for now, we'll use this patched plugin. I don't understand yet what makes the docker-rm command delay. Do you have some hints to share?
            Hide
            svanoort Sam Van Oort added a comment -

            Gustavo Chaves Which version of Docker itself are you using, and do you see the timeout error, or just the error "Failed to rm container" ?

            Show
            svanoort Sam Van Oort added a comment - Gustavo Chaves Which version of Docker itself are you using, and do you see the timeout error, or just the error "Failed to rm container" ?

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                kevanescence Kevin REMY
              • Votes:
                34 Vote for this issue
                Watchers:
                48 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: