Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37769

Too long stop docker container(default docker stop timeout=10s)

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: docker-workflow-plugin
    • Labels:
      None
    • Environment:
      CloudBees Docker Pipeline 1.7
      Jenkins ver. 2.7
    • Similar Issues:

      Description

      Please use docker stop --time 0 when stop container for inside{}
      inside construction generate following cli call

      00:00:55.724 $ docker run -t -d -u 1000:1000 -w ... -v ...:rw -v ...@tmp:rw -e * tag cat
      00:00:56.357 + <shell command>
      00:00:56.369 $ docker stop 
      00:01:06.542 $ docker rm -f 
      

      but docker stop stop this cat command 10 seconds - too long

      docker stop --help
      
      Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
      
      Stop one or more running containers
      
      Options:
            --help       Print usage
        -t, --time int   Seconds to wait for stop before killing it (default 10)
      

      Experiment 1

      $ time docker run -t -d --name test ubuntu cat
      af6c228448ace32c66dba70efa2cc6189bc35dccdb2b950544e2a2e807a1d955
      
      real    0m0.289s
      user    0m0.016s
      sys     0m0.004s
      
      $ time docker stop test
      test
      
      real    0m10.206s
      user    0m0.008s
      sys     0m0.008s
      
      $ time docker rm -f test
      test
      
      real    0m0.021s
      user    0m0.008s
      sys     0m0.008s
      

      Experiment 2

      $ time docker run -t -d --name test ubuntu cat
      5d02979c9928e47455981f162707590122b5b124ec8748740273846201704a17
      
      real    0m0.324s
      user    0m0.008s
      sys     0m0.004s
      
      $ time docker stop --time 0 test
      test
      
      real    0m0.197s
      user    0m0.012s
      sys     0m0.000s
      
      $ time docker rm -f test
      test
      
      real    0m0.020s
      user    0m0.008s
      sys     0m0.004s
      

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            At first I thought it was this problem with systemd, but --stop-signal=RTMIN+3 does not work. Anyway the overridden --entrypoint means that systemd is not in play.

            Show
            jglick Jesse Glick added a comment - At first I thought it was this problem with systemd , but --stop-signal=RTMIN+3 does not work. Anyway the overridden --entrypoint means that systemd is not in play.
            Hide
            jglick Jesse Glick added a comment -

            Not following what is going on. If you run

            bash -c 'trap "echo term" EXIT; sleep 120'
            

            and then kill this bash process from another terminal window, you see

            term
            Terminated
            

            immediately. But if you run

            docker run --rm --name test ubuntu bash -c 'trap "echo term" EXIT; sleep 120'
            

            and from another window

            docker stop test
            

            you will see

            term
            

            and then ten seconds will elapse before the container exits. Somehow the TERM signal is getting sent, but not properly handled. (sleep is still running during this time.)

            Show
            jglick Jesse Glick added a comment - Not following what is going on. If you run bash -c 'trap "echo term" EXIT; sleep 120' and then kill this bash process from another terminal window, you see term Terminated immediately. But if you run docker run --rm --name test ubuntu bash -c 'trap "echo term" EXIT; sleep 120' and from another window docker stop test you will see term and then ten seconds will elapse before the container exits. Somehow the TERM signal is getting sent, but not properly handled. ( sleep is still running during this time.)
            Hide
            jglick Jesse Glick added a comment -

            Stranger still:

            docker run --rm --name test ubuntu sleep infinity
            

            Now docker exec test ps fauxwww confirms that sleep is PID 1, yet docker exec test kill -9 1 does not do anything. docker stop test will pause for ten seconds. I suspect that somehow signal delivery is just getting blocked to PID 1. On the other hand, with

            docker run --rm --name test --stop-signal=KILL ubuntu sleep infinity
            

            you will see that docker stop test works promptly. Perhaps in this case the stop command is just being clever and knows that it makes no sense to wait after sending SIGKILL, so behaves like --time=0?

            Really not sure what is going on. Passing --time 0 does correct the symptom, without addressing the underlying problem. But it is less than clear that a clean TERM is being delivered to any spawned processes even in the current state—it seems that TERM is delivered to the entrypoint ((cat) but does nothing.

            I am (dimly) aware of zombie reaping issues but that does not seem to be relevant in this case.

            Show
            jglick Jesse Glick added a comment - Stranger still: docker run --rm --name test ubuntu sleep infinity Now docker exec test ps fauxwww confirms that sleep is PID 1, yet docker exec test kill -9 1 does not do anything. docker stop test will pause for ten seconds. I suspect that somehow signal delivery is just getting blocked to PID 1. On the other hand, with docker run --rm --name test --stop-signal=KILL ubuntu sleep infinity you will see that docker stop test works promptly. Perhaps in this case the stop command is just being clever and knows that it makes no sense to wait after sending SIGKILL , so behaves like --time=0 ? Really not sure what is going on. Passing --time 0 does correct the symptom, without addressing the underlying problem. But it is less than clear that a clean TERM is being delivered to any spawned processes even in the current state—it seems that TERM is delivered to the entrypoint (( cat ) but does nothing. I am (dimly) aware of zombie reaping issues but that does not seem to be relevant in this case.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java
            src/test/java/org/jenkinsci/plugins/docker/workflow/client/DockerClientTest.java
            http://jenkins-ci.org/commit/docker-workflow-plugin/0fed7a702751f743eb3603092e1731f8979930a5
            Log:
            [FIXED JENKINS-37769] Failing to stop containers cleanly, so at least stopping them quickly.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java src/test/java/org/jenkinsci/plugins/docker/workflow/client/DockerClientTest.java http://jenkins-ci.org/commit/docker-workflow-plugin/0fed7a702751f743eb3603092e1731f8979930a5 Log: [FIXED JENKINS-37769] Failing to stop containers cleanly, so at least stopping them quickly.
            Hide
            docwhat Christian Höltje added a comment -

            This is due to the fact that signals are properly propagated without the -init flag:

             

            docker run --rm --name=test --init ubuntu sleep infinity
            

            See JENKINS-45888 for an issue for adding -init by default.  Changing the -time= flag is just papering over the real problem.

            Show
            docwhat Christian Höltje added a comment - This is due to the fact that signals are properly propagated without the -init flag:   docker run --rm --name=test --init ubuntu sleep infinity See JENKINS-45888 for an issue for adding - init by default.  Changing the -time= flag is just papering over the real problem.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                t_rex Oleksii Trekhov
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: