Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49446

Regression with 1.15 and WithContainerStep

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: docker-workflow-plugin
    • Labels:
      None
    • Environment:
      Docker Pipeline 1.15 + Jenkins core 2.89.3
    • Similar Issues:

      Description

      I noticed that after upgrade to 1.15, steps such as {{docker.image().inside }} have begun to fail with:

      
      java.io.IOException: Failed to run top '7924b7207cfe14b8abba497c6051504cf0de0c02b40190b3688b78d680f3ee81'. Error: Error response from daemon: Container 7924b7207cfe14b8abba497c6051504cf0de0c02b40190b3688b78d680f3ee81 is not running
      	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.listProcess(DockerClient.java:140)
      	at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:185)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
      	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:135)
      	at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:66)
      	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:123)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineScript.runImage(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy:57)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:67)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:53)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/jenkins_home/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:63)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

      I have witnessed this with several Pipelines using docker.image.inside in Scripted Pipeline on ci.jenkins.io

      The only remediation I could find was to downgrade to 1.14

        Attachments

          Issue Links

            Activity

            Hide
            myoung34 marc young added a comment - - edited

            I'd also like to point out (aside from breaking everything at 3 of my contracts): you've based everything on "requirements" that are actually suggestions.

            The error I'm now facing in dozens of builds:

            04:11:59 ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).
            

            Youve stated as required by . However those docs clearly say:

            All official images should provide a consistent interface. 
            

            Should != require. Your use-case and assumptions are not a standard to hold an entire community to

            Please remember: I'm using other peoples official containers to run in jenkins. I cannot (and will not) fork every single one of these to meet your guidelines.

            Show
            myoung34 marc young added a comment - - edited I'd also like to point out (aside from breaking everything at 3 of my contracts): you've based everything on "requirements" that are actually suggestions. The error I'm now facing in dozens of builds: 04:11:59 ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https: //github.com/docker-library/official-images#consistency for entrypoint consistency requirements). Youve stated as required by . However those docs clearly say: All official images should provide a consistent interface . Should != require. Your use-case and assumptions are not a standard to hold an entire community to Please remember: I'm using other peoples official containers to run in jenkins. I cannot (and will not) fork every single one of these to meet your guidelines.
            Hide
            ndeloof Nicolas De Loof added a comment -

            @myoung34 you're perfectly right, this is not a requirement but a convention on official image (it's required to get official image approved), and many custom image don't follow this.

            Our issue here is we are abusing the container lifecycle : an arbitrary docker image is not designed to run arbitrary command like we use them for.

            One option is to override entrypoint to run a custom command. But then we disable the initial entrypoint designed by image author, which in many cases is required for the image to make any sense. Typically, Selenium images do use it to run a X11 server. This option has been adopted in the past, introducing https://issues.jenkins-ci.org/browse/JENKINS-41316 regression.

            The other is to assume newcomers mostly will try docker pipeline using official images. So we offer a solution which works out-of-the box, and can't report issue with target image if detected not to match our requirements. Those who are already used with docker-pipeline then get a documented direction on how to fix their image or update their pipeline.

            I welcome any suggestion for a 3rd option. As one can't "docker exec" in a stopped container, we have to find some way for the container to run a "pause" command, and this is definitively not part of docker spec to support this.

            Show
            ndeloof Nicolas De Loof added a comment - @myoung34 you're perfectly right, this is not a requirement but a convention on official image (it's required to get official image approved), and many custom image don't follow this. Our issue here is we are abusing the container lifecycle : an arbitrary docker image is  not designed to run arbitrary command like we use them for. One option is to override entrypoint to run a custom command. But then we disable the initial entrypoint designed by image author, which in many cases is required for the image to make any sense. Typically, Selenium images do use it to run a X11 server. This option has been adopted in the past, introducing https://issues.jenkins-ci.org/browse/JENKINS-41316  regression. The other is to assume newcomers mostly will try docker pipeline using official images. So we offer a solution which works out-of-the box, and can't report issue with target image if detected not to match our requirements. Those who are already used with docker-pipeline then get a documented direction on how to fix their image or update their pipeline. I welcome any suggestion for a 3rd option. As one can't " docker exec " in a stopped container, we  have to find some way for the container to run a "pause" command, and this is definitively not part of docker spec to support this.
            Hide
            ndeloof Nicolas De Loof added a comment -

            Andrew Bayer 

            > "abusing container lifecycle" is, frankly, irrelevant here. What matters is that we keep from breaking users any more than absolutely necessary

            Sorry to say it has bee absolutely necessary. We need entrypoint support for many major use-cases, disabling them was a breaking change. It just has been restored, and introduced diagnostic code to assist end-user in fixing his pipeline/image.

            And we do abuse container lifecycle forcing the container to run a command it has not been designed for.

             

            Show
            ndeloof Nicolas De Loof added a comment - Andrew Bayer   > "abusing container lifecycle" is, frankly, irrelevant here. What matters is that we keep from breaking users any more than absolutely necessary Sorry to say it has bee absolutely necessary. We need entrypoint support for many major use-cases, disabling them was a breaking change. It just has been restored, and introduced diagnostic code to assist end-user in fixing his pipeline/image. And we  do abuse container lifecycle forcing the container to run a command it has not been designed for.  
            Hide
            mark_russell Mark Russell added a comment -

            This issue is stopping up from upgrading this plugins.  Is anyone looking at it?  Is there any chance for a solution on this?

            Show
            mark_russell Mark Russell added a comment - This issue is stopping up from upgrading this plugins.  Is anyone looking at it?  Is there any chance for a solution on this?
            Hide
            ndeloof Nicolas De Loof added a comment -

            docker-workflow plugin comes with constraints, which unfortunately never have been clearly documented.

            Those are pretty comparable to https://github.com/knative/docs/blob/master/build/builder-contract.md

            But there's no way we support arbitrary docker images and entrypoint script. Adapt your docker images so they fit into our model

            Show
            ndeloof Nicolas De Loof added a comment - docker-workflow plugin comes with constraints, which unfortunately never have been clearly documented. Those are pretty comparable to https://github.com/knative/docs/blob/master/build/builder-contract.md But there's no way we support arbitrary docker images and entrypoint script. Adapt your docker images so they fit into our model

              People

              • Assignee:
                ndeloof Nicolas De Loof
                Reporter:
                rtyler R. Tyler Croy
              • Votes:
                13 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: