Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49558

Run stages in separate Docker containers when top level agent declared

    Details

    • Similar Issues:

      Description

      When specifying either docker or dockerfile as the top level agent this will run the entire pipeline and all of it's stages in the same container instance created at the beginning.

      pipeline {
          agent {
              docker {
                  image 'python3.6'
              }
          }
          /* These stages all run in the same python3.6 container
           * created at the beginning of the pipeline.
           */
          stages {
              stage('Stage 1') {
                  steps {
                      sh 'echo foo'
                  }
              }
              stage('Stage 2') {
                  steps {
                      sh 'echo bar'
                  }
              }
          }
      }
      

       
      It would be nice if there were some setting to run each stage in it's own container instance.

      pipeline {
          agent {
              docker {
                  image 'python3.6'
                  runPerStage true
              }
          }
          /* These stages all create their own python3.6 container
           * created at the beginning of each stage.
           */
          stages {
              stage('Stage 1') {
                  steps {
                      sh 'echo foo'
                  }
              }
              stage('Stage 2') {
                  steps {
                      sh 'echo bar'
                  }
              }
          }
      }
      

      Additionally, it would be nice to declare on a per stage basis if a new container instance is needed while other stages can use the instance created at the top level.

      pipeline {
          agent {
              docker {
                  image 'python3.6'
              }
          }
          stages {
              /* This stage creates it's own python3.6 container instance. */
              stage('Stage 1') {
                  agent {
                      docker {
                          createNew true
                      }
                  }
                  steps {
                      sh 'echo foo'
                  }
              }
              /* This stage uses the python3.6 container instance created
               * at the top level.
               */
              stage('Stage 2') {
                  steps {
                      sh 'echo bar'
                  }
              }
          }
      }
      

      Finally, when declaring a top level docker or dockerfile agent and later specifying a stage level docker or dockerfile agent Jenkins will try and do a docker-in-docker type thing breaking the pipeline and doesn't seem to handle this at all.

      It would be nice to have the top level agent as the default container used for stages but if a stage declares it's own agent than that container is used instead. This seem to match more closely to how agent works when not using docker or dockerfile.

      pipeline {
          agent {
              docker {
                  image 'python3.6'
              }
          }
          stages {
              /* This stage creates it's own python2.7 container instance. */
              stage('Stage 1') {
                  agent {
                      docker {
                          image 'python2.7'
                      }
                  }
                  steps {
                      sh 'echo foo'
                  }
              }
              /* This stage uses the python3.6 container instance created
               * at the top level.
               */
              stage('Stage 2') {
                  steps {
                      sh 'echo bar'
                  }
              }
          }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            awiddersheim Andrew Widdersheim added a comment -

            Andrew Bayer sure, makes sense. Looks good to me! Though not sure it helps with comment-328884 at all. That said, you mentioned that issue is related to something else.

            Show
            awiddersheim Andrew Widdersheim added a comment - Andrew Bayer sure, makes sense. Looks good to me! Though not sure it helps with comment-328884 at all. That said, you mentioned that issue is related to something else.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/AbstractDockerAgent.java
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/ContainerPerStage.java
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/ContainerPerStage/help.html
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
            pipeline-model-definition/src/test/resources/agentDockerContainerPerStage.groovy
            pipeline-model-definition/src/test/resources/agentDockerWithoutContainerPerStage.groovy
            http://jenkins-ci.org/commit/pipeline-model-definition-plugin/773530a1eade1bb1b4b14845cc9b1f0800d7ad29
            Log:
            [FIXED JENKINS-49558] Add containerPerStage option

            If given with a top-level Docker or Dockerfile agent, the top-level
            will magically switch to just a label agent, while each stage without
            an explicit agent specified will end up using the root agent
            definition with reuseNode enabled. The result will be that each stage
            gets a new container. Tada.

            This could theoretically cause problems in resuming builds after
            upgrade, but not likely, since we actually instantiate the
            DeclarativeAgent instance just in time, not at the beginning of the
            build. So the new field AbstractDockerAgent#containerPerStageRoot
            should get populated correctly.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/AbstractDockerAgent.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/ContainerPerStage.java pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/ContainerPerStage/help.html pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java pipeline-model-definition/src/test/resources/agentDockerContainerPerStage.groovy pipeline-model-definition/src/test/resources/agentDockerWithoutContainerPerStage.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/773530a1eade1bb1b4b14845cc9b1f0800d7ad29 Log: [FIXED JENKINS-49558] Add containerPerStage option If given with a top-level Docker or Dockerfile agent, the top-level will magically switch to just a label agent, while each stage without an explicit agent specified will end up using the root agent definition with reuseNode enabled. The result will be that each stage gets a new container. Tada. This could theoretically cause problems in resuming builds after upgrade, but not likely, since we actually instantiate the DeclarativeAgent instance just in time, not at the beginning of the build. So the new field AbstractDockerAgent#containerPerStageRoot should get populated correctly.
            Hide
            abayer Andrew Bayer added a comment -

            Merged - this'll be in the upcoming Declarative 1.2.8.

            Show
            abayer Andrew Bayer added a comment - Merged - this'll be in the upcoming Declarative 1.2.8.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            content/doc/book/pipeline/syntax.adoc
            http://jenkins-ci.org/commit/jenkins.io/d5c680bc49fd66e799230a89681ccf4078b3c181
            Log:
            JENKINS-49558 Add doc for newContainerPerStage Declarative option

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/d5c680bc49fd66e799230a89681ccf4078b3c181 Log: JENKINS-49558 Add doc for newContainerPerStage Declarative option
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Liam Newman
            Path:
            content/doc/book/pipeline/syntax.adoc
            http://jenkins-ci.org/commit/jenkins.io/75ca4f256a23448a6e94edbe118ace58d76ab07f
            Log:
            Merge pull request #1443 from abayer/jenkins-49558

            JENKINS-49558 Add doc for newContainerPerStage Declarative option

            Compare: https://github.com/jenkins-infra/jenkins.io/compare/4e5266831a46...75ca4f256a23

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Liam Newman Path: content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/75ca4f256a23448a6e94edbe118ace58d76ab07f Log: Merge pull request #1443 from abayer/jenkins-49558 JENKINS-49558 Add doc for newContainerPerStage Declarative option Compare: https://github.com/jenkins-infra/jenkins.io/compare/4e5266831a46...75ca4f256a23

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                awiddersheim Andrew Widdersheim
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: