Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58236

Provide mechanism to define dynamic stages in declarative pipelines

    Details

    • Similar Issues:

      Description

      Hello, it would be great if we had a mechanism to define dynamic stages within declarative pipelines. The use case is that I want to create a pipeline function for micro service repositories. Each repository contains a number of container images that needs to be built in parallel. The only way I found is using the these pseudo stages wrapped by the `parallel` step:

      pipeline {
        agent any
        stages {
          stage('build') {
            steps {
              runParallel items: ("a".."f").collect { "Stage ${it}" }
            }
          }
        }
      }
      
      def runParallel(args) {
        parallel args.items.collectEntries { name -> [ "${name}": {
          stage("${name}") {
            echo name
          }
        }]}
      }
      

      This is bad because the sub-stages can neither have an agent nor contain other sub-stages. And it requires a separate function (I couldn't get it working without).
      On the other hand, this shouldn't make the declarative pipeline too scripty. A minimal scriptiness approach would look like this:

      pipeline {
        agent none
        stages {
          stage('build') {
            stages {
              ("A".."F").collect {
                stage("Stage ${it}") {
                  agent any
                  steps {
                    echo "Hello from stage ${it}!"
                  }
                }
              }
            }
          }
        }
      }
      

        Attachments

          Activity

          hendrikhalkow Hendrik Halkow created issue -
          hendrikhalkow Hendrik Halkow made changes -
          Field Original Value New Value
          Description Hello, it would be great if we had a mechanism to define dynamic stages within declarative pipelines. The use case is that I want to create a pipeline function for micro service repositories. Each repository contains a number of container images that needs to be built in parallel. The only way I found is using the these pseudo stages wrapped by the `parallel` step:

          {code}
          pipeline {
            agent any
            stages {
              stage('build') {
                steps {
                  runParallel items: ("a".."f").collect { "Stage ${it}" }
                }
              }
            }
          }

          def runParallel(args) {
            parallel args.items.collectEntries { name -> [ "${name}": {
              stage("${name}") {
                echo name
              }
            }]}
          }
          {code}

          This is bad because the sub-stages can neither have an agent nor contain other sub-stages. And it requires a
          On the other hand, this shouldn't make the declarative pipeline too scripty. A minimal scriptiness approach would look like this:

          {code}
          pipeline {
            agent none
            stages {
              stage('build') {
                stages {
                  ("A".."F").collect {
                    stage("Stage ${it}") {
                      agent any
                      steps {
                        echo "Hello from stage ${it}!"
                      }
                    }
                  }
                }
              }
            }
          }
          {code}
          Hello, it would be great if we had a mechanism to define dynamic stages within declarative pipelines. The use case is that I want to create a pipeline function for micro service repositories. Each repository contains a number of container images that needs to be built in parallel. The only way I found is using the these pseudo stages wrapped by the `parallel` step:

          {code}
          pipeline {
            agent any
            stages {
              stage('build') {
                steps {
                  runParallel items: ("a".."f").collect { "Stage ${it}" }
                }
              }
            }
          }

          def runParallel(args) {
            parallel args.items.collectEntries { name -> [ "${name}": {
              stage("${name}") {
                echo name
              }
            }]}
          }
          {code}

          This is bad because the sub-stages can neither have an agent nor contain other sub-stages. And it requires a separate function (I couldn't get it working without).
          On the other hand, this shouldn't make the declarative pipeline too scripty. A minimal scriptiness approach would look like this:

          {code}
          pipeline {
            agent none
            stages {
              stage('build') {
                stages {
                  ("A".."F").collect {
                    stage("Stage ${it}") {
                      agent any
                      steps {
                        echo "Hello from stage ${it}!"
                      }
                    }
                  }
                }
              }
            }
          }
          {code}
          Hide
          luckyhorang Hokwang Lee added a comment -

          The number of Dockerfile is increasing more and more, so this feature is really needed and important to all users.

          Show
          luckyhorang Hokwang Lee added a comment - The number of Dockerfile is increasing more and more, so this feature is really needed and important to all users.
          Hide
          zionyx KY Lee added a comment -

          There is a way to achieve this. See this response: https://stackoverflow.com/a/51308400/795137

          The way he described is to create these dynamic stages within script closure. Wrap it around in a parallel closure and it should get what you want.

          Show
          zionyx KY Lee added a comment - There is a way to achieve this. See this response: https://stackoverflow.com/a/51308400/795137 The way he described is to create these dynamic stages within script closure. Wrap it around in a parallel closure and it should get what you want.
          Hide
          hendrikhalkow Hendrik Halkow added a comment -

          This is pretty much the same as my first example that I don't want because of the explained reasons.

          Show
          hendrikhalkow Hendrik Halkow added a comment - This is pretty much the same as my first example that I don't want because of the explained reasons.
          abayer Andrew Bayer made changes -
          Assignee Andrew Bayer [ abayer ]

            People

            • Assignee:
              Unassigned
              Reporter:
              hendrikhalkow Hendrik Halkow
            • Votes:
              6 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: