Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41334

Support parallel execution of stages in Declarative

    Details

    • Epic Link:
    • Sprint:
      Declarative - 1.2
    • Similar Issues:

      Description

      Improvement on roadmap

      This improvement is on the Blue Ocean project roadmap. Check the roadmap page for updates.

      Proposal
      Note - this is a post-1.0 feature.

      Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, parallel doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

      I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested stages sections, like this:

      pipeline {
        agent any
        
        stages {
          stage('first') {
            steps {
              echo 'first, non-parallel stage'
            }
          }
      
          stage('top-parallel') {
            stages {
              stage('first-parallel') {
                steps {
                  echo 'First of the parallel stages without further nesting'
                  sleep 60
                }
              }
              stage('second-parallel') {
                stages {
                  stage('first-nested-parallel') {
                    steps {
                       echo 'the first of the nested parallel stages'
                       sleep 30
                    }
                 }
                 stage('second-nested-parallel') {
                    steps {
                       echo 'the second of the nested parallel stages'
                       sleep 30
                    }
                 }
              }
           }
        }
      }
      

      So in this scenario, stage('first') runs first. When it completes, stage('top-parallel') starts and immediately goes into its nested stages in parallel. stage('first-parallel') starts and goes for 60 seconds, while stage('second-parallel') starts at the same time and descends into its nested stages in parallel as well.

      I don't know yet where I'd allow agent and friends, but within a stage, you would need to have one and only one of steps or stages - i.e., a stage either could have steps it executes or it could be a container for parallel stages.

      Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.

        Attachments

          Issue Links

            Activity

            abayer Andrew Bayer created issue -
            abayer Andrew Bayer made changes -
            Field Original Value New Value
            Status Open [ 1 ] In Progress [ 3 ]
            roidelapluie Julien Pivotto made changes -
            Comment [ Would also love to be able to code those parallel:

            {code}
            pipeline {
              stages {
                def parallelstages = [:]

                paralellstages['foo'] = {
                  stage {
                    agent {docker 'hello' }
                  }
                }
                parallel parallelstages
              }
            }
            {code} ]
            abayer Andrew Bayer made changes -
            Link This issue is duplicated by JENKINS-41198 [ JENKINS-41198 ]
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "RFC doc for review (Web Link)" [ 15496 ]
            michaelneale Michael Neale made changes -
            Link This issue is blocked by JENKINS-42364 [ JENKINS-42364 ]
            michaelneale Michael Neale made changes -
            Link This issue is blocked by JENKINS-42362 [ JENKINS-42362 ]
            jamesdumay James Dumay made changes -
            Description Note - this is a post-1.0 feature.

            Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, {{parallel}} doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

            I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this:

            {code}
            pipeline {
              agent any
              
              stages {
                stage('first') {
                  steps {
                    echo 'first, non-parallel stage'
                  }
                }

                stage('top-parallel') {
                  stages {
                    stage('first-parallel') {
                      steps {
                        echo 'First of the parallel stages without further nesting'
                        sleep 60
                      }
                    }
                    stage('second-parallel') {
                      stages {
                        stage('first-nested-parallel') {
                          steps {
                             echo 'the first of the nested parallel stages'
                             sleep 30
                          }
                       }
                       stage('second-nested-parallel') {
                          steps {
                             echo 'the second of the nested parallel stages'
                             sleep 30
                          }
                       }
                    }
                 }
              }
            }
            {code}

            So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well.

            I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages.

            Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.
            *Scope*
            *

            *Proposal*
            Note - this is a post-1.0 feature.

            Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, {{parallel}} doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

            I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this:

            {code}
            pipeline {
              agent any
              
              stages {
                stage('first') {
                  steps {
                    echo 'first, non-parallel stage'
                  }
                }

                stage('top-parallel') {
                  stages {
                    stage('first-parallel') {
                      steps {
                        echo 'First of the parallel stages without further nesting'
                        sleep 60
                      }
                    }
                    stage('second-parallel') {
                      stages {
                        stage('first-nested-parallel') {
                          steps {
                             echo 'the first of the nested parallel stages'
                             sleep 30
                          }
                       }
                       stage('second-nested-parallel') {
                          steps {
                             echo 'the second of the nested parallel stages'
                             sleep 30
                          }
                       }
                    }
                 }
              }
            }
            {code}

            So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well.

            I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages.

            Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.
            abayer Andrew Bayer made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "PR #152 (Web Link)" [ 16107 ]
            jamesdumay James Dumay made changes -
            Description *Scope*
            *

            *Proposal*
            Note - this is a post-1.0 feature.

            Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, {{parallel}} doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

            I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this:

            {code}
            pipeline {
              agent any
              
              stages {
                stage('first') {
                  steps {
                    echo 'first, non-parallel stage'
                  }
                }

                stage('top-parallel') {
                  stages {
                    stage('first-parallel') {
                      steps {
                        echo 'First of the parallel stages without further nesting'
                        sleep 60
                      }
                    }
                    stage('second-parallel') {
                      stages {
                        stage('first-nested-parallel') {
                          steps {
                             echo 'the first of the nested parallel stages'
                             sleep 30
                          }
                       }
                       stage('second-nested-parallel') {
                          steps {
                             echo 'the second of the nested parallel stages'
                             sleep 30
                          }
                       }
                    }
                 }
              }
            }
            {code}

            So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well.

            I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages.

            Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.
            {panel:title=Improvement on roadmap|titleBGColor=#3878de|titleColor=white}
            This improvement is on the Blue Ocean project roadmap. Check the [roadmap page|https://jenkins.io/projects/blueocean/roadmap/] for updates.
            {panel}

            *Proposal*
            Note - this is a post-1.0 feature.

            Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, {{parallel}} doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

            I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this:

            {code}
            pipeline {
              agent any
              
              stages {
                stage('first') {
                  steps {
                    echo 'first, non-parallel stage'
                  }
                }

                stage('top-parallel') {
                  stages {
                    stage('first-parallel') {
                      steps {
                        echo 'First of the parallel stages without further nesting'
                        sleep 60
                      }
                    }
                    stage('second-parallel') {
                      stages {
                        stage('first-nested-parallel') {
                          steps {
                             echo 'the first of the nested parallel stages'
                             sleep 30
                          }
                       }
                       stage('second-nested-parallel') {
                          steps {
                             echo 'the second of the nested parallel stages'
                             sleep 30
                          }
                       }
                    }
                 }
              }
            }
            {code}

            So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well.

            I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages.

            Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.
            abayer Andrew Bayer made changes -
            Link This issue is duplicated by JENKINS-40699 [ JENKINS-40699 ]
            abayer Andrew Bayer made changes -
            Link This issue relates to JENKINS-40986 [ JENKINS-40986 ]
            jamesdumay James Dumay made changes -
            Epic Link JENKINS-45425 [ 183593 ]
            jamesdumay James Dumay made changes -
            Sprint Declarative - 1.2 [ 366 ]
            jamesdumay James Dumay made changes -
            Rank Ranked lower
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            abayer Andrew Bayer made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            kshultz Karl Shultz made changes -
            Attachment screenshot-1.png [ 39846 ]
            quas Jakub Pawlinski made changes -
            Attachment OldVsNew.htm [ 39897 ]
            quas Jakub Pawlinski made changes -

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                abayer Andrew Bayer
              • Votes:
                42 Vote for this issue
                Watchers:
                86 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: