Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53734

Declarative pipeline - Support for parallel stages inside sequential stages

    Details

    • Similar Issues:

      Description

      Yesterday, I tried using sequential stages. I have a pipeline consisting of 2 distinct parts, let's say JAVA PART and DOCKER PART, each with a different agent and several stages. Thus, I grouped them into two stages, assigned an agent and well, it worked as expected. Things went south when I wanted to use parallel stages inside JAVA PART:

      pipeline {
          agent none
          
          stages {
              stage("JAVA PART") {
                  agent A
      
                  stages {
                      stage("Java one") {
                          steps {
                              echo "Java one"
                          }
                      }
                      
                      stage("Java two") {
                          parallel {
                              stage("Java parallel one") {
                                  steps {
                                      echo "Java parallel one"
                                  }
                              }
                              
                              stage("Java parallel two") {
                                  steps {
                                      echo "Java parallel two"
                                  }
                              }
                          }
                      }
                      
                      stage("Java three") {
                          steps {
                              echo "Java three"
                          }
                      }
                  }
              }
              
              stage("DOCKER PART") {
                  agent B
                  
                  steps {
                      echo "Docker one"
                  }
              }
          }
      }
      

      I ended up with 

      Parallel stages or branches can only be included in a top-level stage.
      

      I wonder what the rationale behind this restriction is, because when I switch from parallel to stages and make it serial, everything is fine.

      pipeline {
          agent none
          
          stages {
              stage("JAVA PART") {
                  agent A
      
                  stages {
                      stage("Java one") {
                          steps {
                              echo "Java one"
                          }
                      }
                      
                      stage("Java two") {
                          stages {
                              stage("Java parallel one") {
                                  steps {
                                      echo "Java parallel one"
                                  }
                              }
                              
                              stage("Java parallel two") {
                                  steps {
                                      echo "Java parallel two"
                                  }
                              }
                          }
                      }
                      
                      stage("Java three") {
                          steps {
                              echo "Java three"
                          }
                      }
                  }
              }
              
              stage("DOCKER PART") {
                  agent B
                  
                  steps {
                      echo "Docker one"
                  }
              }
          }
      }
      

        Attachments

          Issue Links

            Activity

            tomis Martin Tee created issue -
            Hide
            tomis Martin Tee added a comment -

            Found a workaround via scripted pipelines, but it is kinda ugly:

            pipeline {
                agent none
                
                stages {
                    stage("JAVA PART") {
                        agent A
            
                        stages {
                            stage("Java one") {
                                steps {
                                    echo "Java one"
                                }
                            }
                            
                            stage("Java two") {
                                steps {
                                    script {
                                        def parallelStepsMap = [:]
                                        parallelStepsMap["Java parallel one"] = {
                                            echo "Java parallel one"
                                        }
                                        parallelStepsMap["Java parallel two"] = {
                                            echo "Java parallel two"
                                        }
            
                                        parallel parallelStepsMap
                                    }
                                }
                            }
                            
                            stage("Java three") {
                                steps {
                                    echo "Java three"
                                }
                            }
                        }
                    }
                    
                    stage("DOCKER PART") {
                        agent B
                        
                        steps {
                            echo "Docker one"
                        }
                    }
                }
            }
            

            I believe declarative pipelines should isolate me from having to do such things.

            Show
            tomis Martin Tee added a comment - Found a workaround via scripted pipelines, but it is kinda ugly: pipeline { agent none stages { stage( "JAVA PART" ) { agent A stages { stage( "Java one" ) { steps { echo "Java one" } } stage( "Java two" ) { steps { script { def parallelStepsMap = [:] parallelStepsMap[ "Java parallel one" ] = { echo "Java parallel one" } parallelStepsMap[ "Java parallel two" ] = { echo "Java parallel two" } parallel parallelStepsMap } } } stage( "Java three" ) { steps { echo "Java three" } } } } stage( "DOCKER PART" ) { agent B steps { echo "Docker one" } } } } I believe declarative pipelines should isolate me from having to do such things.
            abayer Andrew Bayer made changes -
            Field Original Value New Value
            Link This issue is duplicated by JENKINS-53659 [ JENKINS-53659 ]
            abayer Andrew Bayer made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            abayer Andrew Bayer made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            abayer Andrew Bayer added a comment -

            So the motivation was that we don't want to allow arbitrarily deep parallel nesting - nested parallels won't be visualized properly by Blue Ocean, most notably.

            That said, https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/292 - this would allow nesting parallel within a stage's stages if and only if the stage isn't itself already within a parallel. I think that it won't visualize particularly nicely in Blue Ocean (i.e., it may just visualize the whole nested stage with a parallel on its own, without visualizing its children independently), but I think we can live with that anyway.

            Show
            abayer Andrew Bayer added a comment - So the motivation was that we don't want to allow arbitrarily deep parallel nesting - nested parallels won't be visualized properly by Blue Ocean, most notably. That said, https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/292 - this would allow nesting parallel within a stage 's stages if and only if the stage isn't itself already within a parallel . I think that it won't visualize particularly nicely in Blue Ocean (i.e., it may just visualize the whole nested stage with a parallel on its own, without visualizing its children independently), but I think we can live with that anyway.
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "PR #292 (Web Link)" [ 21926 ]
            Hide
            tkleiber Torsten Kleiber added a comment -

            I don't agree that this should not be rendered! Visualization of a pipeline for end user is important! Every other job control software has no problem with such parallelisation's. And for optimization of build time I need a lot of parallelisation's. 

            So I would request the corresponding changes in blue ocean too after implementation of this change.

            In the meantime I run in several issues in Jenkins, where implementing one change in some plugin contradicts other restrictions in another  plugin.

            Show
            tkleiber Torsten Kleiber added a comment - I don't agree that this should not be rendered! Visualization of a pipeline for end user is important! Every other job control software has no problem with such parallelisation's. And for optimization of build time I need a lot of parallelisation's.  So I would request the corresponding changes in blue ocean too after implementation of this change. In the meantime I run in several issues in Jenkins, where implementing one change in some plugin contradicts other restrictions in another  plugin.
            Hide
            onodera Alexey Marin added a comment -

            I agree that this will be very useful. I use sequential stages to lock a single resource (DB) for multiple stages (as suggested in JENKINS-43336)

             

            For example, a pipeline like this one is currently impossible:

            pipeline {
              agent any
              stages {
                stage ('Non-production') {
                  options {
                    lock 'DB1'
                  }
                  stages {
                    stage ('Deploy') {
                      steps {
                        echo 'Deploy'
                      }
                    }
                    stage ('Test') {
                      parallel {
                        stage ('Unit testing') {
                          steps {
                            echo 'Unit'
                          }
                        }
                        stage ('E2E testing') {
                          steps {
                            echo 'E2E'
                          }
                        }
                      }
                    }
                  }
                }    
                stage ('Production') {
                  stages {
                    stage ('Deploy') {
                      steps {
                        echo 'Deploy'
                      }
                    }
                  }
                }
              }
            }
            
            
            Show
            onodera Alexey Marin added a comment - I agree that this will be very useful. I use sequential stages to lock a single resource (DB) for multiple stages (as suggested in JENKINS-43336 )   For example, a pipeline like this one is currently impossible: pipeline { agent any stages { stage ( 'Non-production' ) { options { lock 'DB1' } stages { stage ( 'Deploy' ) { steps { echo 'Deploy' } } stage ( 'Test' ) { parallel { stage ( 'Unit testing' ) { steps { echo 'Unit' } } stage ( 'E2E testing' ) { steps { echo 'E2E' } } } } } } stage ( 'Production' ) { stages { stage ( 'Deploy' ) { steps { echo 'Deploy' } } } } } }
            Hide
            abayer Andrew Bayer added a comment -

            Merged, releasing in 1.3.4 momentarily.

            Show
            abayer Andrew Bayer added a comment - Merged, releasing in 1.3.4 momentarily.
            abayer Andrew Bayer made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            bitwiseman Liam Newman added a comment -

            Bulk closing resolved issues.

            Show
            bitwiseman Liam Newman added a comment - Bulk closing resolved issues.
            bitwiseman Liam Newman made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                tomis Martin Tee
              • Votes:
                2 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: