Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39464

Pipeline with parallel and no stages does not get visualised correctly

    Details

    • Similar Issues:
    • Sprint:
      tethys

      Description

      Problem
      If you have a parallel declaration, but no stages, in classic stage view nothing shows up (you just have to look at log). In blue ocean, you get a single node (stage) which shows only the first branch.
      Expected behavior: should show 3 parallel branches (ideally)

      Example

      parallel 'branch1': {
              node('node1') {
                  stage('Setup') {
                      checkout([details removed])
                  }
                  stage('Unit and Integration Tests') {
                      bat '"my command to execute tests"'
                  }
              }
      }, 'branch2': {
              node('node2') {
                  stage('Setup') {
                      checkout([details removed])
                  }
                  stage('Unit and Integration Tests') {
                      bat '"my command to execute tests"'
                  }
              }
      }
      

      Workaround
      Simply wrap the parallels in a stage and it will work correctly.

      stage('Build and Test') {
        parallel 'branch1': {
                node('node1') {
                    stage('Setup') {
                        checkout([details removed])
                    }
                    stage('Unit and Integration Tests') {
                        bat '"my command to execute tests"'
                    }
                }
        }, 'branch2': {
                node('node2') {
                    stage('Setup') {
                        checkout([details removed])
                    }
                    stage('Unit and Integration Tests') {
                        bat '"my command to execute tests"'
                    }
                }
        }
      }
      

      It will then be visualised like:

      Other examples

      Parallel defined between stages but not nested in its own stage

      node {
      	stage('Checkout') {
      		checkout scm
      	}
      }
      
      parallel linux: {
      	node('Linux') {
      		stage('Build') {
      			echo 'Build linux'
      		}
      		stage('Tests') {
      			echo 'Tests linux'
      		}
      		stage('Static analysis') {
      			echo 'Static analysis linux'
      		}
      	}
      },
      windows: {
      	node('Windows') {
      		stage('Build') {
      			echo 'Build windows'
      		}
      		stage('Tests') {
      			echo 'Tests windows'
      		}
      		stage('Static analysis') {
      			echo 'Static analysis windows'
      		}
      	}
      }
      

      Result looks like this:

        Attachments

          Issue Links

            Activity

            Hide
            michaelneale Michael Neale added a comment -

            Not sure if this should be fixed in API or front end. Leaning to former.

            Show
            michaelneale Michael Neale added a comment - Not sure if this should be fixed in API or front end. Leaning to former.
            Hide
            jamesdumay James Dumay added a comment -

            R. Tyler Croy thanks for reporting this one. We are likely to take this on in the coming weeks. As a work around you can wrap your parallel in a stage.

            Show
            jamesdumay James Dumay added a comment - R. Tyler Croy thanks for reporting this one. We are likely to take this on in the coming weeks. As a work around you can wrap your parallel in a stage.
            Hide
            jamesdumay James Dumay added a comment -

            Thorsten Scherler / Vivek Pandey is this likely to be a frontend or backend issue?

            Show
            jamesdumay James Dumay added a comment - Thorsten Scherler / Vivek Pandey is this likely to be a frontend or backend issue?
            Hide
            jamesdumay James Dumay added a comment -

            Now we have a start dot we can fix this problem as part of JENKINS-39628

            Show
            jamesdumay James Dumay added a comment - Now we have a start dot we can fix this problem as part of JENKINS-39628
            Hide
            michaelneale Michael Neale added a comment -

            yeah this comes up again and again... so worth catering to IMO

            Show
            michaelneale Michael Neale added a comment - yeah this comes up again and again... so worth catering to IMO
            Hide
            jamesdumay James Dumay added a comment - - edited

            Sam Van Oort Vivek Pandey is there some way we could have this show up as if it was wrapped in a stage? not sure if we should do this in the frontend code, Blue Ocean or bismuth level.

            Show
            jamesdumay James Dumay added a comment - - edited Sam Van Oort Vivek Pandey is there some way we could have this show up as if it was wrapped in a stage? not sure if we should do this in the frontend code, Blue Ocean or bismuth level.
            Hide
            jlpinardon jlpinardon added a comment -

            Dear all,

            I made a trial
            stagesNestedinParallel.xml
            that is equivalent to the example given here. And, for sure, I felt into the same bug.
            Here is what it is displayed :

            I suggest that such a parallel pipeline (configure, make and test sequential stages run within several parallel "branches") could be displayed as :

            Note that the "legacy" visualization is even unable to correctly display the duration of the stages :

            Show
            jlpinardon jlpinardon added a comment - Dear all, I made a trial stagesNestedinParallel.xml that is equivalent to the example given here. And, for sure, I felt into the same bug. Here is what it is displayed : I suggest that such a parallel pipeline (configure, make and test sequential stages run within several parallel "branches") could be displayed as : Note that the "legacy" visualization is even unable to correctly display the duration of the stages :
            Hide
            jamesdumay James Dumay added a comment - - edited

            jlpinardon could you share the Jenkinsfile/Script please? You might also want try the workaround provided in the ticket description.

            Show
            jamesdumay James Dumay added a comment - - edited jlpinardon could you share the Jenkinsfile/Script please? You might also want try the workaround provided in the ticket description.
            Hide
            jlpinardon jlpinardon added a comment -

            Here it is :

            env.sleepTime=10
            node('MTS_RH6_X64') {
                env.initialName='Level1'
                // Initializing level 1
                stage('Start') {
                    env.stageName=env.initialName
                    sh('''
                            #!/bn/bash
                            echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                            sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                    ''')
                }
            
                parallel 'MRS_LDB':{
            
                    node('ENODEB_MRS_LDB') {
                        env.stageName='configure'
                        stage ('configure') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')
                        }
                        env.stageName='make'
                        stage ('make') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')        }
                        env.stageName='test'
                        stage ('test') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')
                        }
                    }
                   
                }, 
                'MTS_LRH5':{
                    node('MTS_LRH5') {
                        env.stageName='configure'
                        stage ('configure') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')
                        }
                        env.stageName='make'
                        stage ('make') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')        }
                        env.stageName='test'
                        stage ('test') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')
                        }
                    }
                },
                'LB': {
                    node('MTS_RH6_X64') {
                        env.stageName='configure'
                        stage ('configure') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')
                        }
                        env.stageName='make'
                        stage ('make') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')        }
                        env.stageName='test'
                        stage ('test') {
                        sh('''
                                #!/bn/bash
                                echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                                sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                        ''')
                        }
                    }
                }
                env.stageName=env.initialName+'_Deploy'
                stage('Deploy') {
                sh('''
                    #!/bin/bash
                    sleep $(( 1 + $RANDOM % ${sleepTime} ))
                    echo "This is the end for ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                ''')          
                }       
            }
            
            Show
            jlpinardon jlpinardon added a comment - Here it is : env.sleepTime=10 node( 'MTS_RH6_X64' ) { env.initialName= 'Level1' // Initializing level 1 stage( 'Start' ) { env.stageName=env.initialName sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } parallel 'MRS_LDB' :{ node( 'ENODEB_MRS_LDB' ) { env.stageName= 'configure' stage ( 'configure' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'make' stage ( 'make' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'test' stage ( 'test' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } } }, 'MTS_LRH5' :{ node( 'MTS_LRH5' ) { env.stageName= 'configure' stage ( 'configure' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'make' stage ( 'make' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'test' stage ( 'test' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } } }, 'LB' : { node( 'MTS_RH6_X64' ) { env.stageName= 'configure' stage ( 'configure' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'make' stage ( 'make' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'test' stage ( 'test' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } } } env.stageName=env.initialName+ '_Deploy' stage( 'Deploy' ) { sh(''' #!/bin/bash sleep $(( 1 + $RANDOM % ${sleepTime} )) echo "This is the end for ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" ''') } }
            Hide
            jlpinardon jlpinardon added a comment - - edited

            With the workaround, the display is indeed "correct". I mean it is exact. But I think (as far as it is feasible) that the sequential stuff within could be displayed as in the my proposed schema. At least for the first level.

            Nevertheless, the display during the build run is not correct as shown below :

            Parallel blocks are shown as running (which is correct) but stages within are all displayed sequentially after, and lokke like achieved (which is for sure totally wrong).

            Show
            jlpinardon jlpinardon added a comment - - edited With the workaround, the display is indeed "correct". I mean it is exact. But I think (as far as it is feasible) that the sequential stuff within could be displayed as in the my proposed schema. At least for the first level. Nevertheless, the display during the build run is not correct as shown below : Parallel blocks are shown as running (which is correct) but stages within are all displayed sequentially after, and lokke like achieved (which is for sure totally wrong).
            Hide
            jamesdumay James Dumay added a comment - - edited

            jlpinardon absolutely - we can't render those stage's within the parallels (see JENKINS-38442).

            The reason why it doesn't show correctly after you change the structure of the Pipeline while executing is that we rely on the previous run to "preview" what the flow of the Pipeline will be. Unfortunately as Pipeline is a scripting language we can't read ahead the structure without executing it. The good news is that the new declarative syntax for Pipeline doesn't have this problem - we can read the Jenkinsfile like a config file

            If you run it a second time you shouldn't see the problem.

            Show
            jamesdumay James Dumay added a comment - - edited jlpinardon absolutely - we can't render those stage 's within the parallels (see JENKINS-38442 ). The reason why it doesn't show correctly after you change the structure of the Pipeline while executing is that we rely on the previous run to "preview" what the flow of the Pipeline will be. Unfortunately as Pipeline is a scripting language we can't read ahead the structure without executing it. The good news is that the new declarative syntax for Pipeline doesn't have this problem - we can read the Jenkinsfile like a config file If you run it a second time you shouldn't see the problem.
            Hide
            jlpinardon jlpinardon added a comment -

            Thanks for the good news I will try this as soon as possible.
            But concerning the display while running,the bad news is that it is still broken after several run.

            Show
            jlpinardon jlpinardon added a comment - Thanks for the good news I will try this as soon as possible. But concerning the display while running,the bad news is that it is still broken after several run.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Sam Van Oort Vivek Pandey would it be feasible to synthesise a stage around a parallel when it is not within a stage itself? We're starting to see more and more questions about why the visualization doesn't work (obviously not an issue with Declarative). I would like to get an idea of how feasible this is and a rough estimate.

            Show
            jamesdumay James Dumay added a comment - - edited Sam Van Oort Vivek Pandey would it be feasible to synthesise a stage around a parallel when it is not within a stage itself? We're starting to see more and more questions about why the visualization doesn't work (obviously not an issue with Declarative). I would like to get an idea of how feasible this is and a rough estimate.
            Hide
            vivek Vivek Pandey added a comment -

            James Dumay I will start with what I think is work involved or issues we need to solve and then make determination:

            • This virtual stage would collate all orphan (with no parent stage) consecutive top level parallels (with no stages in between)
            • What would be the display name or label of each virtual stage?
            • Current implementation is event based parsing API (bismuth), it sends events for all nodes. For stage it sends events that help compute stages, status, timing info etc.
            • * Computation of stage would require copying/collating status from underlying parallel
            • * Computation of timing will involve capture timing from underlying parallel blocks. This might be incorrect unless we capture all out-of-stage nodes (steps as well) and then compute timing info (duration, start and end)
            • Sythentic stage id generation. node id is generated during execution, its simple integer. I am hoping CpsFlowExecution.iota() might just do it but maybe there are edge cases where there are chances of id conflict - like pipeline is midway execution so id of synthetic stage could be reused later? Or we generate UUID for these nodes ourselves?

            Sam Van Oort are there other complications it might cause or better approach to handle it?

            Show
            vivek Vivek Pandey added a comment - James Dumay I will start with what I think is work involved or issues we need to solve and then make determination: This virtual stage would collate all orphan (with no parent stage) consecutive top level parallels (with no stages in between) What would be the display name or label of each virtual stage? Current implementation is event based parsing API (bismuth), it sends events for all nodes. For stage it sends events that help compute stages, status, timing info etc. * Computation of stage would require copying/collating status from underlying parallel * Computation of timing will involve capture timing from underlying parallel blocks. This might be incorrect unless we capture all out-of-stage nodes (steps as well) and then compute timing info (duration, start and end) Sythentic stage id generation. node id is generated during execution, its simple integer. I am hoping CpsFlowExecution.iota() might just do it but maybe there are edge cases where there are chances of id conflict - like pipeline is midway execution so id of synthetic stage could be reused later? Or we generate UUID for these nodes ourselves? Sam Van Oort are there other complications it might cause or better approach to handle it?
            Hide
            jamesdumay James Dumay added a comment -

            Vivek Pandey if we have to synthesise a stage I am happy with it being called "Parallel". Would it be possible to attach some data to the synthesised node so that we can show a warning in the UI about how it has been transformed? I'd like to educate users more about the usage of Stage.

            Show
            jamesdumay James Dumay added a comment - Vivek Pandey if we have to synthesise a stage I am happy with it being called "Parallel". Would it be possible to attach some data to the synthesised node so that we can show a warning in the UI about how it has been transformed? I'd like to educate users more about the usage of Stage.
            Hide
            jamesdumay James Dumay added a comment -

            This fix will be released in 1.0.0-b19 next week.

            Show
            jamesdumay James Dumay added a comment - This fix will be released in 1.0.0-b19 next week.
            Hide
            quas Jakub Pawlinski added a comment - - edited

            Hi, running Blue Ocean 1.1.4 in declarative pipeline I can't make it work:

             
            stages {  stage('Parallel1') { 
                parallel 'branch1': {
                  node('n1') { stage('Unit 1') { echo "1" }}{{} }}{
                }, 'branch2': {
                  node('n2') { stage('Unit 2') { echo "2" }
            }}

            • Result: Starting with version 0.5, steps in a stage must be in a steps block
               
              stages {
                steps {
                  parallel 'branch1': { 
                    node('n1') { stage('Unit 1') { echo "1" } } 
                  }, 'branch2': {
                    node('n2') { stage('Unit 2') { echo "2" } }
              }}
            • Result: Invalid step "stage" used - not allowed in this context - The stage step cannot be used in Declarative Pipelines
              Running any of above outside of stages{} results with – Expected a stage exception
               
              I have created https://groups.google.com/forum/#!topic/jenkinsci-users/ObAZcvsiZA8 so hopefully will find workaround there
               
            Show
            quas Jakub Pawlinski added a comment - - edited Hi, running Blue Ocean 1.1.4 in declarative pipeline I can't make it work:   stages {   stage('Parallel1') {       parallel 'branch1': {       node('n1') {  stage('Unit 1') {  echo "1" }}{{} }}{     }, 'branch2': {       node('n2') {  stage('Unit 2') {  echo "2"  }  } }} Result: Starting with version 0.5, steps in a stage must be in a steps block   stages {   steps {     parallel 'branch1': {        node('n1') {  stage('Unit 1') {  echo "1"  }  }      }, 'branch2': {       node('n2') {  stage('Unit 2') {  echo "2"  }   } }} Result: Invalid step "stage" used - not allowed in this context - The stage step cannot be used in Declarative Pipelines Running any of above outside of stages{} results with – Expected a stage exception   I have created https://groups.google.com/forum/#!topic/jenkinsci-users/ObAZcvsiZA8  so hopefully will find workaround there  
            Hide
            tsvi Tsvi Mostovicz added a comment -

            Hi, I'm running Blue Ocean 1.3.5 and am building the following:

            stages {
              stage('Build') {
                parallel {
                  stage('Synthesis') {
                    ...
                  }
                  stage('Compilation') {
                    // Generate compilation matrix
                    ...
                    catchError {
                      parallel compilations
                    }
                  }
                }
              }
            }
            

            I have 3 issues:

            1. All parallel stages are flattened into a long list. Preferably Synthesis and Compilation should split from the main pipeline, and then each of them should split into their own matrices.
            2. The stage that is failing is not the dot that is red. When clicking on the dot in the graph, I get the correct stage, but the color does not reflect the result. See the following screenshots.
            3. A minor issue: the stage name doesn't fit in the column. (but that's unrelated to here).

            Selected the red dot but the stage passed:

            Selected the green dot underneath it, which is the actual stage that failed.

            Show
            tsvi Tsvi Mostovicz added a comment - Hi, I'm running Blue Ocean 1.3.5 and am building the following: stages { stage( 'Build' ) { parallel { stage( 'Synthesis' ) { ... } stage( 'Compilation' ) { // Generate compilation matrix ... catchError { parallel compilations } } } } } I have 3 issues: All parallel stages are flattened into a long list. Preferably Synthesis and Compilation should split from the main pipeline, and then each of them should split into their own matrices. The stage that is failing is not the dot that is red. When clicking on the dot in the graph, I get the correct stage, but the color does not reflect the result. See the following screenshots. A minor issue: the stage name doesn't fit in the column. (but that's unrelated to here). Selected the red dot but the stage passed: Selected the green dot underneath it, which is the actual stage that failed.
            Hide
            trane9991 Taras Postument added a comment -

            Vivek Pandey
            I'm facing the same problem. Any progress here?

            See my complete issue described here: JENKINS-39464

            Show
            trane9991 Taras Postument added a comment - Vivek Pandey I'm facing the same problem. Any progress here? See my complete issue described here:  JENKINS-39464
            Hide
            rthakkar Rishi Thakkar added a comment -

            Why does the resolution to this issue say "Fixed" but the issue itself is still "In Porgress"?

            Show
            rthakkar Rishi Thakkar added a comment - Why does the resolution to this issue say "Fixed" but the issue itself is still "In Porgress"?

              People

              • Assignee:
                vivek Vivek Pandey
                Reporter:
                michaelneale Michael Neale
              • Votes:
                8 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: