Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35836

Steps outside a stage or within a stage but outside a parallel do not get visualized

    Details

    • Sprint:
      1.0-japan-m9, 1.0-m7
    • Similar Issues:

      Description

      This feature request is unplanned and is not actively being worked on

      We are unable to merge orphaned steps (that is, steps that exist outside of a stage or outside of a parallel within a stage) into the closest stage or a synthetic without considerable engineering effort to ensure that all steps are available in the visualization.

      As a workaround we recommend that you wrap any steps within stage and do not include steps sitting outside a parallel in the same stage or try Declarative Pipeline which was designed not to have this problem.

      Scope
      For visualisation purposes:

      • Step outside stage is relocated to the previous stage or next stage (depending if at start, middle or end of the pipeline)
      • Step before a parallel block is relocated to the first parallel branch
      • Step after a parallel block is relocated to the last parallel branch

      Problem
      It is possible that a stage, that has parallel branches under it, also has steps before or after a parallel section.

      Example 1 - step outside stage block
      The output of the step that echos never going to let you down will never be visible from the UI.

      
      node {
        stage "Stage 1" {
          sh "echo never going to give you up"
        }
        sh "echo never going to let you down"
        stage "Stage 2" {
          parallel (
            "Firefox" : { echo "Never run around" },
            "Edge" : { echo "Never desert you" }
          ) 
        }
        
        stage "Stage 3" {
          echo "Make you cry (never)"
          echo "Tell a lie and hurt you (never)"
        }
      }
      
      

      Example 2 - step inside stage but not within parallel
      The output of the step that echos never going to let you down will never be visible from the UI.

      node {
        stage "Stage 1" {
          sh "echo never going to give you up"
        }
        
        stage "Stage 2" {
          sh "echo never going to let you down"
          parallel (
            "Firefox" : { echo "Never run around" },
            "Edge" : { echo "Never desert you" }
          ) 
        }
        
        stage "Stage 3" {
          echo "Make you cry (never)"
          echo "Tell a lie and hurt you (never)"
        }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            sophistifunk Josh McDonald added a comment -

            This becomes easier if we can make some assumptions such as "stages will only be top-level 'constructs', and pipeline will blow up if you put a stage inside parallel or node"

            Show
            sophistifunk Josh McDonald added a comment - This becomes easier if we can make some assumptions such as "stages will only be top-level 'constructs', and pipeline will blow up if you put a stage inside parallel or node"
            Hide
            michaelneale Michael Neale added a comment -

            we can make some assumptions about where stage goes, yes.

            Show
            michaelneale Michael Neale added a comment - we can make some assumptions about where stage goes, yes.
            Hide
            jamesdumay James Dumay added a comment -

            [~mneale] based on our discussion sounds like we should add another node for these before the parallel if steps like this exist.

            Is that good enough?

            Show
            jamesdumay James Dumay added a comment - [~mneale] based on our discussion sounds like we should add another node for these before the parallel if steps like this exist. Is that good enough?
            Hide
            michaelneale Michael Neale added a comment -

            James Dumay yes agree.

            Is an ugly take on it - in this case there is a "Test" stage that has some steps before parallel. It could have after. If there are no steps before or after, we don't need to show those circles.

            Maybe [~jmcdonald] or James Dumay could sketch a way that looks reasonable in either case of there being steps? And where do we show the stage name? (I think that the cirlces should be grouped close to the parallel forking, to make the association clear?)

            Show
            michaelneale Michael Neale added a comment - James Dumay yes agree. Is an ugly take on it - in this case there is a "Test" stage that has some steps before parallel. It could have after. If there are no steps before or after, we don't need to show those circles. Maybe [~jmcdonald] or James Dumay could sketch a way that looks reasonable in either case of there being steps? And where do we show the stage name? (I think that the cirlces should be grouped close to the parallel forking, to make the association clear?)
            Hide
            sophistifunk Josh McDonald added a comment -

            I'll draw a few out to see if I can cook up something I like.

            Show
            sophistifunk Josh McDonald added a comment - I'll draw a few out to see if I can cook up something I like.
            Hide
            sophistifunk Josh McDonald added a comment -

            [^visible grouping.pdf]

            Show
            sophistifunk Josh McDonald added a comment - [^visible grouping.pdf]
            Hide
            michaelneale Michael Neale added a comment -

            I like this latest one from [~jmcdonald] - I think this makes sense.

            Show
            michaelneale Michael Neale added a comment - I like this latest one from [~jmcdonald] - I think this makes sense.
            Hide
            jamesdumay James Dumay added a comment -

            [~jmcdonald] we can give those anon steps a label in italics like "Preparation". I like.

            Show
            jamesdumay James Dumay added a comment - [~jmcdonald] we can give those anon steps a label in italics like "Preparation". I like.
            Hide
            jamesdumay James Dumay added a comment -

            [~mneale] [~jmcdonald] lets make it happen. Its a good fix for this edge case.

            Show
            jamesdumay James Dumay added a comment - [~mneale] [~jmcdonald] lets make it happen. Its a good fix for this edge case.
            Hide
            jamesdumay James Dumay added a comment -

            Michael Neale did we ever come to a conclusion about this one?

            Show
            jamesdumay James Dumay added a comment - Michael Neale did we ever come to a conclusion about this one?
            Hide
            michaelneale Michael Neale added a comment -

            James Dumay no did not... Josh has some proposed designs of where things could hang.

            Show
            michaelneale Michael Neale added a comment - James Dumay no did not... Josh has some proposed designs of where things could hang.
            Hide
            jamesdumay James Dumay added a comment -

            Thanks

            Show
            jamesdumay James Dumay added a comment - Thanks
            Hide
            roidelapluie Julien Pivotto added a comment - - edited

            Given the following pipeline

            node {
                //out of the stage
                sh ('sleep 10')
              stage('S1') {  
                // prepare the parallel jobs
                sh ('sleep 20')
                
                // run them
                parallel 'S1.1.1': { sh('sleep 10') }, 'S1.1.2': { sh('sleep 10') }
                
                // another one
                sh ('sleep 10')
              }
            
                // cleanup
                sh ('sleep 10')
            }
            

            All the steps outstage stage // parallels do not show up in blue ocean.

            This very annoying. You sometimes want to do some prep in a stage before running parallels.

            proposal:

            Show
            roidelapluie Julien Pivotto added a comment - - edited Given the following pipeline node { //out of the stage sh ( 'sleep 10' ) stage( 'S1' ) { // prepare the parallel jobs sh ( 'sleep 20' ) // run them parallel 'S1.1.1' : { sh( 'sleep 10' ) }, 'S1.1.2' : { sh( 'sleep 10' ) } // another one sh ( 'sleep 10' ) } // cleanup sh ( 'sleep 10' ) } All the steps outstage stage // parallels do not show up in blue ocean. This very annoying. You sometimes want to do some prep in a stage before running parallels. proposal:
            Hide
            vlatombe Vincent Latombe added a comment -

            +1 on this. It also makes following the progress of these steps outside parallel impossible since anything output at that stage never appears.

            Show
            vlatombe Vincent Latombe added a comment - +1 on this. It also makes following the progress of these steps outside parallel impossible since anything output at that stage never appears.
            Hide
            jamesdumay James Dumay added a comment -

            Vivek Pandey in this scenario could we fold steps outside the parallel or stage into the nearest stage or parallel ?

            Show
            jamesdumay James Dumay added a comment - Vivek Pandey in this scenario could we fold steps outside the parallel or stage into the nearest stage or parallel ?
            Hide
            vivek Vivek Pandey added a comment -

            Stages are guaranteed to be ordered so folding steps outside stages in to nearest one is doable.

            However steps outside parallel I am not sure. In DAG returned via API, we only capture parallel branches. To fold such steps in to nearest parallel branch, we need parallelBranch event from graph-analysis API to be in order of declaration which is not the case as order of parallel branch execution are not guaranteed to be in order of declaration. Sam Van Oort can confirm or tell us if there is some way we can capture it.

            Show
            vivek Vivek Pandey added a comment - Stages are guaranteed to be ordered so folding steps outside stages in to nearest one is doable. However steps outside parallel I am not sure. In DAG returned via API, we only capture parallel branches. To fold such steps in to nearest parallel branch, we need parallelBranch event from graph-analysis API to be in order of declaration which is not the case as order of parallel branch execution are not guaranteed to be in order of declaration. Sam Van Oort can confirm or tell us if there is some way we can capture it.
            Hide
            roidelapluie Julien Pivotto added a comment - - edited

            If you look at my proposal closer:

            the horizontal axis means "time", not "stage". It means that groups of steps outside of parallel (or stage) are represented by nodes.

            If there are 2 x 2 parallel, they are displayed

            1 3
            2 4

            and not

            1
            2
            3
            4

            Again, it is about time and no longer about stage.

            You can identify the stage by the line on top of the nodes, if the stage covers multiple columns.

            Show
            roidelapluie Julien Pivotto added a comment - - edited If you look at my proposal closer: the horizontal axis means "time", not "stage". It means that groups of steps outside of parallel (or stage) are represented by nodes. If there are 2 x 2 parallel, they are displayed 1 3 2 4 and not 1 2 3 4 Again, it is about time and no longer about stage. You can identify the stage by the line on top of the nodes, if the stage covers multiple columns.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Julien Pivotto I have updated the ticket with a bit of a description of how we would like to handle this use case.

            EDIT: also for your last comment that problem is covered in depth in JENKINS-38442.

            Show
            jamesdumay James Dumay added a comment - - edited Julien Pivotto I have updated the ticket with a bit of a description of how we would like to handle this use case. EDIT: also for your last comment that problem is covered in depth in JENKINS-38442 .
            Hide
            michaelneale Michael Neale added a comment -

            I have made this one dependent on nested stages, as it seems after a closer look to be very very similar (at least the solutions will likely be similar). Ie if we can support nested stages, we can support nested synthetic stages for these post/pre parallel steps

            Show
            michaelneale Michael Neale added a comment - I have made this one dependent on nested stages, as it seems after a closer look to be very very similar (at least the solutions will likely be similar). Ie if we can support nested stages, we can support nested synthetic stages for these post/pre parallel steps
            Hide
            roidelapluie Julien Pivotto added a comment - - edited

            James Dumay This is pretty bad. I think as an immediate action we could at least print a warning when some steps are not displayed – because that will confuse lots of users:

            «Warning! This pipeline contains steps that can not be displayed in Blue Ocean. Please go to [classic UI] to see all the steps. Read more in JENKINS-35836

            JENKINS-35836 is a link to this,
            [classic UI] is a link to the build "pipeline steps" view.

            Show
            roidelapluie Julien Pivotto added a comment - - edited James Dumay This is pretty bad. I think as an immediate action we could at least print a warning when some steps are not displayed – because that will confuse lots of users: «Warning! This pipeline contains steps that can not be displayed in Blue Ocean. Please go to [classic UI] to see all the steps. Read more in JENKINS-35836 .» JENKINS-35836 is a link to this, [classic UI] is a link to the build "pipeline steps" view.
            Hide
            jamesdumay James Dumay added a comment -

            Julien Pivotto we were discussing warnings today. Stay tuned!

            Show
            jamesdumay James Dumay added a comment - Julien Pivotto we were discussing warnings today. Stay tuned!
            Hide
            jamesdumay James Dumay added a comment -

            Hi all,

            I want to be honest that it is unlikely that we will be able to satisfy everyone with any of the suggested resolutions on this ticket and that I will be closing this out.

            However, if any community member has a bright idea on how to solve it and wants to try to make it work we'd be more than happy to guide and assist.

            As per the description, there are two workarounds given that help solve this problem - and one is the brand new Declarative Pipeline syntax which was designed to solve this exact problem.

            Thanks,
            James

            Show
            jamesdumay James Dumay added a comment - Hi all, I want to be honest that it is unlikely that we will be able to satisfy everyone with any of the suggested resolutions on this ticket and that I will be closing this out. However, if any community member has a bright idea on how to solve it and wants to try to make it work we'd be more than happy to guide and assist. As per the description, there are two workarounds given that help solve this problem - and one is the brand new Declarative Pipeline syntax which was designed to solve this exact problem. Thanks, James
            Hide
            giorgiosironi Giorgio Sironi added a comment - - edited

            I'm not sure how the Declarative Pipeline Syntax solves this problem - when I try to define:

            pipeline {
              agent none
              stages {
                stage('My stage') {
                  steps {
                    echo 'outside'
                    parallel {
                        stage('A') {
                            echo 'a'
                        }
                        stage('B') {
                            echo 'a'
                        }
                    }
                  }
                }
              }
            }
            

            I get a syntax error:

            Invalid step "parallel" used - not allowed in this context - The parallel step can only be used as the only top-level step in a stages step block

            My workaround is possibly to use xargs -P or GNU parallel, but it would be a pain if the branches are Groovy code.

            Show
            giorgiosironi Giorgio Sironi added a comment - - edited I'm not sure how the Declarative Pipeline Syntax solves this problem - when I try to define: pipeline { agent none stages { stage( 'My stage' ) { steps { echo 'outside' parallel { stage( 'A' ) { echo 'a' } stage( 'B' ) { echo 'a' } } } } } } I get a syntax error: Invalid step "parallel" used - not allowed in this context - The parallel step can only be used as the only top-level step in a stages step block My workaround is possibly to use xargs -P or GNU parallel, but it would be a pain if the branches are Groovy code.
            Hide
            giorgiosironi Giorgio Sironi added a comment -

            Also, if you have an error like a test failure in the outside are that is not inside parallel, opening the build actually sends you to /blue/organizations/jenkins/:pipeline/detail/:pipeline/:build_number/pipeline which shows everything. So it doesn't seem true that the output is not visualized, just that it is not reachable through navigation. Try this to reproduce:

            node {
                stage 'My stage', {
                    def actions = [:]
                    actions['a'] = {
                        echo 'a'
                    }
                    actions['b'] = {
                        echo 'b'
                    }
                    parallel actions
                    sh 'echo feels good to be outside parallel'
                    throw new RuntimeException("outside, later error")
                }
            }
            

            It gives this perfectly good output:

            However, it only renders this if there is a failure outside parallel.

            As for ideas:

            • make this output available even if there is no failure in that area rather than selecting the last parallel stage by default.
            • make the stage name (on the Start...End line) clickable, to open this full output.
            Show
            giorgiosironi Giorgio Sironi added a comment - Also, if you have an error like a test failure in the outside are that is not inside parallel , opening the build actually sends you to /blue/organizations/jenkins/:pipeline/detail/:pipeline/:build_number/pipeline which shows everything. So it doesn't seem true that the output is not visualized, just that it is not reachable through navigation. Try this to reproduce: node { stage 'My stage' , { def actions = [:] actions[ 'a' ] = { echo 'a' } actions[ 'b' ] = { echo 'b' } parallel actions sh 'echo feels good to be outside parallel' throw new RuntimeException( "outside, later error" ) } } It gives this perfectly good output: However, it only renders this if there is a failure outside parallel . As for ideas: make this output available even if there is no failure in that area rather than selecting the last parallel stage by default. make the stage name (on the Start ... End line) clickable, to open this full output.

              People

              • Assignee:
                Unassigned
                Reporter:
                jamesdumay James Dumay
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: