Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38442

View sequential stages in the pipeline visualization graph

    Details

    • Similar Issues:
    • Released As:
      Blue Ocean 1.8.0

      Description

      In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.

      I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

      Example 1

      def test = [:]
      
      test["a"] = {
          stage ("a") {
              stage ("ab") {
                  sh "echo stage abc"
              }
              stage ("xyz") {
                  sh "echo stage xyz"
              }
          }
      }
      
      test["b"] = {
          stage ("b") {
              stage ("bb") {
                  sh "echo stage bb"
              }
              stage ("bxz") {
                  sh "echo stagebxyz"
              }
          }
      }
      node {
         //stage 'start'
         parallel test
         stage ('middle') {
             sh "echo middle"
         }
         
      }
      

      Example 2

      node() {
        stage('Build') {
          println 'I prepare the build for the parallel steps'
        }
      
        stage('Test') {
         parallel (
       "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
       "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
       "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
      )
        }
      }
      

        Attachments

        1. After.png
          After.png
          6 kB
        2. alternate zoom in.png
          alternate zoom in.png
          85 kB
        3. Before.PNG
          Before.PNG
          8 kB
        4. blueocean.sketch 2016-09-28 15-03-57.png
          blueocean.sketch 2016-09-28 15-03-57.png
          29 kB
        5. blueocean-failure.JPG
          blueocean-failure.JPG
          60 kB
        6. Capture.JPG
          Capture.JPG
          17 kB
        7. example.png
          example.png
          44 kB
        8. image-2017-04-12-16-16-17-144.png
          image-2017-04-12-16-16-17-144.png
          148 kB
        9. image-2017-04-12-16-17-38-285.png
          image-2017-04-12-16-17-38-285.png
          131 kB
        10. image-2017-08-14-10-34-10-452.png
          image-2017-08-14-10-34-10-452.png
          10 kB
        11. image-2017-10-19-08-24-00-504.png
          image-2017-10-19-08-24-00-504.png
          3 kB
        12. jenkins-weird-masked2.png
          jenkins-weird-masked2.png
          52 kB
        13. pipeline_mockup.png
          pipeline_mockup.png
          11 kB
        14. PipelineGraphProposal.png
          PipelineGraphProposal.png
          62 kB
        15. Proposed Parallel Stages.JPG
          Proposed Parallel Stages.JPG
          21 kB
        16. screenshot-1.png
          screenshot-1.png
          4 kB
        17. Screen Shot 2016-10-03 at 4.13.46 PM.png
          Screen Shot 2016-10-03 at 4.13.46 PM.png
          25 kB

          Issue Links

            Activity

            mbadran mohamed badran created issue -
            Hide
            michaelneale Michael Neale added a comment -

            Ah, is this is nested stages, they won't be supported visually for some time (there is some question on how to show them, labelled steps below the parent stage, or a nested block of stages in the graph).

            For now, don't use nested stages, it won't do much good. I am curious what your aim is - you want to label steps or blocks of steps as doing something significant?

            Show
            michaelneale Michael Neale added a comment - Ah, is this is nested stages, they won't be supported visually for some time (there is some question on how to show them, labelled steps below the parent stage, or a nested block of stages in the graph). For now, don't use nested stages, it won't do much good. I am curious what your aim is - you want to label steps or blocks of steps as doing something significant?
            jamesdumay James Dumay made changes -
            Field Original Value New Value
            Issue Type Bug [ 1 ] Story [ 10002 ]
            jamesdumay James Dumay made changes -
            Summary Nested labelled blocks breaks blueocean Support nested stages in the Pipeline Graph
            jamesdumay James Dumay made changes -
            Description I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel title="Support for nested stages is currently unplanned"}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel title="Support for nested stages is currently unplanned"}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title="Support for nested stages is currently unplanned"}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel:title="Support for nested stages is currently unplanned"}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title="Support for nested stages is currently unplanned"|titleBGColor=blue}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel:title="Support for nested stages is currently unplanned"|titleBGColor=blue}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title="Support for nested stages is currently unplanned"|titleBGColor=grey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel:title="Support for nested stages is currently unplanned"|titleBGColor=grey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}

            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            Hide
            jamesdumay James Dumay added a comment -

            mohamed badran thanks for reporting this. Do you have a concrete use case you could share with us for nested stages?

            Show
            jamesdumay James Dumay added a comment - mohamed badran thanks for reporting this. Do you have a concrete use case you could share with us for nested stages?
            Hide
            mbadran mohamed badran added a comment -

            Well currently I'm using it to be able to be able to have a like a container for the shell script in order to know what this shell script is doing since i'm not able to add a description to the shell script that will show up next to Shell Script in blueocean, as well as pipeline visualization. My ideal solution in here is to have availability to add description next to shell script so if i'm running multiple shell scripts in one stage i'd be able to easily know what is that shell script doing. Currently there are no way to do it as i understand so I've tried using nested stages to be able to separate the shell scripts and know what they do. Usage of nested stages like this is a workaround for lack of ability to add description for the shell script.

            Regarding a concrete use case of using nested stages would be being able to relate multiple stages to one stage, for example if I've a major step which prepares the environment, and that preparation consists of fewer stages for example 1- cleaning old data, 2- copy new data, 3- prepare area to run builds. My ideal look for that would be a parent stage which is called Prepare environment, and then when pressing on that node it would fork 3 other nodes (which are the 3 stages I've mentioned), and when you select any of the inner stages you'll be able to get the shell scripts and any other pipeline commands you were doing in such stage.

            Show
            mbadran mohamed badran added a comment - Well currently I'm using it to be able to be able to have a like a container for the shell script in order to know what this shell script is doing since i'm not able to add a description to the shell script that will show up next to Shell Script in blueocean, as well as pipeline visualization. My ideal solution in here is to have availability to add description next to shell script so if i'm running multiple shell scripts in one stage i'd be able to easily know what is that shell script doing. Currently there are no way to do it as i understand so I've tried using nested stages to be able to separate the shell scripts and know what they do. Usage of nested stages like this is a workaround for lack of ability to add description for the shell script. Regarding a concrete use case of using nested stages would be being able to relate multiple stages to one stage, for example if I've a major step which prepares the environment, and that preparation consists of fewer stages for example 1- cleaning old data, 2- copy new data, 3- prepare area to run builds. My ideal look for that would be a parent stage which is called Prepare environment, and then when pressing on that node it would fork 3 other nodes (which are the 3 stages I've mentioned), and when you select any of the inner stages you'll be able to get the shell scripts and any other pipeline commands you were doing in such stage.
            Hide
            michaelneale Michael Neale added a comment -

            mohamed badran so for now it shows up as a log prefix which helps you a bit - but ideally it would show up as a labelled step (or group of scripts, that are labelled as one thing you can "expand" into?)

            Perhaps we need to draw out what the options could be for this.

            But to me the main thing is you need to label a step, or group together a bunch of steps under a label... I think? Which implies that you really want stages to be the top level things, and when you drill down into them, you see groupings of labelled steps - woudl that be fair to say?

            Show
            michaelneale Michael Neale added a comment - mohamed badran so for now it shows up as a log prefix which helps you a bit - but ideally it would show up as a labelled step (or group of scripts, that are labelled as one thing you can "expand" into?) Perhaps we need to draw out what the options could be for this. But to me the main thing is you need to label a step, or group together a bunch of steps under a label... I think? Which implies that you really want stages to be the top level things, and when you drill down into them, you see groupings of labelled steps - woudl that be fair to say?
            jamesdumay James Dumay made changes -
            Epic Link JENKINS-35750 [ 171713 ]
            Hide
            mbadran mohamed badran added a comment -

            Michael Neale You got it correctly Michael that i need to have label to a step (if possible we can make that label applicable on any kind of step not just a shell script), or group together bunch of steps under a label. This implies what you've mentioned as well that i need to have a top level stage that i can drill down into it to see grouping of labeled steps, or i see in the log a step next to it a label in order to know what is that step doing.

            In my real use case I could have for example like 16 shell scripts running under one stage in parallel to build variant platforms, looking at this in blueocean, or pipeline view would show me 16 shell scripts and that's it which makes it pretty hard for developers to know which build actually failed, unless they go to pipeline steps to know the stage name and they figure out which build actually failed, or they've to do a full log view in order to find out the command that was used for a build.

            Show
            mbadran mohamed badran added a comment - Michael Neale You got it correctly Michael that i need to have label to a step (if possible we can make that label applicable on any kind of step not just a shell script), or group together bunch of steps under a label. This implies what you've mentioned as well that i need to have a top level stage that i can drill down into it to see grouping of labeled steps, or i see in the log a step next to it a label in order to know what is that step doing. In my real use case I could have for example like 16 shell scripts running under one stage in parallel to build variant platforms, looking at this in blueocean, or pipeline view would show me 16 shell scripts and that's it which makes it pretty hard for developers to know which build actually failed, unless they go to pipeline steps to know the stage name and they figure out which build actually failed, or they've to do a full log view in order to find out the command that was used for a build.
            Hide
            michaelneale Michael Neale added a comment - - edited

            mohamed badran well, some slight good news: there is a proposal that more information about scripts is returned so instead of saying "shell script" it can show a snippet of the script itself (just a little bit) or the output if run (that isn't being done as part of blue ocean project though, as deeper changes are needed) - so that helps a bit.

            But to clarify, this is clearer. So you are happy with a stage graph at a high level, up the top, but need ways to group multiple steps into one, and label them (down the bottom). Maybe nested stage is useful for that, but not sure. To me nested stage implies a nesting of the visual graph, but we will see...

            James Dumay perhaps this is a case for labelled steps of some kind?

            Show
            michaelneale Michael Neale added a comment - - edited mohamed badran well, some slight good news: there is a proposal that more information about scripts is returned so instead of saying "shell script" it can show a snippet of the script itself (just a little bit) or the output if run (that isn't being done as part of blue ocean project though, as deeper changes are needed) - so that helps a bit. But to clarify, this is clearer. So you are happy with a stage graph at a high level, up the top, but need ways to group multiple steps into one, and label them (down the bottom). Maybe nested stage is useful for that, but not sure. To me nested stage implies a nesting of the visual graph, but we will see... James Dumay perhaps this is a case for labelled steps of some kind?
            mbadran mohamed badran made changes -
            Attachment Capture.JPG [ 34084 ]
            Hide
            mbadran mohamed badran added a comment -

            Michael Neale The stage graph i think should be enhanced as well to give any indication that the the node actually is formed from multi-stages for example you can have a node with double circles around it to give a visual indicator that something is different in this stage.

            For example in the code I've given earlier if you actually uncomment //stage 'start' it will show in the graph something like this if node "a" and node "b" have double rings around them then this could mean to the user that something is different in that stage. I think to support nested stages maybe when clicking on a node that is formed of multistage you can remove the original high level graph and just zoom in the small graph only so in my case when i press on node "a" that would remove the original graph, and show new graph of "a" as start then "ab", and "xyz" as children for it, and if we've further down stages you can always clean the visualization and show just all the previous nodes as a line formed having multi nodes in there, and only the deepest stage you should start to expand it. Once the user finishes there could be a reset button to reset the graphs to the parent, or they just press on first node. (This is just an proposal, but the simplest would be just a different indicator for a node that is formed of multi stages).

            Regarding the bottom part I'd like in the attached image to see the first shell script under a/ab, and the second shell script under a/xyz so it could be something like this

            steps a/ab
            shell script

            steps a/xyz
            shell script

            These are all thoughts to try to help in how to visualize things but definitely there could be other ways. To me for example i'd want something like
            shell script : (This is platform 1)
            shell script: (This is platform 2)

            So yes i need something either in the visualization part, or in the lower level that would show that nesting in someway and show the grouping, and preferably both parts. But i just need to give me any means of feedback of what's happening.

            Show
            mbadran mohamed badran added a comment - Michael Neale The stage graph i think should be enhanced as well to give any indication that the the node actually is formed from multi-stages for example you can have a node with double circles around it to give a visual indicator that something is different in this stage. For example in the code I've given earlier if you actually uncomment //stage 'start' it will show in the graph something like this if node "a" and node "b" have double rings around them then this could mean to the user that something is different in that stage. I think to support nested stages maybe when clicking on a node that is formed of multistage you can remove the original high level graph and just zoom in the small graph only so in my case when i press on node "a" that would remove the original graph, and show new graph of "a" as start then "ab", and "xyz" as children for it, and if we've further down stages you can always clean the visualization and show just all the previous nodes as a line formed having multi nodes in there, and only the deepest stage you should start to expand it. Once the user finishes there could be a reset button to reset the graphs to the parent, or they just press on first node. (This is just an proposal, but the simplest would be just a different indicator for a node that is formed of multi stages). Regarding the bottom part I'd like in the attached image to see the first shell script under a/ab, and the second shell script under a/xyz so it could be something like this steps a/ab shell script steps a/xyz shell script These are all thoughts to try to help in how to visualize things but definitely there could be other ways. To me for example i'd want something like shell script : (This is platform 1) shell script: (This is platform 2) So yes i need something either in the visualization part, or in the lower level that would show that nesting in someway and show the grouping, and preferably both parts. But i just need to give me any means of feedback of what's happening.
            jamesdumay James Dumay made changes -
            Hide
            jamesdumay James Dumay added a comment -

            One way we could make this work is:

            Where we have a Pipeline like

              stage ("Build") {
                stage ("Inner 1") {
                   ....
                }
                stage ("Inner 2") {
                    ....
                }
              }
            

            Then again I've heard use cases where these become boxes around stages in the pipeline graph or lines connecting stages together within stages.... need to go back to the Pipeline developers and get guidance around what the intended use case was here.

            Show
            jamesdumay James Dumay added a comment - One way we could make this work is: Where we have a Pipeline like stage ( "Build" ) { stage ( "Inner 1" ) { .... } stage ( "Inner 2" ) { .... } } Then again I've heard use cases where these become boxes around stages in the pipeline graph or lines connecting stages together within stages.... need to go back to the Pipeline developers and get guidance around what the intended use case was here.
            Hide
            mbadran mohamed badran added a comment -

            James Dumay and Michael Neale

            Actually that last comment image looks good, and it actually follows one of my suggestions as well so that will be good enhancement. One point as you mentioned in the upper graph to add like any visualization indicator that this is a nested stage as a box, or as anything else.

            Show
            mbadran mohamed badran added a comment - James Dumay and Michael Neale Actually that last comment image looks good, and it actually follows one of my suggestions as well so that will be good enhancement. One point as you mentioned in the upper graph to add like any visualization indicator that this is a nested stage as a box, or as anything else.
            Hide
            jamesdumay James Dumay added a comment -

            Im still hesitant because we have no idea what pipelines like this should look like

            stage ("outer") {
                stage ("inner") {
                   stage ("most inner") {
                      stage ("turtles all the way down") {
                         ...
                      }
                   }
                }
              }
            
            Show
            jamesdumay James Dumay added a comment - Im still hesitant because we have no idea what pipelines like this should look like stage ( " outer " ) { stage ( " inner " ) { stage ( "most inner " ) { stage ( "turtles all the way down" ) { ... } } } }
            Hide
            michaelneale Michael Neale added a comment - - edited

            Right - are there nested stages in a visual stage graph, or are they grouping of steps? if the latter, then there is an assumption that nested stages are represented that way that isn't obvious from the config.

            I don't think anyone has clearly asked for nested stages (given no one can explain what they are meant to be). What has been asked for is a way to label a step, or group steps under a label. nested stages does not achieve this clearly.

            Show
            michaelneale Michael Neale added a comment - - edited Right - are there nested stages in a visual stage graph, or are they grouping of steps? if the latter, then there is an assumption that nested stages are represented that way that isn't obvious from the config. I don't think anyone has clearly asked for nested stages (given no one can explain what they are meant to be). What has been asked for is a way to label a step, or group steps under a label. nested stages does not achieve this clearly.
            Hide
            mbadran mohamed badran added a comment -

            Michael Neale

            Nested stages doesn't do the the label part but it is a workaround, however nested stages could be needed in cases as I've mentioned earlier. Visual graph(I'm talking about Cloudbees one not blueocean upper graph) doesn't understand any nesting each stage will always show each stage as a separate block which is really ugly since you never know that this step is related to anything.

            James Dumay

            Having nested stages could still be expanded in a flat way. This could happen the same way you did it previously but add like a path to it so that would look like:

            Steps Outer
            Inner Stage inner
            Inner Stage inner/most inner

            I believe by this way users will always be able to see things clearly

            Show
            mbadran mohamed badran added a comment - Michael Neale Nested stages doesn't do the the label part but it is a workaround, however nested stages could be needed in cases as I've mentioned earlier. Visual graph(I'm talking about Cloudbees one not blueocean upper graph) doesn't understand any nesting each stage will always show each stage as a separate block which is really ugly since you never know that this step is related to anything. James Dumay Having nested stages could still be expanded in a flat way. This could happen the same way you did it previously but add like a path to it so that would look like: Steps Outer Inner Stage inner Inner Stage inner/most inner I believe by this way users will always be able to see things clearly
            kdawgwilk Kaden Wilkinson made changes -
            Hide
            kdawgwilk Kaden Wilkinson added a comment -

            I think the reason many are wanting to even have nested stages is to help solve this problem:

            We would like to give these shell scripts a more descriptive name but are unable to.

            Show
            kdawgwilk Kaden Wilkinson added a comment - I think the reason many are wanting to even have nested stages is to help solve this problem: We would like to give these shell scripts a more descriptive name but are unable to.
            Hide
            michaelneale Michael Neale added a comment - - edited

            Kaden Wilkinson exactly - that is exactly what I was thinking. Do you want to label those steps explicitly, or if they showed a more helpful description (say, the first N chars of the script) would that help?

            Show
            michaelneale Michael Neale added a comment - - edited Kaden Wilkinson exactly - that is exactly what I was thinking. Do you want to label those steps explicitly, or if they showed a more helpful description (say, the first N chars of the script) would that help?
            Hide
            kdawgwilk Kaden Wilkinson added a comment - - edited

            I would like to have a syntax in the Jenkinfile to be able to say:

            sh([cmd: 'script.sh', name: 'Deploy script']) 
            

            or maybe something more generic that would work for all build steps?

            step('Deploy script') {
                sh 'script.sh'
            }
            
            Show
            kdawgwilk Kaden Wilkinson added a comment - - edited I would like to have a syntax in the Jenkinfile to be able to say: sh([cmd: 'script.sh' , name: 'Deploy script' ]) or maybe something more generic that would work for all build steps? step( 'Deploy script' ) { sh 'script.sh' }
            Hide
            kdawgwilk Kaden Wilkinson added a comment -

            Even the name of the script being called would help eg.

            sh 'script.sh'
            

            if the step was displayed as "script.sh" it would be much better than it is right now

            Show
            kdawgwilk Kaden Wilkinson added a comment - Even the name of the script being called would help eg. sh 'script.sh' if the step was displayed as "script.sh" it would be much better than it is right now
            Hide
            michaelneale Michael Neale added a comment -

            Kaden Wilkinson good to know. Yes showing something meaningful certainly should happen no matter what. Labelling is also very nice.

            Show
            michaelneale Michael Neale added a comment - Kaden Wilkinson good to know. Yes showing something meaningful certainly should happen no matter what. Labelling is also very nice.
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            jamesdumay James Dumay made changes -
            Rank Ranked lower
            Hide
            michaelneale Michael Neale added a comment -

            James Dumay if we get labelled steps (via any means) probably can close this.

            Show
            michaelneale Michael Neale added a comment - James Dumay if we get labelled steps (via any means) probably can close this.
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            Code used:

            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {stage} needs to be revisited.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {stage} needs to be revisited.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            Hide
            jglick Jesse Glick added a comment -

            IMO you should stop showing individual steps at all (display the complete output per block), and just let stage be used to show substructure in the natural way.

            Show
            jglick Jesse Glick added a comment - IMO you should stop showing individual steps at all (display the complete output per block), and just let stage be used to show substructure in the natural way.
            jamesdumay James Dumay made changes -
            Link This issue relates to JENKINS-35836 [ JENKINS-35836 ]
            jamesdumay James Dumay made changes -
            Link This issue relates to JENKINS-35836 [ JENKINS-35836 ]
            Hide
            jamesdumay James Dumay added a comment - - edited

            Alan Wu mentioned on the mailing list he would like to represent a Pipeline like the following:

            Says if I now have the parallel block within a stage called "Build and test" and add in a "deploy" stage after it,
            here is my two cents on how I imagine my use case should look. 
            
                                     / Slave1 -- configure -- build -- test \
              Build and test - Slave2 -- configure -- build -- test - deploy
                                     \ Slave3 -- configure -- build -- test /
            
            Show
            jamesdumay James Dumay added a comment - - edited Alan Wu mentioned on the mailing list he would like to represent a Pipeline like the following: Says if I now have the parallel block within a stage called "Build and test" and add in a "deploy" stage after it, here is my two cents on how I imagine my use case should look. / Slave1 -- configure -- build -- test \ Build and test - Slave2 -- configure -- build -- test - deploy \ Slave3 -- configure -- build -- test /
            Hide
            michaelneale Michael Neale added a comment -

            James Dumay so would that could be something like:

            stage('build and test') {
               parallel('Slave1' : {
                   stage('configure') {
                   }
                   stage('build') {
                   }
                   stage('test') {
                   }
               }
            }
            
            stage('deploy') {
            
            }
            

            Given that everything needs to happen under a parent stage. The nested stages in parallel are the different groups of steps for configure etc.

            Show
            michaelneale Michael Neale added a comment - James Dumay so would that could be something like: stage('build and test') { parallel('Slave1' : { stage('configure') { } stage('build') { } stage('test') { } } } stage('deploy') { } Given that everything needs to happen under a parent stage. The nested stages in parallel are the different groups of steps for configure etc.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Michael Neale yes I'd say that syntax looks about right for what Alan Wu is proposing. Though it gets ugly with multiple parallel branches as you have to repeat the stage structure. Perhaps we need to come up with a specialised syntax to support this structure? Worth exploring IMO cc Andrew Bayer

            Show
            jamesdumay James Dumay added a comment - - edited Michael Neale yes I'd say that syntax looks about right for what Alan Wu is proposing. Though it gets ugly with multiple parallel branches as you have to repeat the stage structure. Perhaps we need to come up with a specialised syntax to support this structure? Worth exploring IMO cc Andrew Bayer
            Hide
            predivo Patrick Redivo added a comment -

            Hi
            I am very concerned by the same issue:
            for me in BlueOcean, only stages, but all stages, should be visible with Green balls.
            branch names (SlaveX) should be a kind of label above the arrow but not a green ball by itself.
            Therefore, if a branch has no label, then you would see just a line (with its label).
            If a branch has a serie of N stages, then you would see those stages in that line (with its label), as Alan Wu proposed, I guess.

            Furthermore, if a stage within a branch consists of a new parallel block or a serie of parallel blocks, then all these sub-stages/branches should be seen as well in their relative branch in the same way.
            I know there is a question regarding how deep to display such a kind of nested graph, maybe could you add a 'depth' parameter in the page (default: 2?) and display the graph accordingly?
            Thanks
            Regards
            Patrick

            Show
            predivo Patrick Redivo added a comment - Hi I am very concerned by the same issue: for me in BlueOcean, only stages, but all stages, should be visible with Green balls. branch names (SlaveX) should be a kind of label above the arrow but not a green ball by itself. Therefore, if a branch has no label, then you would see just a line (with its label). If a branch has a serie of N stages, then you would see those stages in that line (with its label), as Alan Wu proposed, I guess. Furthermore, if a stage within a branch consists of a new parallel block or a serie of parallel blocks, then all these sub-stages/branches should be seen as well in their relative branch in the same way. I know there is a question regarding how deep to display such a kind of nested graph, maybe could you add a 'depth' parameter in the page (default: 2?) and display the graph accordingly? Thanks Regards Patrick
            Hide
            alanwu Alan Wu added a comment -

            Regarding how deep the nested graph should show, I think making each of the dot displaying stage collapsible will be quite helpful. Clicking on a parent stage will open up the next level of children stages.

            Show
            alanwu Alan Wu added a comment - Regarding how deep the nested graph should show, I think making each of the dot displaying stage collapsible will be quite helpful. Clicking on a parent stage will open up the next level of children stages.
            Hide
            jamesdumay James Dumay added a comment -

            Alan Wu we may not allow for infinite nesting in the way that you have described (comes with a lot of design challenges). It's more likely that declarative pipeline will be updated to support a single "sub-stage" under a top level stage.

            Show
            jamesdumay James Dumay added a comment - Alan Wu we may not allow for infinite nesting in the way that you have described (comes with a lot of design challenges). It's more likely that declarative pipeline will be updated to support a single "sub-stage" under a top level stage.
            Hide
            alanwu Alan Wu added a comment -

            James Dumay Sure, I can imagine the difficulties of dealing with layers upon layers of stages.

            I also agree with you that a specialised syntax to allow multiple parallel brenches to be executed in the same manner without having to copy and paste blocks of code will be very useful.

            Show
            alanwu Alan Wu added a comment - James Dumay Sure, I can imagine the difficulties of dealing with layers upon layers of stages. I also agree with you that a specialised syntax to allow multiple parallel brenches to be executed in the same manner without having to copy and paste blocks of code will be very useful.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Alan Wu thats great. FWIW, we want to build the best tool for the job rather than a bunch of round shape things and then fitting them to square holes. Just takes a but longer but I guarantee will be worth the wait

            Show
            jamesdumay James Dumay added a comment - - edited Alan Wu thats great. FWIW, we want to build the best tool for the job rather than a bunch of round shape things and then fitting them to square holes. Just takes a but longer but I guarantee will be worth the wait
            Hide
            jamesdumay James Dumay added a comment - - edited

            Patrick Redivo unfortunately it isn't that simple. The visualization isn't a generic graph rendering that can just render things like the way you described without serious changes. Without making any promises a nesting feature is on our radar after the release of Blue Ocean 1.0 early next year.

            Show
            jamesdumay James Dumay added a comment - - edited Patrick Redivo unfortunately it isn't that simple. The visualization isn't a generic graph rendering that can just render things like the way you described without serious changes. Without making any promises a nesting feature is on our radar after the release of Blue Ocean 1.0 early next year.
            jamesdumay James Dumay made changes -
            Link This issue relates to JENKINS-39464 [ JENKINS-39464 ]
            jamesdumay James Dumay made changes -
            Link This issue relates to JENKINS-40619 [ JENKINS-40619 ]
            jamesdumay James Dumay made changes -
            Link This issue relates to JENKINS-40619 [ JENKINS-40619 ]
            jamesdumay James Dumay made changes -
            Link This issue is duplicated by JENKINS-40619 [ JENKINS-40619 ]
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            jamesdumay James Dumay made changes -
            Rank Ranked lower
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "Jenkins/Jenkins tried to use a matrix style parallel (Web Link)" [ 15226 ]
            Hide
            teilo James Nord added a comment -

            I thought this was the reason that stage was made into a block step?

            Show
            teilo James Nord added a comment - I thought this was the reason that stage was made into a block step?
            Hide
            jamesdumay James Dumay added a comment -

            James Nord yes but we failed to figure out how it should be visualized and if it can be visualized. There seems to be different ways users expect the visualization to be displayed and the clarity of the visualization decreases the deeper the stages are nested. As described in the description of this ticket we need to solve both of these issues before we can make it work. I am sure we could implement something but without rigor applied I am not certain it will be good.

            Show
            jamesdumay James Dumay added a comment - James Nord yes but we failed to figure out how it should be visualized and if it can be visualized. There seems to be different ways users expect the visualization to be displayed and the clarity of the visualization decreases the deeper the stages are nested. As described in the description of this ticket we need to solve both of these issues before we can make it work. I am sure we could implement something but without rigor applied I am not certain it will be good .
            Hide
            kaulk Kuber Kaul added a comment -

            Any resolution dates on this ? I have the same issue with the nested stages not being displayed at all.

            Show
            kaulk Kuber Kaul added a comment - Any resolution dates on this ? I have the same issue with the nested stages not being displayed at all.
            Hide
            jamesdumay James Dumay added a comment -

            Kuber Kaul as the issue description says we currently do not have a resolution for this just yet. Make sure you watch this ticket for updates.

            Show
            jamesdumay James Dumay added a comment - Kuber Kaul as the issue description says we currently do not have a resolution for this just yet. Make sure you watch this ticket for updates.
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            Hide
            kaulk Kuber Kaul added a comment -

            got it, thanks.

            Show
            kaulk Kuber Kaul added a comment - got it, thanks.
            michaelneale Michael Neale made changes -
            Link This issue blocks JENKINS-35836 [ JENKINS-35836 ]
            ileitch Ian Leitch made changes -
            Attachment PipelineGraphProposal.png [ 35539 ]
            Hide
            ileitch Ian Leitch added a comment - - edited

            Proposal

            This proposal takes the position that the vast majority of uses cases can be adequately represented in the graph by only 2 levels, with deeper nesting merely indicated on the graph. Nesting beyond the 2nd level defaults the list view with indentation indicating nesting depth.

            Representing more than 2 levels in the graph would lead to complex hierarchies that would detract from the role of the graph as an overview.

            1. New 'parallel_stage' command (I'm not proposing this exact naming, just the concept)

            • Acts like the existing 'stage' command, however can be executed at the same time as other 'parallel_stage' regardless of nesting level
            • In this design, all 18 'parallel_stage' of the "Test" stage containing a 'node' are added to the build queue simultaneously. You can think of this a bit like a matrix, however without fixed axes.

            2. 'stage' and 'parallel_stage' can be nested arbitrarily

            • The graph only shows the 1st and 2nd levels
            • If all stages at the 2nd level do not themselves contain further stages, they are presented as a vertical linear chain (e.g Build stage in design)
            • If any of the stages at the 2nd level contain nested stages, then they are represented as siblings (e.g Test stage in design)
            • Stages at the 3rd level are indicated in the graph by the number of immediate descendants on the parent node (e.g "Functional (iPhone 7 10.2)" in the Build stage).
            • Stages at the 3rd level and beyond are presented in the list view, using indentation to indicate nesting level
            • Neither can be called within a 'node' block
            • Not sure what desired behavior should be if a 'stage' is nested within a 'parallel_stage', perhaps only 'parallel_stage' can be nested within another 'parallel_stage'?

            3. The existing 'parallel' command is superseded by multiple `parallel_stage` and can be deprecated

            stage("Lint") { node { sh "lint" }  }
            
            stage("Build") {
              parallel_stage("Shared") { node { sh "build" } }
              parallel_stage("App 1") { node { sh "build" } }
              parallel_stage("App 2") { node { sh "build" } }
            }
            
            stage("Test") {
              parallel_stage("Shared") {
                parallel_stage("Unit (9.3)") { node { sh "test" } }
                parallel_stage("Unit (10.2)") { node { sh "test" } }
                parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
              }
            
              parallel_stage("App 1") {
                parallel_stage("Unit (9.3)") { node { sh "test" } }
                parallel_stage("Unit (10.2)") { node { sh "test" } }
                parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
                parallel_stage("Functional (iPhone 7 10.2)") {
                  parallel_stage("Batch 1") { node { sh "test" } }
                  parallel_stage("Batch 2" { node { sh "test" } }
                  parallel_stage("Batch 3") { node { sh "test" } }
                }
              }
            
              parallel_stage("App 2") {
                parallel_stage("Unit (9.3)") { node { sh "test" } }
                parallel_stage("Unit (10.2)") { node { sh "test" } }
                parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
                parallel_stage("Functional (iPhone 7 10.2)") {
                  parallel_stage("Batch 1") { node { sh "test" } }
                  parallel_stage("Batch 2" { node { sh "test" } }
                  parallel_stage("Batch 3") { node { sh "test" } }
                }
                parallel_stage("Functional (iPad Air 2 10.2)") { node { sh "test" } }
                parallel_stage("Functional (iPad Pro 10.2)") { node { sh "test" } }
              }
            }
            
            stage("Collect") { node { sh "collect" }  }
            
            Show
            ileitch Ian Leitch added a comment - - edited Proposal This proposal takes the position that the vast majority of uses cases can be adequately represented in the graph by only 2 levels, with deeper nesting merely indicated on the graph. Nesting beyond the 2nd level defaults the list view with indentation indicating nesting depth. Representing more than 2 levels in the graph would lead to complex hierarchies that would detract from the role of the graph as an overview. 1. New 'parallel_stage' command (I'm not proposing this exact naming, just the concept) Acts like the existing 'stage' command, however can be executed at the same time as other 'parallel_stage' regardless of nesting level In this design, all 18 'parallel_stage' of the "Test" stage containing a 'node' are added to the build queue simultaneously. You can think of this a bit like a matrix, however without fixed axes. 2. 'stage' and 'parallel_stage' can be nested arbitrarily The graph only shows the 1st and 2nd levels If all stages at the 2nd level do not themselves contain further stages, they are presented as a vertical linear chain (e.g Build stage in design) If any of the stages at the 2nd level contain nested stages, then they are represented as siblings (e.g Test stage in design) Stages at the 3rd level are indicated in the graph by the number of immediate descendants on the parent node (e.g "Functional (iPhone 7 10.2)" in the Build stage). Stages at the 3rd level and beyond are presented in the list view, using indentation to indicate nesting level Neither can be called within a 'node' block Not sure what desired behavior should be if a 'stage' is nested within a 'parallel_stage', perhaps only 'parallel_stage' can be nested within another 'parallel_stage'? 3. The existing 'parallel' command is superseded by multiple `parallel_stage` and can be deprecated stage( "Lint" ) { node { sh "lint" } } stage( "Build" ) { parallel_stage( "Shared" ) { node { sh "build" } } parallel_stage( "App 1" ) { node { sh "build" } } parallel_stage( "App 2" ) { node { sh "build" } } } stage( "Test" ) { parallel_stage( "Shared" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } } parallel_stage( "App 1" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPhone 7 10.2)" ) { parallel_stage( "Batch 1" ) { node { sh "test" } } parallel_stage( "Batch 2" { node { sh "test" } } parallel_stage( "Batch 3" ) { node { sh "test" } } } } parallel_stage( "App 2" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPhone 7 10.2)" ) { parallel_stage( "Batch 1" ) { node { sh "test" } } parallel_stage( "Batch 2" { node { sh "test" } } parallel_stage( "Batch 3" ) { node { sh "test" } } } parallel_stage( "Functional (iPad Air 2 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPad Pro 10.2)" ) { node { sh "test" } } } } stage( "Collect" ) { node { sh "collect" } }
            Hide
            alanwu Alan Wu added a comment -

            Ian Leitch Thanks a lot for the update, can you please comment on how the following case can be represent with your proposal?

            stage('build and test') {
            parallel('Slave1' : {
            stage('configure') {
            }
            stage('build') {
            }
            stage('test') {
            }
            }
            parallel('Slave2' : {
            stage('configure') {
            }
            stage('build') {
            }
            stage('test') {
            }
            }
            }

            If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

            Show
            alanwu Alan Wu added a comment - Ian Leitch Thanks a lot for the update, can you please comment on how the following case can be represent with your proposal? stage('build and test') { parallel('Slave1' : { stage('configure') { } stage('build') { } stage('test') { } } parallel('Slave2' : { stage('configure') { } stage('build') { } stage('test') { } } } If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?
            Hide
            jamesdumay James Dumay added a comment -

            If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

            Alan Wu yes, that is correct - we can't display those right now.

            Show
            jamesdumay James Dumay added a comment - If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired? Alan Wu yes, that is correct - we can't display those right now.
            Hide
            ileitch Ian Leitch added a comment -

            Alan Wu James Dumay I'd still love for some kind of `named_step`, which has nothing to do without how the graph is shown, but for naming/grouping the individual steps taken within each stage.

            Show
            ileitch Ian Leitch added a comment - Alan Wu James Dumay I'd still love for some kind of `named_step`, which has nothing to do without how the graph is shown, but for naming/grouping the individual steps taken within each stage.
            Hide
            alanwu Alan Wu added a comment -

            Ian Leitch That's great, it will be very useful to have named_step. Thank you.

            Show
            alanwu Alan Wu added a comment - Ian Leitch That's great, it will be very useful to have named_step. Thank you.
            jamesdumay James Dumay made changes -
            Link This issue is duplicated by JENKINS-41781 [ JENKINS-41781 ]
            rhutchison Ryan Hutchison made changes -
            Link This issue is related to JENKINS-41156 [ JENKINS-41156 ]
            Hide
            jglick Jesse Glick added a comment -

            FWIW here is an example of a build where information about different substages has been omitted. It looks worse during the build, when some of the nested stages are temporarily displayed, but in a confused order.

            Show
            jglick Jesse Glick added a comment - FWIW here is an example of a build where information about different substages has been omitted. It looks worse during the build, when some of the nested stages are temporarily displayed, but in a confused order.
            Hide
            michaelneale Michael Neale added a comment -

            Jesse Glick yes it should be flattened out sensibly (there are a few things going on that should eventually fix that) so it is less confusing even without showing it as explicitly nested

            Show
            michaelneale Michael Neale added a comment - Jesse Glick yes it should be flattened out sensibly (there are a few things going on that should eventually fix that) so it is less confusing even without showing it as explicitly nested
            jamesdumay James Dumay made changes -
            Link This issue blocks JENKINS-42062 [ JENKINS-42062 ]
            michaelneale Michael Neale made changes -
            Link This issue relates to JENKINS-41205 [ JENKINS-41205 ]
            Hide
            michaelneale Michael Neale added a comment -

            There has been a fair bit of off-list discussion on this, the vast majority seem to be satisfied by: 

             

            • One level of stages shown in parallel streams of execution
            • some first class support for matrix of combinations (in paralllel)

             

            I haven't heard a strong case for nested parallel "circles" inside a non parallel stage (in that case people want to label a step or a group of steps, which is different to having a different circle/node during a parallel stream of execution). 

            Show
            michaelneale Michael Neale added a comment - There has been a fair bit of off-list discussion on this, the vast majority seem to be satisfied by:    One level of stages shown in parallel streams of execution some first class support for matrix of combinations (in paralllel)   I haven't heard a strong case for nested parallel "circles" inside a non parallel stage (in that case people want to label a step or a group of steps, which is different to having a different circle/node during a parallel stream of execution). 
            b_dean Ben Dean made changes -
            Attachment example.png [ 36599 ]
            b_dean Ben Dean made changes -
            Attachment pipeline_mockup.png [ 36602 ]
            Hide
            b_dean Ben Dean added a comment - - edited

            Here's a real world example of why one might have parallel stages and with some of those stages needing parallel "circles". We have an application that has these basic stages:

            1. Clone SCM
            2. Build
            3. Build documentation on Windows slave (slow)
            4. Custom static analysis tool (slow)
            5. Multiple Test suites
            6. Archive build artifacts and test results

            Steps 1-3 can all run in parallel and since 2 and 3 both take a long time, it's actually desirable for them to be running while we're doing all our various suites in parallel. Here's what the ideal Jenkinsfile would look like (IMHO):

            Jenkinsfile
            stage('Clone') {
                checkout scm
            }
            
            testSuites = [ 'suite1', 'suite2', 'suite3' ]
            
            parallel([
                buildAndTest: {
                    stage('Build') {
                        node('linux') {
                            sh 'make'
                            stash includes: 'pkg/**/*', name: 'build-artifacts'
                        }
                    }
                    stage('Test') {
                        testSteps = [:]
                        for (suite in testSuites) {
                            testSteps[suite] = {
                                node('linux') {
                                    unstash 'build-artifacts'
                                    sh "run_tests $suite"
                                    stash includes: 'test_results/**/*.xml', name: "test-$suite"
                                }
                            }
                        }
                        parallel(testSteps)
                    }
                },
                docs: {
                    stage('Documentation') {
                        node('windows') {
                            bat 'build_docs.exe'
                        }
                    }
                },
                staticAnalysis: {
                    stage('Static Analysis') {
                        node('linux') {
                            sh 'run_static_analysis'
                        }
                    }
                }
            ])
            
            stage('Archive') {
                unstash 'build-artifacts'
                for (suite in testSuites) {
                    unstash "test-$suite"
                }
                archive 'pkg/**/*'
                junit 'test_results/**/*.xml'
            }
            

            I'm not exactly sure about where that checkout scm should go with all the parallel stuff, I haven't actually run this particular thing yet. Also while writing this I wasn't really sure what to name the parallel sections at the top level, since I really want them to get their names from the nested stages.

            As for how this would look in the UI (only really thinking of Blue Ocean), I would want it to look how this graphviz DAG would render:

            example.dot
            digraph {
              graph [rankdir=LR]
            
              Clone -> Build
              Clone -> Documentation
              Clone -> "Static Analysis"
            
              Documentation -> Archive
              "Static Analysis" -> Archive
            
              Build -> "Test suite1"
              Build -> "Test suite2"
              Build -> "Test suite3"
            
              "Test suite1" -> Archive
              "Test suite2" -> Archive
              "Test suite3" -> Archive
            }
            

            Here's what I was thinking it might look like in Blue Ocean

            The workaround I have planned for this particular build is to have the docs and static analysis happen in other builds that I then have to wait for them to finish. Something like this:

            Jenkinsfile
            stage('Clone') {
                checkout scm
            }
            
            stage('Build') {
                parallel([
                    build: {
                        node('linux') {
                            sh 'make'
                            stash includes: 'pkg/**/*', name: 'build'
                        }
                    },
                    docs: {
                        build job: 'docs', wait: false
                    },
                    staticAnalysis: {
                        build job: 'staticAnalysis', wait: false
                    }
                ])
            }
            
            testSuites = [ 'suite1', 'suite2', 'suite3' ]
            
            stage('Test') {
                testSteps = [:]
                for (suite in testSuites) {
                    testSteps[suite] = {
                        node('linux') {
                            unstash 'build'
                            sh "run_tests $suite"
                            stash includes: 'test_results/**/*.xml', name: "test-$suite"
                        }
                    }
                }
                parallel(testSteps)
            }
            
            // method to lookup the docs and staticAnalysis jobs we started and see if they're done
            @NonCPS
            def findSomeBuild(jobName)
                // do Jenkins groovy stuff
                return correctBuildNumber || null
            end
            
            stage('Archive') {
                unstash 'build'
            
                def docsBuildNumber = null    
                waitUntil {
                    docsBuildNumber = findSomeBuild('docs')
                }
                
                def staticAnalysisBuildNumber = null
                waitUntil {
                    staticAnalysisBuildNumber = findSomeBuild('staticAnalysis')
                }
                
                step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'docs', selector: [$class: 'SpecificBuildSelector', buildNumber: docsBuildNumber]])
                step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'staticAnalysis', selector: [$class: 'SpecificBuildSelector', buildNumber: staticAnalysisBuildNumber]])
            
                unstash 'documentation'
                unstash 'static_analysis'
                for (suite in testSuites) {
                    unstash "test-$suite"
                }
                archive 'pkg/**/*'
                junit 'test_results/**/*.xml'
            }
            
            Show
            b_dean Ben Dean added a comment - - edited Here's a real world example of why one might have parallel stages and with some of those stages needing parallel "circles". We have an application that has these basic stages: Clone SCM Build Build documentation on Windows slave (slow) Custom static analysis tool (slow) Multiple Test suites Archive build artifacts and test results Steps 1-3 can all run in parallel and since 2 and 3 both take a long time, it's actually desirable for them to be running while we're doing all our various suites in parallel. Here's what the ideal Jenkinsfile would look like (IMHO): Jenkinsfile stage( 'Clone' ) { checkout scm } testSuites = [ 'suite1' , 'suite2' , 'suite3' ] parallel([ buildAndTest: { stage( 'Build' ) { node( 'linux' ) { sh 'make' stash includes: 'pkg /**/ *' , name: 'build-artifacts' } } stage( 'Test' ) { testSteps = [:] for (suite in testSuites) { testSteps[suite] = { node( 'linux' ) { unstash 'build-artifacts' sh "run_tests $suite" stash includes: 'test_results /**/ *.xml' , name: "test-$suite" } } } parallel(testSteps) } }, docs: { stage( 'Documentation' ) { node( 'windows' ) { bat 'build_docs.exe' } } }, staticAnalysis: { stage( 'Static Analysis' ) { node( 'linux' ) { sh 'run_static_analysis' } } } ]) stage( 'Archive' ) { unstash 'build-artifacts' for (suite in testSuites) { unstash "test-$suite" } archive 'pkg /**/ *' junit 'test_results /**/ *.xml' } I'm not exactly sure about where that checkout scm should go with all the parallel stuff, I haven't actually run this particular thing yet. Also while writing this I wasn't really sure what to name the parallel sections at the top level, since I really want them to get their names from the nested stages. As for how this would look in the UI (only really thinking of Blue Ocean), I would want it to look how this graphviz DAG would render: example.dot digraph { graph [rankdir=LR] Clone -> Build Clone -> Documentation Clone -> "Static Analysis" Documentation -> Archive "Static Analysis" -> Archive Build -> "Test suite1" Build -> "Test suite2" Build -> "Test suite3" "Test suite1" -> Archive "Test suite2" -> Archive "Test suite3" -> Archive } Here's what I was thinking it might look like in Blue Ocean The workaround I have planned for this particular build is to have the docs and static analysis happen in other builds that I then have to wait for them to finish. Something like this: Jenkinsfile stage( 'Clone' ) { checkout scm } stage( 'Build' ) { parallel([ build: { node( 'linux' ) { sh 'make' stash includes: 'pkg /**/ *' , name: 'build' } }, docs: { build job: 'docs' , wait: false }, staticAnalysis: { build job: 'staticAnalysis' , wait: false } ]) } testSuites = [ 'suite1' , 'suite2' , 'suite3' ] stage( 'Test' ) { testSteps = [:] for (suite in testSuites) { testSteps[suite] = { node( 'linux' ) { unstash 'build' sh "run_tests $suite" stash includes: 'test_results /**/ *.xml' , name: "test-$suite" } } } parallel(testSteps) } // method to lookup the docs and staticAnalysis jobs we started and see if they're done @NonCPS def findSomeBuild(jobName) // do Jenkins groovy stuff return correctBuildNumber || null end stage( 'Archive' ) { unstash 'build' def docsBuildNumber = null waitUntil { docsBuildNumber = findSomeBuild( 'docs' ) } def staticAnalysisBuildNumber = null waitUntil { staticAnalysisBuildNumber = findSomeBuild( 'staticAnalysis' ) } step([$class: 'CopyArtifact' , filter: 'pkg /**/ *' , projectName: 'docs' , selector: [$class: 'SpecificBuildSelector' , buildNumber: docsBuildNumber]]) step([$class: 'CopyArtifact' , filter: 'pkg /**/ *' , projectName: 'staticAnalysis' , selector: [$class: 'SpecificBuildSelector' , buildNumber: staticAnalysisBuildNumber]]) unstash 'documentation' unstash 'static_analysis' for (suite in testSuites) { unstash "test-$suite" } archive 'pkg /**/ *' junit 'test_results /**/ *.xml' }
            Hide
            gillet Laurent Gillet added a comment - - edited

            @Ben Dean: I totally agree with your proposition!

            We have a similar need. Currently, we are using FreestyleJob and DeliveryPipeline plugin to handle it. I'm looking forward to having such feature implemented to switch to JenkinsFile and BlueOcean.

            Show
            gillet Laurent Gillet added a comment - - edited @ Ben Dean : I totally agree with your proposition! We have a similar need. Currently, we are using FreestyleJob and DeliveryPipeline plugin to handle it. I'm looking forward to having such feature implemented to switch to JenkinsFile and BlueOcean.
            Hide
            ikari7789 Andrew Miller added a comment -

            May I suggest the following as a means to represent nested stages visually?

            Display the top-most stages and consider the result of the children stages as their success/failure. Additionally, to view the nested stages, you could then click on a stage in the display, which would "drill-down" and show the sub-stages within that stage, like a zoom of sorts.

            This seems it would allow for an infinite level of nesting without creating a drastic overhead in display logic.

            I know it doesn't give a 100% overhead of all tasks simultaneously, but it would at least be some kind of a middle-road in my opinion.

            Show
            ikari7789 Andrew Miller added a comment - May I suggest the following as a means to represent nested stages visually? Display the top-most stages and consider the result of the children stages as their success/failure. Additionally, to view the nested stages, you could then click on a stage in the display, which would "drill-down" and show the sub-stages within that stage, like a zoom of sorts. This seems it would allow for an infinite level of nesting without creating a drastic overhead in display logic. I know it doesn't give a 100% overhead of all tasks simultaneously, but it would at least be some kind of a middle-road in my opinion.
            Hide
            jamesdumay James Dumay added a comment -

            Andrew Miller could you draw a picture and upload it here please?

            Show
            jamesdumay James Dumay added a comment - Andrew Miller could you draw a picture and upload it here please?
            ikari7789 Andrew Miller made changes -
            Attachment image-2017-04-12-16-16-17-144.png [ 37113 ]
            ikari7789 Andrew Miller made changes -
            Attachment image-2017-04-12-16-17-38-285.png [ 37114 ]
            Hide
            ikari7789 Andrew Miller added a comment -

            Sorry for my lack of artistic ability, but I've tried to portray what I'm thinking in my mind.

             

            So at the top layer, you'd be presented with the normal staged output for the pipeline, if any stages have nested stages, it would display a zoom-in button or something (this is obviously not final).

             

            When you click the zoom button, it will then display the stage(s) within that stage (in a single level increment):

             

            This process could basically allow for an infinite expansion inwards.

            The red/green status would be determined by the final result of all children stages.

            Show
            ikari7789 Andrew Miller added a comment - Sorry for my lack of artistic ability, but I've tried to portray what I'm thinking in my mind.   So at the top layer, you'd be presented with the normal staged output for the pipeline, if any stages have nested stages, it would display a zoom-in button or something (this is obviously not final).   When you click the zoom button, it will then display the stage(s) within that stage (in a single level increment):   This process could basically allow for an infinite expansion inwards. The red/green status would be determined by the final result of all children stages.
            Hide
            praverma Prateek VERMA added a comment -

            Please look into this issue as this will give neater and intuitive UI for debugging purpose where we can see separate logs stage wise.

            In my opinion, syntax given in Example2 should be the correct syntax to visualize stages in parallel and also the word "Parallel" can be removed from parallel step and may be we can show Labels there instead over the parallel steps with stage names for each steps.

            Lable foo
            ---O--O--O-----------

               s1    s2    s3  
             
            Lable bar

            ---O--O--O-
            s1 s2 s3

            Show
            praverma Prateek VERMA added a comment - Please look into this issue as this will give neater and intuitive UI for debugging purpose where we can see separate logs stage wise. In my opinion, syntax given in Example2 should be the correct syntax to visualize stages in parallel and also the word "Parallel" can be removed from parallel step and may be we can show Labels there instead over the parallel steps with stage names for each steps. Lable foo --- O -- O -- O -----------    s1    s2    s3     Lable bar --- O -- O -- O - s1 s2 s3
            praverma Prateek VERMA made changes -
            Assignee Jesse Glick [ jglick ]
            michaelneale Michael Neale made changes -
            Assignee Jesse Glick [ jglick ]
            Hide
            praverma Prateek VERMA added a comment -

            Above illustration is not looking good in visual mode, please ignore the table structure that got created. Stages for each label are in parallel with label name over stages of a particular line of execution.

            Show
            praverma Prateek VERMA added a comment - Above illustration is not looking good in visual mode, please ignore the table structure that got created. Stages for each label are in parallel with label name over stages of a particular line of execution.
            praverma Prateek VERMA made changes -
            Attachment Proposed Parallel Stages.JPG [ 37115 ]
            Hide
            praverma Prateek VERMA added a comment -

            Please refer to "Proposed Parallel Stages.JPG" attached here.

            Show
            praverma Prateek VERMA added a comment - Please refer to "Proposed Parallel Stages.JPG" attached here.
            fiztlen ben powers made changes -
            Attachment alternate zoom in.png [ 37120 ]
            Hide
            fiztlen ben powers added a comment -

            I like Andrew's idea, perhaps simply represent the sub-pipeline as a pipeline in itself? Continuing to re-use existing resources

            As he mentioned, this should provide a UI framework to support additional nesting.

            Here's a sketch that might help Andrew feel better about his artwork  Note the continuation from & to the rest of the build, perhaps add the previous & next step names and/or a better location for zoom out icons.

            Show
            fiztlen ben powers added a comment - I like Andrew's idea, perhaps simply represent the sub-pipeline as a pipeline in itself? Continuing to re-use existing resources As he mentioned, this should provide a UI framework to support additional nesting. Here's a sketch that might help Andrew feel better about his artwork   Note the continuation from & to the rest of the build, perhaps add the previous & next step names and/or a better location for zoom out icons.
            tusharmahale Tushar Mahale made changes -
            Assignee Tushar Mahale [ tusharmahale ]
            jamesdumay James Dumay made changes -
            Assignee Tushar Mahale [ tusharmahale ]
            Hide
            jeanmertz Jean Mertz added a comment -

            I agree, "zooming" into a nested stage seems like the most sensible and scalable solution, with the best possible user experience for all potential situations.

            It also allows for re-usability of existing designs/assets, as this would be an addition on top of the existing way of working, instead of a re-thinking of the current way of working.

            Show
            jeanmertz Jean Mertz added a comment - I agree, "zooming" into a nested stage seems like the most sensible and scalable solution, with the best possible user experience for all potential situations. It also allows for re-usability of existing designs/assets, as this would be an addition on top of the existing way of working, instead of a re-thinking of the current way of working.
            jamesdumay James Dumay made changes -
            Epic Link JENKINS-35750 [ 171713 ] JENKINS-43953 [ 181485 ]
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey|titleColor=white}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey|titleColor=white}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=#3878de|titleColor=white}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jamesdumay James Dumay made changes -
            Description {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=#3878de|titleColor=white}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jguigui Jean-Paul G made changes -
            Link This issue is related to JENKINS-44820 [ JENKINS-44820 ]
            jamesdumay James Dumay made changes -
            Link This issue is duplicated by JENKINS-45664 [ JENKINS-45664 ]
            macdrega Joerg Schwaerzler made changes -
            Attachment image-2017-08-14-10-29-55-373.png [ 39258 ]
            macdrega Joerg Schwaerzler made changes -
            Attachment image-2017-08-14-10-34-10-452.png [ 39259 ]
            Hide
            macdrega Joerg Schwaerzler added a comment - - edited

            How about implementing a small improvement first, like only one level of stages inside parallel blocks, e.g.:

            stage ('Build') {
                // Build something
            }
            parallel('Test1' : {
                stage('JUnit') {
                }
                stage('Firefox') {
                }
            }, 'Test2' : {
                stage('DBUnit') {
                }
                stage('Edge') {
                }
            }, 'Test3' : {
                stage('Jasmine') {
                }
                stage('Safari') {
                }    
            }
            stage ('Dev') {
            }
            
            

             

            Of course it'd be nice if the parallel block's names would be displayed somehow - but most important to me would be just the possibility to somehow split the different parallel blocks into smaller junks using something like stages.

             

            Show
            macdrega Joerg Schwaerzler added a comment - - edited How about implementing a small improvement first, like only one level of stages inside parallel blocks, e.g.: stage ( 'Build' ) { // Build something } parallel( 'Test1' : { stage( 'JUnit' ) { } stage( 'Firefox' ) { } }, 'Test2' : { stage( 'DBUnit' ) { } stage( 'Edge' ) { } }, 'Test3' : { stage( 'Jasmine' ) { } stage( 'Safari' ) { } } stage ( 'Dev' ) { }   Of course it'd be nice if the parallel block's names would be displayed somehow - but most important to me would be just the possibility to somehow split the different parallel blocks into smaller junks using something like stages.  
            Hide
            michaelneale Michael Neale added a comment -

            Joerg Schwaerzler that I think is what most people agree on as some kind of next step for this. 

            Show
            michaelneale Michael Neale added a comment - Joerg Schwaerzler that I think is what most people agree on as some kind of next step for this. 
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Great than, was not sure about that.

            Show
            macdrega Joerg Schwaerzler added a comment - Great than, was not sure about that.
            macdrega Joerg Schwaerzler made changes -
            Attachment image-2017-08-14-10-29-55-373.png [ 39258 ]
            Hide
            manschwetus Florian Manschwetus added a comment - - edited

            Joerg Schwaerzler +1, no more words on this needed from my side.

             

            Hm, the one you removed, was even more my favorite. To visualize enclosing Steps, some kind of background bubble could be used and if a pipeline intermixes nested stages between parallel splits, then just draw the pipeline as far as it is done and identify the current steps by there name to display an estimate for each.

            Show
            manschwetus Florian Manschwetus added a comment - - edited Joerg Schwaerzler +1, no more words on this needed from my side.   Hm, the one you removed, was even more my favorite. To visualize enclosing Steps, some kind of background bubble could be used and if a pipeline intermixes nested stages between parallel splits, then just draw the pipeline as far as it is done and identify the current steps by there name to display an estimate for each.
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Florian Manschwetus

            Sorry for that - just wanted to be crystal clear about what I think should be the next step.

            Leaving out any possibility to have nested stages at all probably would make it easier to at least get a first shot for partitioning a parallel block into smaller parts.

            But of course on the long run I'd be happy to see if we could also put the parallel block itself inside a stage, and possibly even have stages in stages in parallel blocks which can have parallel blocks

            However for the time being I'd suggest to concentrate on a hopefully tiny additional feature for visualizing stages inside parallel blocks. Probably it'll even be easier to discuss about the remaining features once this is available so everybody is able to validate this against their use cases.

            Show
            macdrega Joerg Schwaerzler added a comment - Florian Manschwetus Sorry for that - just wanted to be crystal clear about what I think should be the next step. Leaving out any possibility to have nested stages at all probably would make it easier to at least get a first shot for partitioning a parallel block into smaller parts. But of course on the long run I'd be happy to see if we could also put the parallel block itself inside a stage, and possibly even have stages in stages in parallel blocks which can have parallel blocks However for the time being I'd suggest to concentrate on a hopefully tiny additional feature for visualizing stages inside parallel blocks. Probably it'll even be easier to discuss about the remaining features once this is available so everybody is able to validate this against their use cases.
            jamesdumay James Dumay made changes -
            Link This issue is duplicated by JENKINS-46345 [ JENKINS-46345 ]
            michaelneale Michael Neale made changes -
            Remote Link This issue links to "Page (Jenkins Wiki)" [ 17861 ]
            xtrom0rt Henri Anttila made changes -
            Attachment screenshot-1.png [ 40062 ]
            xtrom0rt Henri Anttila made changes -
            Attachment image-2017-10-19-08-24-00-504.png [ 40063 ]
            Hide
            xtrom0rt Henri Anttila added a comment -

            Coming from the embedded development world I'd like this in the form of an ability to run same stages for multiple platforms in parallel. I guess I'm after "parallel sets of stages". Like so:

            These should be able to complete independently, but if one fails, the whole pipeline should be marked as unstable.

            Show
            xtrom0rt Henri Anttila added a comment - Coming from the embedded development world I'd like this in the form of an ability to run same stages for multiple platforms in parallel. I guess I'm after "parallel sets of stages". Like so: These should be able to complete independently, but if one fails, the whole pipeline should be marked as unstable.
            Hide
            michaelneale Michael Neale added a comment -

            Henri Anttila do you mean more like a matrix of combinations? 

            Show
            michaelneale Michael Neale added a comment - Henri Anttila do you mean more like a matrix of combinations? 
            Hide
            sophistifunk Josh McDonald added a comment -

            Maybe we should start thinking about a BO-style renderer for matrix jobs?

            Show
            sophistifunk Josh McDonald added a comment - Maybe we should start thinking about a BO-style renderer for matrix jobs?
            Hide
            michaelneale Michael Neale added a comment -

            Josh McDonald yes - and I know Andrew Bayer wants it as an explicit thing in declarative (there are a few precursors though - like being able to have statuses/results per stage or step so it isn't all yellow or red or green, but nuanced) - once that is in place, yeah a matrix is absolutely needed. 

            Show
            michaelneale Michael Neale added a comment - Josh McDonald yes - and I know Andrew Bayer wants it as an explicit thing in declarative (there are a few precursors though - like being able to have statuses/results per stage or step so it isn't all yellow or red or green, but nuanced) - once that is in place, yeah a matrix is absolutely needed. 
            Hide
            xtrom0rt Henri Anttila added a comment -

            I'm not entirely sure, to be honest. Never fiddled much with the GUI jobs of old, as I went straight for the pipeline as code. It certainly sounds about right. Basically, what I'd need to accomplish is running the same set of steps for a number of different platforms. The steps are the same, but how they are accomplished for a particular platform might vary. That difference is handled in the shared library based on some parameterization.

            For example, a build step for one device might just be a single compilation of latest commit on git master branch. Another device might need to compile one set of latest master source code and after that a set of latest master source code from another git repo using artifacts from the first one. Same goes for the other steps. The stage goal is the same (checkout, build, deploy for test, test, etc), but the way to accomplish it might vary.

            After all the steps are through, the job result should reflect the status for that particular platform. Even though different platforms might utilize the same source code, they are separate products and I need to know everything for that particular product is working as intended

            Show
            xtrom0rt Henri Anttila added a comment - I'm not entirely sure, to be honest. Never fiddled much with the GUI jobs of old, as I went straight for the pipeline as code. It certainly sounds about right. Basically, what I'd need to accomplish is running the same set of steps for a number of different platforms. The steps are the same, but how they are accomplished for a particular platform might vary. That difference is handled in the shared library based on some parameterization. For example, a build step for one device might just be a single compilation of latest commit on git master branch. Another device might need to compile one set of latest master source code and after that a set of latest master source code from another git repo using artifacts from the first one. Same goes for the other steps. The stage goal is the same (checkout, build, deploy for test, test, etc), but the way to accomplish it might vary. After all the steps are through, the job result should reflect the status for that particular platform. Even though different platforms might utilize the same source code, they are separate products and I need to know everything for that particular product is working as intended
            Hide
            xxann5 Chris Majoros added a comment -

            I am actually looking to do the same thing as Henri Anttila

            Show
            xxann5 Chris Majoros added a comment - I am actually looking to do the same thing as Henri Anttila
            Hide
            cryptomatt Mathew Joseph added a comment -

            Went to stack overflow asking for suggestions for a problem that requires a nested parallel stage and was pointed to this item. Have there been any updates to this at all?

            Show
            cryptomatt Mathew Joseph added a comment - Went to stack overflow asking for suggestions for a problem that requires a nested parallel stage and was pointed to this item. Have there been any updates to this at all?
            Hide
            michaelneale Michael Neale added a comment -

            Mathew Joseph not yet no. One level of nesting has some low level support in the visualisation (ie a paralell branch with multiple stages) but is not glued to the api yet. 

            Show
            michaelneale Michael Neale added a comment - Mathew Joseph not yet no. One level of nesting has some low level support in the visualisation (ie a paralell branch with multiple stages) but is not glued to the api yet. 
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal UX-584 (Web Link)" [ 18213 ]
            Hide
            nroth Nicholas Roth added a comment -

            This would be really nice to have!

            Show
            nroth Nicholas Roth added a comment - This would be really nice to have!
            Hide
            hellspam Roy Arnon added a comment - - edited

            +1 from me as well.

            In our company, I also have a use case for nested parallel stages. We have a huge monorepo building multiple maven modules.  We are currently converting them to pipelines, and we parallelize a lot of steps:

            1. Build Jars 
            2. Package & Test (Parallel)
              1. Build RPM 
              2. Build Docker
              3. Run Tests (Parallel)
            3. Promote build

            We have about 20k unit tests which take a long while to run - to speed up the build, we paralleized the tests on multiple nodes, and during the test run, we also want to perform other steps.

            Currently, when the Run Tests stage is ran, everything in blue ocean is broken - the view does not display any nodes even, only steps. 

            I also think the Andrew Miller suggestion seems like the best option.

             

            Show
            hellspam Roy Arnon added a comment - - edited +1 from me as well. In our company, I also have a use case for nested parallel stages. We have a huge monorepo building multiple maven modules.  We are currently converting them to pipelines, and we parallelize a lot of steps: Build Jars  Package & Test (Parallel) Build RPM  Build Docker Run Tests (Parallel) Promote build We have about 20k unit tests which take a long while to run - to speed up the build, we paralleized the tests on multiple nodes, and during the test run, we also want to perform other steps. Currently, when the Run Tests stage is ran, everything in blue ocean is broken - the view does not display any nodes even, only steps.  I also think the Andrew Miller suggestion seems like the best option.  
            Hide
            fwilhe Florian Wilhelm added a comment -

            I'm a software developer at SAP, and we use Jenkins including Blue Ocean to provide a build pipeline for our partners. We've also stumbled upon this issue, which holds us back from enabling parallel test execution by default. We introduced a feature toggle for this, which we would like to remove as soon as this issue is resolved.

            I understand this issue is not easy to solve, but for our short term needs it would be good enough if nested parallel stages were not displayed at all in Blue Ocean. Is it possible to do this in the foreseeable future?

            Best regards,
            Florian

            Show
            fwilhe Florian Wilhelm added a comment - I'm a software developer at SAP, and we use Jenkins including Blue Ocean to provide a build pipeline for our partners. We've also stumbled upon this issue, which holds us back from enabling parallel test execution by default. We introduced a feature toggle for this, which we would like to remove as soon as this issue is resolved. I understand this issue is not easy to solve, but for our short term needs it would be good enough if nested parallel stages were not displayed at all in Blue Ocean. Is it possible to do this in the foreseeable future? Best regards, Florian
            gabloe Gabriel Loewen made changes -
            Attachment Before.PNG [ 41786 ]
            gabloe Gabriel Loewen made changes -
            Attachment After.png [ 41787 ]
            Hide
            gabloe Gabriel Loewen added a comment -

            Something that I find interesting is that nested stage visualizations kind of work while your pipeline is running and when your nested stage is created within a script block. If I have some pipeline code such as the following:

            pipeline {
                agent any
                stages {
                    stage("Testing") {
                        steps {
                            script {
                                stage("A") {
                                    parallel "1": {
                                        println "1"
            			    sleep 500
                                    }, "2": {
                                        println "2"
            			    sleep 500
                                    }, "3": {
                                        println "3"
            			    sleep 500
                                    }
                                }
                            }
                        }
                    }
                }
            }
            

            Then while this pipeline is running I see the following in Blue Ocean:

             

            But after the pipeline finishes the nested stage and the parent stage get merged:

            What would be really great is if the visualization that I see while the pipeline is running does not change when the pipeline finishes.

            Show
            gabloe Gabriel Loewen added a comment - Something that I find interesting is that nested stage visualizations kind of work while your pipeline is running and when your nested stage is created within a script block. If I have some pipeline code such as the following: pipeline {     agent any     stages {         stage( "Testing" ) {             steps {                 script {                     stage( "A" ) {                         parallel "1" : {                             println "1" sleep 500                         }, "2" : {                             println "2" sleep 500                         }, "3" : {                             println "3" sleep 500                         }                     }                 }             }         }     } } Then while this pipeline is running I see the following in Blue Ocean:   But after the pipeline finishes the nested stage and the parent stage get merged: What would be really great is if the visualization that I see while the pipeline is running does not change when the pipeline finishes.
            Hide
            ccaraivan Costin Caraivan added a comment - - edited

            This is quite important for many CI scenarios. Quick example:

            first stage: build - executed on a build node
            parallel launch of 3 logical "blocks":
            ___ 1 block for 1 type of integration test - executed on a specialized integration node

            ___ 1 block for 1 other type of integration test - executed on another specialized integration node
            ___ 1 bigger block for deployment and functional tests:
                   ___1 stage for deployment - executed on the target app environment
                   ___1 stage for functional tests - executed on a specialized functional test runner node

            Unfortunately... normal stages can only nest either "parallel" or "steps", not further stages. But here we need another stage cause the third block is actually split into two stages - steps don't work because they can't be executed on different nodes. And these stages are sequential, can't be put in a parallel block: running the tests before deployment is useless.

            In my opinion perfect is the enemy of good.

            Just define a maximum level of nesting and stick to it (call it a design decision, limitation, whatever). I know it's hard to say what that level should be, and we all know the apocryphal statement by Gates ("640k of RAM ought to be enough for anyone"), but I'd say 3 levels of nesting should be enough.

            This should simplify the implementation of both the actual pipeline code and also the visualization.

            Show
            ccaraivan Costin Caraivan added a comment - - edited This is quite important for many CI scenarios. Quick example: first stage: build - executed on a build node parallel launch of 3 logical "blocks": ___ 1 block for 1 type of integration test - executed on a specialized integration node ___ 1 block for 1 other type of integration test - executed on another specialized integration node ___ 1 bigger block for deployment and functional tests:        ___1 stage for deployment - executed on the target app environment        ___1 stage for functional tests - executed on a specialized functional test runner node Unfortunately... normal stages can only nest either "parallel" or "steps", not further stages. But here we need another stage cause the third block is actually split into two stages - steps don't work because they can't be executed on different nodes. And these stages are sequential, can't be put in a parallel block: running the tests before deployment is useless. In my opinion perfect is the enemy of good. Just define a maximum level of nesting and stick to it (call it a design decision, limitation, whatever). I know it's hard to say what that level should be, and we all know the apocryphal statement by Gates ("640k of RAM ought to be enough for anyone"), but I'd say 3 levels of nesting should be enough. This should simplify the implementation of both the actual pipeline code and also the visualization.
            Hide
            abouquet Anthony BOUQUET added a comment -

            +1 for Costin Caraivan

            Have long running step and can't execute it in a parallel way because of this nested pipeline limitation.

            3 level of nested pipeline should be enought

            Show
            abouquet Anthony BOUQUET added a comment - +1 for Costin Caraivan Have long running step and can't execute it in a parallel way because of this nested pipeline limitation. 3 level of nested pipeline should be enought
            Hide
            luckyhorang Hokwang Lee added a comment - - edited

            +1

            I think this is really really needed for making flexible pipeline.

             

            Because of feature toggling, 

            we need

            checkout - build - test

            checkout - build - test

            checkout - build - test 

            ... (so many)

            (only different is source code checkouted and build command line)

             

            If this feature can not be implemented, I will search another tool.

             

            Show
            luckyhorang Hokwang Lee added a comment - - edited +1 I think this is really really needed for making flexible pipeline.   Because of feature toggling,  we need checkout - build - test checkout - build - test checkout - build - test  ... (so many) (only different is source code checkouted and build command line)   If this feature can not be implemented, I will search another tool.  
            Hide
            tdensmore Todd Densmore added a comment -

            Intel is also very interested in this visualization enhancement.

            Show
            tdensmore Todd Densmore added a comment - Intel is also very interested in this visualization enhancement.
            Hide
            jbriden Jenn Briden added a comment -

            We are working on the visualization now for sequential/nested stages. PR for the support in Declarative 1.3: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227. 

            Show
            jbriden Jenn Briden added a comment - We are working on the visualization now for sequential/nested stages. PR for the support in Declarative 1.3: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227.  
            jbriden Jenn Briden made changes -
            Assignee CloudBees Inc. [ cloudbees ]
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Great to hear 

            Two questions:

            1. Are you also planning to support the scripted pipeline?
            2. As far as I see this the most important thing is to support stages within parallel blocks (in the first step no further nesting required). Will this be covered? According to the description I fear that this will not be included.
            Show
            macdrega Joerg Schwaerzler added a comment - Great to hear  Two questions: Are you also planning to support the scripted pipeline? As far as I see this the most important thing is to support stages within parallel blocks (in the first step no further nesting required). Will this be covered? According to the description I fear that this will not be included.
            jbriden Jenn Briden made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jbriden Jenn Briden made changes -
            Description {panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}


            *Original ticket*
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.


            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}

            *Example 2*

            {code}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel}


            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}
            *Original ticket*
             I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}
            *Example 2*
            {code:java}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jbriden Jenn Briden made changes -
            Description {panel}


            There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

            It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

            The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
            {panel}
            *Original ticket*
             I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}
            *Example 2*
            {code:java}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel}
            In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.{panel}
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}
            *Example 2*
            {code:java}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            jbriden Jenn Briden made changes -
            Summary Support nested stages in the Pipeline Graph View sequential stages in the pipeline visualization graph
            Hide
            dsmoljanovic Domagoj Smoljanovic added a comment -

            I see that the above PR is merged but issue is still in progress. Is this still being worked on?

            When done will it support nested parallels (parallel in parallel)? As shown on pipeline_mockup.png ? Because with version 1.3.1 it still doesn't do that.

             

            Thanx

            Show
            dsmoljanovic Domagoj Smoljanovic added a comment - I see that the above PR is merged but issue is still in progress. Is this still being worked on? When done will it support nested parallels (parallel in parallel)? As shown on pipeline_mockup.png ? Because with version 1.3.1 it still doesn't do that.   Thanx
            Hide
            sophistifunk Josh McDonald added a comment -

            AFAIK there's no plan to support nested parallels at the moment.

            Show
            sophistifunk Josh McDonald added a comment - AFAIK there's no plan to support nested parallels at the moment.
            Hide
            kurzy Daniel Kurzynski added a comment -

            We really need that feature. Currently, we have two levels of parallelization and the visualization in BlueOcean is totally broken. Really, looking forward to supporting at least two level of parallelization (parallel within parallel).

            Show
            kurzy Daniel Kurzynski added a comment - We really need that feature. Currently, we have two levels of parallelization and the visualization in BlueOcean is totally broken. Really, looking forward to supporting at least two level of parallelization (parallel within parallel).
            Hide
            hasselmm Mathias Hasselmann added a comment -

            I might be mistake as I don't have numbers, but the parallel pipeline described in https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/ seems a pretty common use-case for pipelines. It might be the most natural and most common pipeline even. Therefore at least this kind of pipeline should get appropiate visualiation, IMHO.

            Show
            hasselmm Mathias Hasselmann added a comment - I might be mistake as I don't have numbers, but the parallel pipeline described in https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/  seems a pretty common use-case for pipelines. It might be the most natural and most common pipeline even. Therefore at least this kind of pipeline should get appropiate visualiation, IMHO.
            gertvdijk Gert van Dijk made changes -
            Attachment jenkins-weird-masked2.png [ 43737 ]
            Hide
            gertvdijk Gert van Dijk added a comment - - edited

            First of all, I really like nested sequential stages, because I use a build tool already running things in maximum parallel (bazel), I don't want to run things with parallel stages which will be blocking on the Bazel server anyway.

            However, I noticed a very annoying visualization issue when a sub-stage has FAILED (due to a failed step): the parent stage is marked SUCCESS. I would assume the failed state would propagate up to its parent.

            It not only hides the error in the graph and makes it's hard to spot where in the pipeline the error occurred, it also hides the existence of the sub-stage. In the screenshot below, the "Code Style" stage has multiple sub-stages, that got somehow merged with its parent in this view, but the steps shown are part of one of the sub-stages.

            Sub-stages are shown during the run, though.

            As this Jira issue is still in-progress, I didn't want to open a new one. And I couldn't find any other issue reporting similar issues.
            (Seeing this on 1.7.2 / Core 2.121.2)

            Update: in 1.8.2 I now see all stages, but all flattened, as if there is no nesting in stages. It has improved the situation a lot to spot the errors, though, but it's far from ideal.

            Show
            gertvdijk Gert van Dijk added a comment - - edited First of all, I really like nested sequential stages, because I use a build tool already running things in maximum parallel (bazel), I don't want to run things with parallel stages which will be blocking on the Bazel server anyway. However, I noticed a very annoying visualization issue when a sub-stage has FAILED (due to a failed step): the parent stage is marked SUCCESS. I would assume the failed state would propagate up to its parent. It not only hides the error in the graph and makes it's hard to spot where in the pipeline the error occurred, it also hides the existence of the sub-stage . In the screenshot below, the "Code Style" stage has multiple sub-stages, that got somehow merged with its parent in this view, but the steps shown are part of one of the sub-stages. Sub-stages are shown during the run , though. As this Jira issue is still in-progress, I didn't want to open a new one. And I couldn't find any other issue reporting similar issues. (Seeing this on 1.7.2 / Core 2.121.2) Update: in 1.8.2 I now see all stages, but all flattened, as if there is no nesting in stages. It has improved the situation a lot to spot the errors, though, but it's far from ideal.
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Blue Ocean 1.8.x adds support for visualizing sequential stages, so I am closing this as a dupe of JENKINS-49050 (although note that I believe that issue is specific to declarative pipeline).

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Blue Ocean 1.8.x adds support for visualizing sequential stages, so I am closing this as a dupe of JENKINS-49050 (although note that I believe that issue is specific to declarative pipeline).
            dnusbaum Devin Nusbaum made changes -
            Link This issue duplicates JENKINS-49050 [ JENKINS-49050 ]
            dnusbaum Devin Nusbaum made changes -
            Status In Progress [ 3 ] Closed [ 6 ]
            Resolution Duplicate [ 3 ]
            dnusbaum Devin Nusbaum made changes -
            Released As Blue Ocean 1.8.0
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Devin Nusbaum So for nested stages in scripted pipelines, do we need to open another feature request - or is there already something in place?

            Show
            macdrega Joerg Schwaerzler added a comment - Devin Nusbaum So for nested stages in scripted pipelines, do we need to open another feature request - or is there already something in place?
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Joerg Schwaerzler I don't think there are any open feature requests specific to nested stages in scripted pipelines. We could reopen this issue and modify the title to be a little more specific, or we could open a new issue (probably better to have a more focused issue, assuming some of the people here are satisfied with the nested stage support for declarative in blue ocean 1.8), but I think either would be fine.

            Show
            dnusbaum Devin Nusbaum added a comment - Joerg Schwaerzler I don't think there are any open feature requests specific to nested stages in scripted pipelines. We could reopen this issue and modify the title to be a little more specific, or we could open a new issue (probably better to have a more focused issue, assuming some of the people here are satisfied with the nested stage support for declarative in blue ocean 1.8), but I think either would be fine.
            Hide
            kurzy Daniel Kurzynski added a comment - - edited

            I think originally, this ticketed was also about having parallel blocks within parallel blocks. So at least two level of parallelization. 
            Although the declarative syntax does not support it, you can have scripted parallel blocks within a parallel block in a declarative pipeline. The result is not visualized properly.

             

             

            Show
            kurzy Daniel Kurzynski added a comment - - edited I think originally, this ticketed was also about having parallel blocks within parallel blocks. So at least two level of parallelization.  Although the declarative syntax does not support it, you can have scripted parallel blocks within a parallel block in a declarative pipeline. The result is not visualized properly.    
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Daniel Kurzynski Yes, this issue has multiple feature requests in the comments, so it seems to me like it would make more sense to open individual issues so each request can be discussed separately, but we could also just reopen this issue.

            Show
            dnusbaum Devin Nusbaum added a comment - Daniel Kurzynski Yes, this issue has multiple feature requests in the comments, so it seems to me like it would make more sense to open individual issues so each request can be discussed separately, but we could also just reopen this issue.
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Devin Nusbaum: Hmm, looking at the examples above think this issue was originally targeted to scripted Pipeline. However as I like the idea of having smaller feature requests which are easier to implement on its own. Therefore I will create a new feature request hoping it'll not be merged with similar feature requests.

            Show
            macdrega Joerg Schwaerzler added a comment - Devin Nusbaum : Hmm, looking at the examples above think this issue was originally targeted to scripted Pipeline. However as I like the idea of having smaller feature requests which are easier to implement on its own. Therefore I will create a new feature request hoping it'll not be merged with similar feature requests.
            Hide
            macdrega Joerg Schwaerzler added a comment -

            For those interested: I raised a new fr here: JENKINS-53162

            Show
            macdrega Joerg Schwaerzler added a comment - For those interested: I raised a new fr here: JENKINS-53162
            dnusbaum Devin Nusbaum made changes -
            Link This issue relates to JENKINS-53162 [ JENKINS-53162 ]
            Hide
            philsegal Phil Segal added a comment -

            Josh McDonald is there still no plans to support nested parallels? we have built a nested parallel scripted pipeline, and the blue ocean view seems to change between various representations. More worryingly, if a task fails it shows as GREEN  . Something like pipeline_mockups.png from above would be amazing

             

            Show
            philsegal Phil Segal added a comment - Josh McDonald is there still no plans to support nested parallels? we have built a nested parallel scripted pipeline, and the blue ocean view seems to change between various representations. More worryingly, if a task fails it shows as GREEN  . Something like pipeline_mockups.png from above  would be amazing  
            Hide
            kurzy Daniel Kurzynski added a comment -

            Devin Nusbaum As suggested, I have created an issue for two level of parallelization: https://issues.jenkins-ci.org/browse/JENKINS-54010

            Show
            kurzy Daniel Kurzynski added a comment - Devin Nusbaum As suggested, I have created an issue for two level of parallelization:  https://issues.jenkins-ci.org/browse/JENKINS-54010
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54010 [ JENKINS-54010 ]
            feirenliu zengqiao liu made changes -
            Description {panel}
            In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.{panel}
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}
            *Example 2*
            {code:java}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            {panel}

            In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.
            {panel}
            I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

            *Example 1*
            {code:java}
            def test = [:]

            test["a"] = {
                stage ("a") {
                    stage ("ab") {
                        sh "echo stage abc"
                    }
                    stage ("xyz") {
                        sh "echo stage xyz"
                    }
                }
            }

            test["b"] = {
                stage ("b") {
                    stage ("bb") {
                        sh "echo stage bb"
                    }
                    stage ("bxz") {
                        sh "echo stagebxyz"
                    }
                }
            }
            node {
               //stage 'start'
               parallel test
               stage ('middle') {
                   sh "echo middle"
               }
               
            }
            {code}
            *Example 2*
            {code:java}
            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }

              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            {code}
            Hide
            asg Ahmed Sghaier added a comment -

            Is there any luck for implementing the pipeline mockup proposed by Ben Dean? https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647

            Should we create a separate issue for this?

            Show
            asg Ahmed Sghaier added a comment - Is there any luck for implementing the pipeline mockup proposed by Ben Dean? https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647 Should we create a separate issue for this?

              People

              • Assignee:
                cloudbees CloudBees Inc.
                Reporter:
                mbadran mohamed badran
              • Votes:
                206 Vote for this issue
                Watchers:
                229 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: