Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41334

Support parallel execution of stages in Declarative

    XMLWordPrintable

    Details

    • Epic Link:
    • Sprint:
      Declarative - 1.2
    • Similar Issues:

      Description

      Improvement on roadmap

      This improvement is on the Blue Ocean project roadmap. Check the roadmap page for updates.

      Proposal
      Note - this is a post-1.0 feature.

      Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, parallel doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

      I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested stages sections, like this:

      pipeline {
        agent any
        
        stages {
          stage('first') {
            steps {
              echo 'first, non-parallel stage'
            }
          }
      
          stage('top-parallel') {
            stages {
              stage('first-parallel') {
                steps {
                  echo 'First of the parallel stages without further nesting'
                  sleep 60
                }
              }
              stage('second-parallel') {
                stages {
                  stage('first-nested-parallel') {
                    steps {
                       echo 'the first of the nested parallel stages'
                       sleep 30
                    }
                 }
                 stage('second-nested-parallel') {
                    steps {
                       echo 'the second of the nested parallel stages'
                       sleep 30
                    }
                 }
              }
           }
        }
      }
      

      So in this scenario, stage('first') runs first. When it completes, stage('top-parallel') starts and immediately goes into its nested stages in parallel. stage('first-parallel') starts and goes for 60 seconds, while stage('second-parallel') starts at the same time and descends into its nested stages in parallel as well.

      I don't know yet where I'd allow agent and friends, but within a stage, you would need to have one and only one of steps or stages - i.e., a stage either could have steps it executes or it could be a container for parallel stages.

      Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.

        Attachments

          Issue Links

            Activity

            Hide
            jamesdumay James Dumay added a comment -

            Jakub Pawlinski we introduced a cleaner syntax for what you had previously using script blocks. You can also now define environment variables, agents, etc at a parallel stage level just like regular stages. What were you expecting instead?

            Show
            jamesdumay James Dumay added a comment - Jakub Pawlinski we introduced a cleaner syntax for what you had previously using script blocks. You can also now define environment variables, agents, etc at a parallel stage level just like regular stages. What were you expecting instead?
            Hide
            quas Jakub Pawlinski added a comment - - edited

            I was hoping for nested parallels and parallel pipelines, so I could not only do:

            parallel {do {a} do {b}}

            but also:

            parallel {do {a, b, c} do {d}}

            I know that above can be achieved on step level, but step seems to be something small and is not visualised in blue ocean 

            parallel {do { do {a} do {b}} do {c}}

            I'm now aware that those are separated issues and watching them, but initially is seemed like this one could solve those too.

            Show
            quas Jakub Pawlinski added a comment - - edited I was hoping for nested parallels and parallel pipelines, so I could not only do: parallel { do {a} do {b}} but also: parallel { do {a, b, c} do {d}} I know that above can be achieved on step level, but step seems to be something small and is not visualised in blue ocean  parallel { do { do {a} do {b}} do {c}} I'm now aware that those are separated issues and watching them, but initially is seemed like this one could solve those too.
            Hide
            abayer Andrew Bayer added a comment -

            Jakub Pawlinski Yeah, we're adding these things gradually, to make sure the execution, syntax, and visualization all are working together at each step. I expect to get

            parallel {do {a, b, c} do {d}}
            

            before the end of the year, most likely - the nested parallels may be further out, though.

            Show
            abayer Andrew Bayer added a comment - Jakub Pawlinski Yeah, we're adding these things gradually, to make sure the execution, syntax, and visualization all are working together at each step. I expect to get parallel { do {a, b, c} do {d}} before the end of the year, most likely - the nested parallels may be further out, though.
            Hide
            mattkunze Matt Kunze added a comment - - edited

            Is there another another issue to track the progress for having multiple steps within a parallel stream (do {a, b, c})? 

            I think I'm looking for something very similar to others - the ability to use parallel to split to different environments (Linux, Windows, etc) and then run multiple stages in sequence on each (Build, Test, Archive)

            Show
            mattkunze Matt Kunze added a comment - - edited Is there another another issue to track the progress for having multiple steps within a parallel stream ( do {a, b, c })?  I think I'm looking for something very similar to others - the ability to use parallel to split to different environments (Linux, Windows, etc) and then run multiple stages in sequence on each (Build, Test, Archive)
            Hide
            abayer Andrew Bayer added a comment -

            Matt Kunze, Jakub Pawlinski Yup, JENKINS-46809 is what you're looking for.

            Show
            abayer Andrew Bayer added a comment - Matt Kunze , Jakub Pawlinski Yup, JENKINS-46809 is what you're looking for.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                abayer Andrew Bayer
              • Votes:
                42 Vote for this issue
                Watchers:
                85 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: