Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39932

Support more arbitrary Pipeline definitions

    Details

    • Similar Issues:

      Description

      I think the idea of a declarative (and not imperative) pipeline is a lot easier on the writers end (and I assume the implementers end as well), but I think the implementation of several serial stages each fanning out to a number of parallel stages is limiting to the overall description of the dependencies between different steps/nodes. In my case (simplifying, please excuse pseudo-code), I'd like to do something like this:

      stage("checkout") { node () { git XXX } }
      stage("build") {
          parallel ("ubuntu") { node("ubuntu") { make XXX }},
                       ("centos") { node("centos") { make YYY }},
                       // tons more distributions here
      }
      stage ("test") {
          node ("ubuntu") { test-ubuntu ZZZ }
      }
      
      stage ("publish") { node() { publish.sh }
      

      Since I know the "test" stage only depends on the "ubuntu" part of the build stage, I'd like the test stage to fire without waiting on the whole "build" stage completes. From what I see, the parallel block waits for everything inside to complete, but I know the dependencies and can speed it up (in my case, there are a lot more nodes who match the testing than who don't). If I were to write it in bash, I would do something like:

      git checkout XXX
      build ubuntu &
      TEST_PID=$!
      build centos &
      wait $TEST_PID
      test-ubuntu
      wait # wait for all testing and bulding to complete
      publish
      

      This is a rough way to explain the dependency graph to bash by saying "wait on 'build ubuntu' before starting the testing, but centos can take as long as it wants" and "before going to the publish step, make sure everything else is done". In my case, all the build steps for different distributions take forever because there's a finite number of VMs that can be spun up, but there's an abundance of "ubuntu" nodes that can be used for testing and compiling. It slows things down a ton to have to wait for the entire parallel node to finish before moving on, when there are free executors in latter steps that can start running.

        Attachments

          Issue Links

            Activity

            Hide
            abayer Andrew Bayer added a comment -

            fwiw, I have plans for a stage dependency graph execution model as part of Declarative post-1.0, where you'd specify your stages and which other stages they depend on, and the execution would figure out the appropriate ordering based on that, including smart parallelization etc.

            Show
            abayer Andrew Bayer added a comment - fwiw, I have plans for a stage dependency graph execution model as part of Declarative post-1.0, where you'd specify your stages and which other stages they depend on, and the execution would figure out the appropriate ordering based on that, including smart parallelization etc.
            Hide
            abayer Andrew Bayer added a comment -

            We're not going to actually be doing dependency ordering, but in the upcoming 1.3, you'll be able to do sequential lists of stages in parallel with each other (see JENKINS-46809), which I think will do the trick for you.

            Show
            abayer Andrew Bayer added a comment - We're not going to actually be doing dependency ordering, but in the upcoming 1.3, you'll be able to do sequential lists of stages in parallel with each other (see JENKINS-46809 ), which I think will do the trick for you.
            Hide
            bitwiseman Liam Newman added a comment -

            Bulk closing resolved issues.

            Show
            bitwiseman Liam Newman added a comment - Bulk closing resolved issues.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                perilousapricot Andrew Melo
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: