Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46236

Throttle parallel step in pipeline script

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Duplicate
    • Icon: Major Major
    • workflow-cps-plugin
    • None

      It appears that Jenkins pipelines have no way of throttling parallel closures execution.

      Given the example:

      stage('Test') {
        steps {
          script {
            testing_closures = [one: { print("starting one"); sleep 10; print("finishing one") },
                                two: { print("starting two"); sleep 10; print("finishing two") },
                                three: { print("starting three"); sleep 10; print("finishing three") },
                                four: { print("starting four"); sleep 10; print("finishing four") },
                                five: { print("starting five"); sleep 10; print("finishing five") },
                                six: { print("starting six"); sleep 10; print("finishing six") }]
            parallel(testing_closures)
          }
        }
      }
      

      The main goal is to throttle those closures. If we don't want for all six of them to run concurrently, but let's say only 3 at a time, we'd need an argument for 'parallel' step:
      parallel(closures: testing_closures, maxThreadCount: 3)

      Currently, as far as I know, there are no workarounds for this issue. If JENKINS-46235 is resolved, the following workaround could be used:

      1. Dynamically create 3 lockable resources via LockableResourcesManager::createResourceWithLabel() with build-unique labels
      2. Lock them by label in all of the closures
      3. The closures will wait for each other to finish and only 3 at the time would be running.
      4. Delete the closures (no such API method. JENKINS-46235 is for exposing 'delete' method for lockable resource)

            Unassigned Unassigned
            mireksz Mirek Sz
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: