Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45553

Parallel pipeline execution scales poorly

    Details

    • Type: Bug
    • Status: In Review
    • Priority: Critical
    • Resolution: Unresolved
    • Environment:
    • Similar Issues:

      Description

      Execution of parallel blocks scales poorly for values of N > 100.  With ~50 nodes (each with 4 executors, for a total of ~200 slots), the following pipeline job takes extraordinarily long to execute:

       

      def stepsForParallel = [:]
      for (int i = 0; i < Integer.valueOf(params.SUB_JOBS); i++) {
        def s = "subjob_${i}" 
        stepsForParallel[s] = {
          node("darwin") {
            echo "hello"
          }
        }
      }
      parallel stepsForParallel
      

       

      SUB_JOBS   Time (sec)
      ---------------------
       100         10
       200         40
       300         96
       400        214
       500        392
       600        660
       700        960
       800       1500
       900       2220
      1000       gave up...

      At no point does the underlying system become taxed (CPU utilization is very low, as this is a very beefy system – 28 cores, 128GB RAM, SSDs)

      CPU and Thread CPU Time Sampling (via VisualVM) are attached for reference.

       

       

       

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                tskrainar Tom Skrainar
              • Votes:
                4 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: