Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11997

Consider jobs in the Executors for priority sorter

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: prioritysorter-plugin
    • Labels:
      None
    • Environment:
      Ubuntu 10.04
      Jenkins 1.441
    • Similar Issues:

      Description

      The current PrioritySorter plugin does not consider the executing jobs. So it is possible for a job with a lower priority to be started even when there is a higher priority job already executing.

      A great enhancement to PrioritySorter would be to:

      • prevent a job from starting if there is already a job with a higher priority executing.
      • be able to configure this feature on or off. Some sites will want the current behavior.

      One usage scenario is:

      • I have a set of jobs for the integration branch and another set of jobs for the trunk branch (say to do a production release)
      • the int jobs are kicked off by changes to the SCM
      • but when I need to run the trunk jobs, they need to run immediately, i.e. the int jobs can wait.
      • To do that, I set the priority on the trunk jobs to a higher value.
      • I kick off the trunk jobs
      • if an int job comes in at that point, it is sorted last in the queue but also isn't allowed to run until the higher priority trunk jobs are done.

        Attachments

          Activity

          Hide
          bklarson bklarson added a comment -

          I'm not sure I understand - your usage scenario is exactly how we do things at $DAYJOB.

          I suspect you are seeing the lower-priority jobs run because you still have extra executors waiting for a job. Could that be the case? If so, if the lower-priority builds are causing problems/slowing down the high-priority builds, can you adjust your number of executors so that this won't be an issue?

          I'm not sure if there is any official documentation/position on this, but I believe a good rule-of-thumb is to not have more executors on a machine than that machine can reasonably handle at one time without slowing down. For multi-threaded builds, I allow one executor per machine, and for single-threaded builds I allow one executor per CPU core.

          Show
          bklarson bklarson added a comment - I'm not sure I understand - your usage scenario is exactly how we do things at $DAYJOB. I suspect you are seeing the lower-priority jobs run because you still have extra executors waiting for a job. Could that be the case? If so, if the lower-priority builds are causing problems/slowing down the high-priority builds, can you adjust your number of executors so that this won't be an issue? I'm not sure if there is any official documentation/position on this, but I believe a good rule-of-thumb is to not have more executors on a machine than that machine can reasonably handle at one time without slowing down. For multi-threaded builds, I allow one executor per machine, and for single-threaded builds I allow one executor per CPU core.
          Hide
          arrizza John Arrizza added a comment -

          Brad,

          Sorry for the late reply.

          Correct, the lower-priority jobs are running because there are empty executors. I think things are working correctly at your $DAYJOB because you have one executor per machine which (sort of) achieves what I'm asking for. The difference is that I'd like to do it over a set of jobs, not just one job at a time.

          So:

          • I have set the number of executors to be the number of concurrent jobs the Build Server can handle at once.
          • Having a single executor is too restrictive since almost all of the int jobs can run concurrently on a normal day.
          • the Trunk builds have multiple jobs with downstream triggers
          • the int builds have multiple jobs with downstream triggers
          • When I submit a Trunk build, it, as a whole, needs to take precedence, i.e. the set of trunk jobs takes precedence over the set of int jobs (or in other words, any trunk job takes precedence over all int jobs).

          So an ideal sequence of events would be;

          • there are 6 open executors, nothing is running.
          • a trunk build is submitted
          • there are 4 trunk jobs that get started by SCM changes; 3 of these take 5 minutes to run; the other takes 60 minutes; each of these has downstream triggered jobs
          • all 4 trunk jobs are started; there are now 2 open executors
          • an int build gets triggered by an SCM change;
          • these would all wait until all of the trunk jobs (even the long running one) completes.
          • if the trunk jobs trigger a downstream job, it runs before the int jobs in the job queue.
          • the int jobs only start running if there's no trunk jobs in the queue or in the executors.

          John

          Show
          arrizza John Arrizza added a comment - Brad, Sorry for the late reply. Correct, the lower-priority jobs are running because there are empty executors. I think things are working correctly at your $DAYJOB because you have one executor per machine which (sort of) achieves what I'm asking for. The difference is that I'd like to do it over a set of jobs, not just one job at a time. So: I have set the number of executors to be the number of concurrent jobs the Build Server can handle at once. Having a single executor is too restrictive since almost all of the int jobs can run concurrently on a normal day. the Trunk builds have multiple jobs with downstream triggers the int builds have multiple jobs with downstream triggers When I submit a Trunk build, it, as a whole, needs to take precedence, i.e. the set of trunk jobs takes precedence over the set of int jobs (or in other words, any trunk job takes precedence over all int jobs). So an ideal sequence of events would be; there are 6 open executors, nothing is running. a trunk build is submitted there are 4 trunk jobs that get started by SCM changes; 3 of these take 5 minutes to run; the other takes 60 minutes; each of these has downstream triggered jobs all 4 trunk jobs are started; there are now 2 open executors an int build gets triggered by an SCM change; these would all wait until all of the trunk jobs (even the long running one) completes. if the trunk jobs trigger a downstream job, it runs before the int jobs in the job queue. the int jobs only start running if there's no trunk jobs in the queue or in the executors. John
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: emsa23
          Path:
          src/main/java/jenkins/advancedqueue/JobGroup.java
          src/main/java/jenkins/advancedqueue/PriorityConfiguration.java
          src/main/java/jenkins/advancedqueue/RunExclusiveThrottler.java
          src/main/resources/jenkins/advancedqueue/PriorityConfiguration/index.jelly
          http://jenkins-ci.org/commit/priority-sorter-plugin/fe2e1d1cb5451f88b88c16f53ce7351ff60c1a8d
          Log:
          JENKINS-11997 Consider jobs in the Executors for priority sorter

          Introducing "Run Exclusive", with "Run Exclusive" set on a
          JobGroup only Jobs from that JobGroup will be started
          until no more Jobs from that JobGroup is in queue being run.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: emsa23 Path: src/main/java/jenkins/advancedqueue/JobGroup.java src/main/java/jenkins/advancedqueue/PriorityConfiguration.java src/main/java/jenkins/advancedqueue/RunExclusiveThrottler.java src/main/resources/jenkins/advancedqueue/PriorityConfiguration/index.jelly http://jenkins-ci.org/commit/priority-sorter-plugin/fe2e1d1cb5451f88b88c16f53ce7351ff60c1a8d Log: JENKINS-11997 Consider jobs in the Executors for priority sorter Introducing "Run Exclusive", with "Run Exclusive" set on a JobGroup only Jobs from that JobGroup will be started until no more Jobs from that JobGroup is in queue being run.
          Hide
          emsa23 Magnus Sandberg added a comment -

          Fixed in 2.3 using the new "Run Exclusive" mode.

          Show
          emsa23 Magnus Sandberg added a comment - Fixed in 2.3 using the new "Run Exclusive" mode.

            People

            • Assignee:
              emsa23 Magnus Sandberg
              Reporter:
              arrizza John Arrizza
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: