Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2072

Make the Queue implementation replaceable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Component/s: other
    • Labels:
      None
    • Environment:
      Platform: All, OS: All
    • Similar Issues:
      Show 5 results

      Description

        Attachments

          Issue Links

            Activity

            Hide
            fireresq fireresq added a comment -

            added CC

            Show
            fireresq fireresq added a comment - added CC
            Hide
            danrubel danrubel added a comment -

            What if the queue ordering was separated from the algorithm(s) that determined the ordering?

            More specifically, what if each job had an integer representing its priority where smaller numbers
            represented higher priority and jobs with the same numeric priority were built on first come first serve
            basis. All jobs with a given priority would be built and completed before any jobs with lower priority
            were built.

            Given this internal queue implementation, you could have
            1) a field in the job configuration page allowing you to manually specify the priority
            2) an extension point for zero or more algorithms to adjust multiple job priorities when a single job's
            configuration is saved
            3) a facility for a user to manually and temporarily "bump up" a job's priority for a single build

            Separating queue ordering from the the algorithm(s) that determined the ordering allows you to keep the
            queue implementation internal and straightforward, while allowing others to explore different algorithms
            to suit their individual needs. Algorithms can also be composited, applying first one algorithm then
            another to achieve the desired effect.

            Show
            danrubel danrubel added a comment - What if the queue ordering was separated from the algorithm(s) that determined the ordering? More specifically, what if each job had an integer representing its priority where smaller numbers represented higher priority and jobs with the same numeric priority were built on first come first serve basis. All jobs with a given priority would be built and completed before any jobs with lower priority were built. Given this internal queue implementation, you could have 1) a field in the job configuration page allowing you to manually specify the priority 2) an extension point for zero or more algorithms to adjust multiple job priorities when a single job's configuration is saved 3) a facility for a user to manually and temporarily "bump up" a job's priority for a single build Separating queue ordering from the the algorithm(s) that determined the ordering allows you to keep the queue implementation internal and straightforward, while allowing others to explore different algorithms to suit their individual needs. Algorithms can also be composited, applying first one algorithm then another to achieve the desired effect.
            Hide
            mdonohue mdonohue added a comment -

            Ultimately it would be nice to have an extension point just for determining each
            projects build ordering, without having to muck with the entire queue
            implementation. That's the end goal.

            In the interim, I can see how a real number priority could solve a number of the
            problems, but it is also lacking. Notably, it is not very object oriented, so
            doing things like tracking why a job was assigned a given priority is very
            difficult to extract from a real number.

            Composability would be nice, but I don't think it is realistic to expect to
            compose arbitrary priority systems. For example, I would like to queue jobs
            according to their expected run time - faster jobs go first. This maps to a
            real number easily. So do user supplied priorities. But composing them seems
            awfully complicated. How do I define a function from real to real that does
            something meaningful with the initial priority, but still produces a queue that
            orders things according to the expected run time?

            Similarly for the bump up feature. I'd rather have some sort of proper
            datastructure that allows for a predicate that moves a job to the top, rather
            than taking the maximum current priority and adding 1 to it. With a
            datastructure, the manually promoted job can be highlighted in the queue as
            promoted, for example.

            Show
            mdonohue mdonohue added a comment - Ultimately it would be nice to have an extension point just for determining each projects build ordering, without having to muck with the entire queue implementation. That's the end goal. In the interim, I can see how a real number priority could solve a number of the problems, but it is also lacking. Notably, it is not very object oriented, so doing things like tracking why a job was assigned a given priority is very difficult to extract from a real number. Composability would be nice, but I don't think it is realistic to expect to compose arbitrary priority systems. For example, I would like to queue jobs according to their expected run time - faster jobs go first. This maps to a real number easily. So do user supplied priorities. But composing them seems awfully complicated. How do I define a function from real to real that does something meaningful with the initial priority, but still produces a queue that orders things according to the expected run time? Similarly for the bump up feature. I'd rather have some sort of proper datastructure that allows for a predicate that moves a job to the top, rather than taking the maximum current priority and adding 1 to it. With a datastructure, the manually promoted job can be highlighted in the queue as promoted, for example.
            Hide
            danrubel danrubel added a comment -

            You could have an extension point for providing a java.util.Comparator for
            determining which job should be built first. Those could also be composed in that
            if a comparator could not determine which job should be built first, then it
            passes the responsibility down the chain of command. In fact the first comparator
            in chain of command could check for jobs that have been manually/temporarily
            bumped-up in priority by a developer waiting for a build to happen. Perhaps this
            is more along the lines of what you are thinking?

            Show
            danrubel danrubel added a comment - You could have an extension point for providing a java.util.Comparator for determining which job should be built first. Those could also be composed in that if a comparator could not determine which job should be built first, then it passes the responsibility down the chain of command. In fact the first comparator in chain of command could check for jobs that have been manually/temporarily bumped-up in priority by a developer waiting for a build to happen. Perhaps this is more along the lines of what you are thinking?
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            At this point enough aspects of a queue is pluggable that I consider this issue fixed.

            If people need additional aspects pluggable, please open separate issues.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - At this point enough aspects of a queue is pluggable that I consider this issue fixed. If people need additional aspects pluggable, please open separate issues.

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                mdonohue mdonohue
              • Votes:
                17 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: