Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43353

Ability to abort all previous running builds

    Details

    • Similar Issues:

      Description

      It is sometimes desirable for a job (such as a branch project) to simply abort any previously running builds as soon as a new build starts. For example, in a branch project for a pull request, you might want to see test results from an earlier commit even after pushing follow-up commits, but most of the time you only care about the results of the PR head, and computer time might be too valuable to waste on the older ones.

      (I think gerrit-trigger does something like this automatically, and I have seen Alex Gray invent the same kind of pattern with JenkinsPy.)

      Merely setting the job to not be concurrent-capable does not suffice, since then newer builds will queue up waiting for the older ones to finish.

        Attachments

          Issue Links

            Activity

            Hide
            mark_han Mark Han added a comment -

            We should make sure this change will allow for QUEUED builds to cancel RUNNING builds. We are running into a scenario where the amount of executors is limited, and the build server is being super congested by running out-of-date PR builds that should ideally be cancelled by the incoming queued PR builds. 

            Show
            mark_han Mark Han added a comment - We should make sure this change will allow for QUEUED builds to cancel RUNNING builds. We are running into a scenario where the amount of executors is limited, and the build server is being super congested by running out-of-date PR builds that should ideally be cancelled by the incoming queued PR builds. 
            Hide
            jglick Jesse Glick added a comment -

            Pipeline builds are flyweight and normally exit the queue immediately; it is node blocks which would wait in queue. So just put the milestone dance at the top of the Jenkinsfile and you are fine. Working in jenkinsci/bom as far as I know.

            Show
            jglick Jesse Glick added a comment - Pipeline builds are flyweight and normally exit the queue immediately; it is node blocks which would wait in queue. So just put the milestone dance at the top of the Jenkinsfile and you are fine. Working in jenkinsci/bom as far as I know.
            Hide
            mark_han Mark Han added a comment - - edited

            Jesse Glick TIL about flyweight! This is helpful. In our recent past we have had issues in the past with milestones.

             

            Scenario: We have 7 PRs ready using Bitbucket, and 7 PR builds are queued up using the Multibranch Pipeline. After the first one of those gets merged, the other 6 have to rebuild because Develop was changed. However, a few of the previous runs are still trying to execute so a queue forms that looks something like this (Assuming there are multiple builds surrounding these that look similarly): 

            QUEUED JOBS:

            part of Android-Build-Name >> PR-2715 #8

            part of Android-Build-Name >> PR-2715 #7

            RUNNING JOBS:
            part of Android-Build-Name >> PR-2715 #6

            So, ideally we want to kill this running job #6 and the queued job #7 if a newer job #8 is queued up.

            Would something similar to line number 1 in the jenkinsfile for `jenkinsci/bom` solve this?

            Show
            mark_han Mark Han added a comment - - edited Jesse Glick  TIL about flyweight! This is helpful. In our recent past we have had issues in the past with milestones.   Scenario : We have 7 PRs ready using Bitbucket, and 7 PR builds are queued up using the Multibranch Pipeline. After the first one of those gets merged, the other 6 have to rebuild because Develop was changed. However, a few of the previous runs are still trying to execute so a queue forms that looks something like this (Assuming there are multiple builds surrounding these that look similarly):  QUEUED JOBS: part of Android-Build-Name >> PR-2715 #8 part of Android-Build-Name >> PR-2715 #7 RUNNING JOBS: part of Android-Build-Name >> PR-2715 #6 So, ideally we want to kill this running job #6 and the queued job #7 if a newer job #8 is queued up. Would something similar to line number 1 in the jenkinsfile for `jenkinsci/bom` solve this?
            Hide
            jglick Jesse Glick added a comment -

            Yes, it should. When build 8 starts, it should abort builds 6 and 7. When a build is aborted, in turn, it should abort any sh steps already running inside any node blocks, and also cancel any queue items corresponding to node blocks which have not yet started.

            Show
            jglick Jesse Glick added a comment - Yes, it should. When build 8 starts, it should abort builds 6 and 7. When a build is aborted, in turn, it should abort any sh steps already running inside any node blocks, and also cancel any queue items corresponding to node blocks which have not yet started.
            Hide
            mark_han Mark Han added a comment -

            Ended up not using Milestone because it doesn't communicate the build # that was stopped. Instead we're trying to use 

            def cancelPreviousBuilds() {
             // Check for other instances of this particular build, cancel any that are older than the current one
             def jobName = env.JOB_NAME
             def currentBuildNumber = env.BUILD_NUMBER.toInteger()
             def currentJob = Jenkins.instance.getItemByFullName(jobName)
            
             // Loop through all instances of this particular job/branch
             for (def build : currentJob.builds) {
             if (build.isBuilding() && (build.number.toInteger() < currentBuildNumber)) {
             echo "Older build still queued. Sending kill signal to build number: ${build.number}"
             build.doStop()
             }
             }
            }

            If we put this outside of our node blocks, things are working well. We previously had this in a node{} which was horrible, and resolved the issue after you informing me of the heavyweight vs flyweight executors.

            Show
            mark_han Mark Han added a comment - Ended up not using Milestone because it doesn't communicate the build # that was stopped. Instead we're trying to use  def cancelPreviousBuilds() { // Check for other instances of this particular build, cancel any that are older than the current one def jobName = env.JOB_NAME def currentBuildNumber = env.BUILD_NUMBER.toInteger() def currentJob = Jenkins.instance.getItemByFullName(jobName) // Loop through all instances of this particular job/branch for (def build : currentJob.builds) { if (build.isBuilding() && (build.number.toInteger() < currentBuildNumber)) { echo "Older build still queued. Sending kill signal to build number: ${build.number}" build.doStop() } } } If we put this outside of our node blocks, things are working well. We previously had this in a node{} which was horrible, and resolved the issue after you informing me of the heavyweight vs flyweight executors.

              People

              • Assignee:
                amuniz Antonio Muñiz
                Reporter:
                jglick Jesse Glick
              • Votes:
                9 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated: