Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40936

Milestone should be able to abort older builds when reaching the same milestone

    Details

    • Similar Issues:

      Description

      When using a deployment strategy with an user input option older builds are cancelled as soon as the user acknowledges the input. That's working pretty well, however:
      Frequent commits will clock up the executors all waiting on the same input if the user does not acknowledge 'quick' enough. We might be able to optimise this and free executors by automatically aborting old builds which are all sitting in the same milestone.

      build #1

      • milestone 1
      • input
        -milestone 2

      build #2 (newer commit)

      • milestone 1
      • input
      • milestone 2

      When build #2 reaches the input it's effectively in milestone 1, where #1 is currently sitting as well. It could be very good if at this point we would be able to abort #1 as we have a new build in the same milestone.

      I was not able to find a solution achieving above behaviour, but if that's otherwise possible, I would be very happy to implement it, although I believe it would be a good addition to excellent milestone pattern readily available.

        Attachments

          Issue Links

            Activity

            Hide
            hrmpw Patrick Wolf added a comment -

            Depending on how many commits and runs are stacked up the lock step from the lockable resources plugin could help here:

            milestone()
            lock(resource: 'my-pipeline', inversePrecedence: true) {
              milestone()
              input "proceed?"
            }
            

            In this case all of the builds will still stack up at the lock but they will get aborted much quicker. Assume you have 6 commits:

            Build 1 will enter the lock and wait at the input. All other builds after that will wait at the lock.

            When Build 1 passes the input the newest build waiting at the lock (say Build 5) enters the lock, passes the new milestone and waits at the input. As soon as Build 5 passes the new milestone all other builds (Builds 2,3,4) are aborted while Build 5 waits at the input

            This doesn't eliminate the queue but it can dramatically reduce it if you have several builds waiting.

            Show
            hrmpw Patrick Wolf added a comment - Depending on how many commits and runs are stacked up the lock step from the lockable resources plugin could help here: milestone() lock(resource: 'my-pipeline' , inversePrecedence: true ) { milestone() input "proceed?" } In this case all of the builds will still stack up at the lock but they will get aborted much quicker. Assume you have 6 commits: Build 1 will enter the lock and wait at the input . All other builds after that will wait at the lock . When Build 1 passes the input the newest build waiting at the lock (say Build 5) enters the lock , passes the new milestone and waits at the input . As soon as Build 5 passes the new milestone all other builds (Builds 2,3,4) are aborted while Build 5 waits at the input This doesn't eliminate the queue but it can dramatically reduce it if you have several builds waiting.
            Hide
            jeroen_muis jeroen_muis added a comment -

            Hi Patrick,

            Thanks for the suggestion, but unfortunately it won't help enough as the number of testers against the number of commits simply doesn't stack up, and they won't be testing the 'latest and greatest'.

            Best regards,
            Jeroen

            Show
            jeroen_muis jeroen_muis added a comment - Hi Patrick, Thanks for the suggestion, but unfortunately it won't help enough as the number of testers against the number of commits simply doesn't stack up, and they won't be testing the 'latest and greatest'. Best regards, Jeroen
            Hide
            jasonwzs Jason Wen added a comment -

            +1 on this improvement. I am going to file same ER and find this issue has already been filed.

            The use case that Jeroen explains is exactly same use case I have. I think this could be a typical use case to create a deployment pipeline involving manual approval of promoting deployment from QA to staging and from staging to production environment.

            The use case is a newer build reach the same milestone that an old build is waiting at for manual confirmation. In this case, the new build has done all the things the old build did, conceptually it can be regarded as 'passed' the old build, so the old build should be aborted.

            I would suggest adding an optional parameter to support this behavior, e.g
            abortSameMilestone: If the value is true the older builds waiting at a milestone will be aborted when a new build reach the same milestone. The default value is false.

            I am also happy to implement this ER if it is approved.

            Show
            jasonwzs Jason Wen added a comment - +1 on this improvement. I am going to file same ER and find this issue has already been filed. The use case that Jeroen explains is exactly same use case I have. I think this could be a typical use case to create a deployment pipeline involving manual approval of promoting deployment from QA to staging and from staging to production environment. The use case is a newer build reach the same milestone that an old build is waiting at for manual confirmation. In this case, the new build has done all the things the old build did, conceptually it can be regarded as 'passed' the old build, so the old build should be aborted. I would suggest adding an optional parameter to support this behavior, e.g abortSameMilestone: If the value is true the older builds waiting at a milestone will be aborted when a new build reach the same milestone. The default value is false. I am also happy to implement this ER if it is approved.
            Hide
            jglick Jesse Glick added a comment -

            Frequent commits will clock up the executors all waiting on the same input

            Do not use input inside node (in most cases doing so is a bad idea!) and there is no real problem: there will be a bunch of old builds “running”, but there is no cost to them—Jenkins is idle. As soon as someone approves the newest build (or one of the newer ones), the previous ones will get aborted automatically.

            The downside to eagerly aborting older builds as soon as you get to the prompt in a newer build is that it does not guarantee progress can be made: someone looks at a build pending input, starts to do some sanity checks, and then it is canceled in favor of a new one before they can approve, and the cycle repeats. This may or may not be a problem, depending on how long manual testing takes, and how frequently fresh builds appear.

            Show
            jglick Jesse Glick added a comment - Frequent commits will clock up the executors all waiting on the same input Do not use input inside node (in most cases doing so is a bad idea!) and there is no real problem: there will be a bunch of old builds “running”, but there is no cost to them—Jenkins is idle. As soon as someone approves the newest build (or one of the newer ones), the previous ones will get aborted automatically. The downside to eagerly aborting older builds as soon as you get to the prompt in a newer build is that it does not guarantee progress can be made: someone looks at a build pending input, starts to do some sanity checks, and then it is canceled in favor of a new one before they can approve, and the cycle repeats. This may or may not be a problem, depending on how long manual testing takes, and how frequently fresh builds appear.
            Hide
            mathieudurand Mathieu Durand added a comment -

            This issue would be really appreciate. I tried to find a workaround without success...

            It looks like the issue is also that an input step cannot be aborted by a milestone trigger.

            In facts, i tried, for testing purpose, to put before my input step this :

             

            milestone(BUILD_NUMBER) 

             

            But the previous build waiting in input step is never being cancelled.

            Show
            mathieudurand Mathieu Durand added a comment - This issue would be really appreciate. I tried to find a workaround without success... It looks like the issue is also that an input step cannot be aborted by a milestone trigger. In facts, i tried, for testing purpose, to put before my input step this :   milestone(BUILD_NUMBER)    But the previous build waiting in input step is never being cancelled.
            Hide
            rupert Rupert Madden-Abbott added a comment - - edited

            If you only ever want the latest build to be considered at the input step, then could you just do this:

            milestone()
            milestone()
            input "proceed?"
            

            The first build will stop at the input step.

            The second build will pass both milestones, aborting the first build, and then stop at the input step.

            Edit: Okay no this doesn't work and I misunderstood how this step works.

            In case it helps other, this doesn't work because the first and second build still pass both milestones in order. When the second build passes the second milestone, the first build has already passed it. Therefore, the second build has not passed a milestone that the first one has not passed, and so the first build is not aborted.

            Show
            rupert Rupert Madden-Abbott added a comment - - edited If you only ever want the latest build to be considered at the input step, then could you just do this: milestone() milestone() input "proceed?" The first build will stop at the input step. The second build will pass both milestones, aborting the first build, and then stop at the input step. Edit : Okay no this doesn't work and I misunderstood how this step works. In case it helps other, this doesn't work because the first and second build still pass both milestones in order. When the second build passes the second milestone, the first build has already passed it. Therefore, the second build has not passed a milestone that the first one has not passed, and so the first build is not aborted.
            Hide
            jdavidx Jonathan David added a comment -

            This works for me:

             

            stage ('1st Stage') {
              milestone ()
              echo "Stage1"
              sh 'sleep 1'
            }

            stage ('2nd Stage') {
              milestone ()
              echo "Stage 2"
              sh 'sleep 1'
            }

            stage ('3rd Stage') {
              for (int i = 0; i < (BUILD_NUMBER as int); i++) {milestone()}
              input ("Proceed")
              node ('javaAgent') {
                echo "something else to do"
                sh 'sleep 1'
              }

              milestone()

            }

            In Stage 3, prior to the input step, loop through all the past build numbers to set the milestone.  Newer builds will have a higher build number and should cancel out previous builds.

            Show
            jdavidx Jonathan David added a comment - This works for me:   stage ('1st Stage') {   milestone ()   echo "Stage1"   sh 'sleep 1' } stage ('2nd Stage') {   milestone ()   echo "Stage 2"   sh 'sleep 1' } stage ('3rd Stage') {   for (int i = 0; i < (BUILD_NUMBER as int); i++) {milestone()}   input ("Proceed")   node ('javaAgent') {     echo "something else to do"     sh 'sleep 1'   }   milestone() } In Stage 3, prior to the input step, loop through all the past build numbers to set the milestone.  Newer builds will have a higher build number and should cancel out previous builds.

              People

              • Assignee:
                amuniz Antonio Muñiz
                Reporter:
                jeroen_muis jeroen_muis
              • Votes:
                5 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated: