Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45140

Add support of throttling of the entire build in Declarative Pipeline

    Details

    • Similar Issues:

      Description

      So, I'm having this problem that I described in a similar bug for the lockable-resource plugin (JENKINS-45138). I said to myself, "oh, hey, I remember being able to throttle executions on a per-agent basis!"

      Imagine my surprise when I hit the documentation and find that throttle is only applicable inside a step.

      I need to acquire, use, and cleanup exclusive access to a resource on each agent. Will throttle work how I expect?

      step('foo') {
          throttle(['foo-label'])
          bat '... acquire the resource...'
          bat '... use the resource...'
      }
      post {
          always {
              bat '... cleanup the resource...'
          }
      }
      

        Attachments

          Activity

          Hide
          marcus_phi Marcus Philip added a comment - - edited

          I tried Dmitry Mamchurs creative suggestion but on more recent versions this is explicitly prohibited:

          WorkflowScript: 2: pipeline block must be at the top-level, not within another block. @ line 2, column 5.
                 pipeline {
                 ^
          

          Demo pipeline:

           throttle(['myThrottle']) {
              pipeline {
                  agent { label 'mylabel' }
                  stages {
                      stage('first') {
                          steps {
                              sleep 60
                          }
                      }
                  }
              }
          }
          

           

          I've tested many other variants and I claim it is currently (with latest versions) impossible to get node throttling on a declarative pipeline. If someone has a counterexample I would appreciate that.

          Show
          marcus_phi Marcus Philip added a comment - - edited I tried Dmitry Mamchur s creative suggestion but on more recent versions this is explicitly prohibited: WorkflowScript: 2: pipeline block must be at the top-level, not within another block. @ line 2, column 5. pipeline { ^ Demo pipeline:  throttle([ 'myThrottle' ]) { pipeline { agent { label 'mylabel' } stages { stage( 'first' ) { steps { sleep 60 } } } } }   I've tested many other variants and I claim it is currently (with latest versions) impossible to get node throttling on a declarative pipeline. If someone has a counterexample I would appreciate that.
          Hide
          metamilk Dmitry Mamchur added a comment -

          Marcus Philip The secret sauce is having your pipeline defined in a shared library.

          vars/myPipeline.groovy

          def call() {
              pipeline {
                  agent { label 'mylabel' }
                  stages {
                      stage('first') {
                          steps {
                              sleep 60
                          }
                      }
                  }
              }
          }

          jobs/my-pipeline/Jenkinsfile

          throttle(['myThrottle']) {
              myPipeline()
          }
          Show
          metamilk Dmitry Mamchur added a comment - Marcus Philip The secret sauce is having your pipeline defined in a shared library. vars/myPipeline.groovy def call() { pipeline { agent { label 'mylabel' } stages { stage( 'first' ) { steps { sleep 60 } } } } } jobs/my-pipeline/Jenkinsfile throttle([ 'myThrottle' ]) { myPipeline() }
          Hide
          marcus_phi Marcus Philip added a comment -

          Je-s F-ng C-st!

          It's insane that that would make a difference but it does.

          So the 'pipeline block must be at the top-level' check is just on a file (textual) basis, not on the actual code structure.

          Thanks Dmitry Mamchur! You made my day!

          Show
          marcus_phi Marcus Philip added a comment - Je-s F-ng C-st! It's insane that that would make a difference but it does. So the ' pipeline block must be at the top-level ' check is just on a file (textual) basis, not on the actual code structure. Thanks Dmitry Mamchur ! You made my day!
          Hide
          jgrant216 Jeff G added a comment - - edited

          I honestly did not have much hope when I saw this issue having been updated in my e-mail this morning.

          Marcus Philip, I'll second your comment/excitement on finally having a solution to this mystery. 

          Dmitry Mamchur, thank you so much! 

          Our team already has our declarative and scripted pipelines largely live in libraries.  I gave a couple attempts to implement the throttle as described by Dmitry Mamchur and here is a little more on how I think I'll `throttle` in our builds by renaming sharedPipeline to sharedPipelineInner and having sharedPipeline wrap the throttle category around the sharedPipeline() call.  This way, I don't need to update 100+ repositories * number of branches. 

          The current Jenkinsfile looks like this more or less...

          // library imports
          sharedPipeline()

          vars/sharedPipeline.groovy

          def call() {
            throttle(['throttle-category']) {
              sharedPipelineInner()
            }
          }
          

          vars/sharedPipelineInner.groovy

          def call() {
            pipeline {
            agent { label 'mylabel' }
            stages {
              stage('first') {
                steps { 
                  sleep 60
                }
              }
            }
          }
          

           I did try to simply shift the pipeline config into a def within the original file, but that did not work.

          This defect asks for throttle within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this.

          Edit - Then again, the title says entire build so maybe this solution does apply and the example in the body does not. Anthony Mastrean, can you comment?

          Show
          jgrant216 Jeff G added a comment - - edited I honestly did not have much hope when I saw this issue having been updated in my e-mail this morning. Marcus Philip , I'll second your comment/excitement on finally having a solution to this mystery.  Dmitry Mamchur , thank you so much!  Our team already has our declarative and scripted pipelines largely live in libraries.  I gave a couple attempts to implement the throttle as described by Dmitry Mamchur and here is a little more on how I think I'll `throttle` in our builds by renaming sharedPipeline to sharedPipelineInner and having sharedPipeline wrap the throttle category around the sharedPipeline() call.  This way, I don't need to update 100+ repositories * number of branches.  The current Jenkinsfile looks like this more or less... // library imports sharedPipeline() vars/sharedPipeline.groovy def call() { throttle([ 'throttle-category' ]) { sharedPipelineInner() } } vars/sharedPipelineInner.groovy def call() { pipeline { agent { label 'mylabel' } stages { stage( 'first' ) { steps { sleep 60 } } } }  I did try to simply shift the pipeline config into a def within the original file, but that did not work. This defect asks for throttle within a step, which this solution does not provide, so I guess it will need to stay open.  The docs for the plugin should be updated, for sure, with an example like this. Edit - Then again, the title says entire build so maybe this solution does apply and the example in the body does not. Anthony Mastrean , can you comment?
          Hide
          chrop Christian Opitz added a comment - - edited

          What seems to work fine for me in a declarative pipeline is the following (syntax taken from here: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/68)

          pipeline {
              agent any    
          
              options {
                  throttleJobProperty(
                      categories: ['test_3'],
                      throttleEnabled: true,
                      throttleOption: 'category',
                  )
              }
              stages {
                  stage('sleep') {
                      steps {
                          bat "sleep 60"
                          echo "Done"
                      }
                  }
              }
          }
          

          Hope that helps.

          Show
          chrop Christian Opitz added a comment - - edited What seems to work fine for me in a declarative pipeline is the following (syntax taken from here: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/68 ) pipeline { agent any options { throttleJobProperty( categories: [ 'test_3' ], throttleEnabled: true , throttleOption: 'category' , ) } stages { stage( 'sleep' ) { steps { bat "sleep 60" echo "Done" } } } } Hope that helps.

            People

            • Assignee:
              Unassigned
              Reporter:
              anthonymastrean Anthony Mastrean
            • Votes:
              25 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

              • Created:
                Updated: