Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42224

Need ability to create reusable chunks of Declarative Pipeline

    Details

    • Similar Issues:

      Description

      There is no way to write reusable chunks of Declarative Pipeline configuration.
      Jenkinsfile has no facility for this, and
      shared libraries support only scripted groovy (classes, steps, globals).

        Attachments

          Issue Links

            Activity

            Hide
            abayer Andrew Bayer added a comment -

            This is an interesting and legitimately challenging problem - needs thinking.

            Show
            abayer Andrew Bayer added a comment - This is an interesting and legitimately challenging problem - needs thinking.
            Hide
            abayer Andrew Bayer added a comment -

            Ok, been thinking about this some. =) I think we could avoid some of the hassles of inserting GroovyShellDecoratorImpl all over the place for compile-time validating etc by instead putting the Declarative chunks in the resources directory of libraries and using the libraryResource step to get them as strings - then we can use the same logic as in the validateDeclarativePipeline step to do the validation without worrying about breaking other things outside of the Declarative scope.

            That may or may not make sense. =)

            The next challenge is figuring out how to insert the chunk into the runtime model. Still working on that.

            Show
            abayer Andrew Bayer added a comment - Ok, been thinking about this some. =) I think we could avoid some of the hassles of inserting GroovyShellDecoratorImpl all over the place for compile-time validating etc by instead putting the Declarative chunks in the resources directory of libraries and using the libraryResource step to get them as strings - then we can use the same logic as in the validateDeclarativePipeline step to do the validation without worrying about breaking other things outside of the Declarative scope. That may or may not make sense. =) The next challenge is figuring out how to insert the chunk into the runtime model. Still working on that.
            Hide
            nicolasus Nico Navarrete added a comment -

            Actually we can put some declarative code inside shared libraries, please don't break what is actually working.

             

            Thanks,

            Nico

            Show
            nicolasus Nico Navarrete added a comment - Actually we can put some declarative code inside shared libraries, please don't break what is actually working.   Thanks, Nico
            Hide
            bitwiseman Liam Newman added a comment -

            Nico Navarrete
            Could you describe what you have working or provide a link? That would be very useful information.

            Show
            bitwiseman Liam Newman added a comment - Nico Navarrete Could you describe what you have working or provide a link? That would be very useful information.
            Hide
            abayer Andrew Bayer added a comment -

            So, FYI, I'm beginning to think about some aspects of this - first and foremost the ability to programmatically add stages to a Declarative Pipeline via something coming from a shared library. There are a bunch of technical challenges/questions I need to figure out before this could possibly become reality. We do a lot of things to the pipeline block and its contents at compile-time, so both, say, explicit stage blocks or programmatic generation of a stage block in a shared library would present some real difficulties while using the Declarative syntax.

            What I'm vaguely kicking around now is creating an API with a new syntax specifically for use in shared libraries, which would allow adding stages (at least to start) programmatically without having to deal with the special hells of compile-time transformation. So...we'll see.

            Show
            abayer Andrew Bayer added a comment - So, FYI, I'm beginning to think about some aspects of this - first and foremost the ability to programmatically add stages to a Declarative Pipeline via something coming from a shared library. There are a bunch of technical challenges/questions I need to figure out before this could possibly become reality. We do a lot of things to the pipeline block and its contents at compile-time, so both, say, explicit stage blocks or programmatic generation of a stage block in a shared library would present some real difficulties while using the Declarative syntax. What I'm vaguely kicking around now is creating an API with a new syntax specifically for use in shared libraries, which would allow adding stages (at least to start) programmatically without having to deal with the special hells of compile-time transformation. So...we'll see.
            Hide
            timdowney Tim Downey added a comment -

            Hi Andrew Bayer, thanks for looking at this.  Not sure if this helps provide context or not, but here's the use case that I'm dealing with.

            Basically, declarative pipeline is great if you have to define a single (or few) build.  In my case, I'm basically trying to set some standards that will apply across dozens or maybe hundreds of builds where I don't want to copy/paste 100+ lines of a declarative Jenkinsfile across projects.

            I'm looking for stuff like this:

            def config = [foo: 'foo", bar: 'bar']
            buildNpmApp(config)

            or

            def config = [foo: 'foo", bar: 'bar']
            buildSpringBootAppWithDocker(config)
            

            Now, I know that I can do that using global pipeline libraries now, but I'd like to be able to use the clean, simplified nature of declarative in the global pipeline library.  The notation is hard to follow for folks when using the more raw nature of the global pipeline libs (even when using var)

            So, I'm sort of thinking that in my case, even if I could stuff a whole pipeline {...} inside of a shared lib, I'd be in good shape.  Being able to mix and match steps and stages would be even better, but a whole pipeline would probably suffice.

            Tim

            Show
            timdowney Tim Downey added a comment - Hi Andrew Bayer , thanks for looking at this.  Not sure if this helps provide context or not, but here's the use case that I'm dealing with. Basically, declarative pipeline is great if you have to define a single (or few) build.  In my case, I'm basically trying to set some standards that will apply across dozens or maybe hundreds of builds where I don't want to copy/paste 100+ lines of a declarative Jenkinsfile across projects. I'm looking for stuff like this: def config = [foo: 'foo", bar: ' bar'] buildNpmApp(config) or def config = [foo: 'foo", bar: ' bar'] buildSpringBootAppWithDocker(config) Now, I know that I can do that using global pipeline libraries now, but I'd like to be able to use the clean, simplified nature of declarative in the global pipeline library.  The notation is hard to follow for folks when using the more raw nature of the global pipeline libs (even when using var) So, I'm sort of thinking that in my case, even if I could stuff a whole pipeline {...} inside of a shared lib, I'd be in good shape.  Being able to mix and match steps and stages would be even better, but a whole pipeline would probably suffice. Tim
            Hide
            pleibiger Peter Leibiger added a comment -

            Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.

             

            pipeline {
              libraries {
                lib("mylib@master")
              }
            
              ...
            
              stages {
                stage('Checkout') {
                  steps {
                    gitCheckout()
                  }
                }
            
                stage('Build & Verify') {
                  when {
                     anyOf {
                       branch 'master'
                       branch 'PR**'
                       branch 'feature/**'
                    }
                  }
                  steps {
                     mavenVerify('-Pprod')
                  }
                }
            
                stage('Build & Deploy') {
                  when {
                    branch 'develop'
                  }
                  steps {
                    mavenDeploy('-Pprod')
                  }
                }
            
                stage('Build & Release') {
                  when {
                    branch 'release/**'
                  }
                  steps {
                    mavenRelease('-Pprod,release')
                  }
                }
            
                stage('Sonar Analysis') {
                  when {
                    anyOf {
                      branch 'master'
                      branch 'develop'
                      branch 'PR**'
                    }
                  }
                  steps {
                    sonar()
                  }
                }
            
                stage("Sonar Quality Gate"){
                  agent none
                  when {
                    anyOf {
                      branch 'master'
                      branch 'develop'
                    }
                  }
                  steps {
                    timeout(time: 1, unit: 'HOURS') {
                      script {
                        def qg = waitForQualityGate()
                        echo "Sonar quality gate status: ${qg.status}"
                      }
                    }
                  }  
                }
            
                stage('Post Analysis') {
                  steps {
                    script {
                      echo "Collecting data..."
                    }
                  }
                  post {
                    always {
                      script {
                        collectData()
                      }
                    }
                  }
                }
              }
            
              post {
                changed {
                  script {
                    sendBuildStatusChangedNotifications()
                  }
                }
              }
            }
            

             

            In an ideal world, it would become something like this.

             

            pipeline {
              libraries {
                lib("mylib@master")
              }
            
              ...
            
              stages {
                stage gitCheckout()
                stage mavenVerify('-Pprod')
                stage mavenDeploy('-Pprod')
                stage mavenRelease('-Pprod,release')
                stage sonar()
                stage sonarQualityGate()
                stage collectData()
              }
            
              // or better :)
              stages myCustomMavenBuild(config)
              stages myCustomNpmBuild(config)
            
              post sendBuildStatusChangedNotifications()
            }
            

             

             

            Show
            pleibiger Peter Leibiger added a comment - Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage( 'Checkout' ) { steps { gitCheckout() } } stage( 'Build & Verify' ) { when { anyOf { branch 'master' branch 'PR**' branch 'feature/**' } } steps { mavenVerify( '-Pprod' ) } } stage( 'Build & Deploy' ) { when { branch 'develop' } steps { mavenDeploy( '-Pprod' ) } } stage( 'Build & Release' ) { when { branch 'release/**' } steps { mavenRelease( '-Pprod,release' ) } } stage( 'Sonar Analysis' ) { when { anyOf { branch 'master' branch 'develop' branch 'PR**' } } steps { sonar() } } stage( "Sonar Quality Gate" ){ agent none when { anyOf { branch 'master' branch 'develop' } } steps { timeout(time: 1, unit: 'HOURS' ) { script { def qg = waitForQualityGate() echo "Sonar quality gate status: ${qg.status}" } } } } stage( 'Post Analysis' ) { steps { script { echo "Collecting data..." } } post { always { script { collectData() } } } } } post { changed { script { sendBuildStatusChangedNotifications() } } } }   In an ideal world, it would become something like this.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage gitCheckout() stage mavenVerify( '-Pprod' ) stage mavenDeploy( '-Pprod' ) stage mavenRelease( '-Pprod,release' ) stage sonar() stage sonarQualityGate() stage collectData() } // or better :) stages myCustomMavenBuild(config) stages myCustomNpmBuild(config) post sendBuildStatusChangedNotifications() }    
            Hide
            nicolasus Nico Navarrete added a comment -

            Hi, 

             

            actually we're reusing bits of declarative pipelines doing like this:

             

            • In a shared library, inside the vars dir web declare the common code, ReleaseBoot.groovy:
            import x.y.z.jenkins.spring.boot.DeployConfig;
            
            def call(body) {
            
                def mail = new x.y.z.jenkins.notifications.MailSender(this)
            
                // evaluate the body block, and collect configuration into the object
                def config = [:]
                body.resolveStrategy = Closure.DELEGATE_FIRST
                body.delegate = config
                body()
            
                pipeline {
                    options{   
                        skipDefaultCheckout()   
                    }
                    agent none
                    // node
                    stages {
                        stage('Input Parameters'){
                            agent any
                            steps{
                                timeout(time:1, unit:'DAYS') {
                                    script {
                                        def inputVar = input id: 'RELEASE_INPUT', message: '¿Release new version?', ok: 'Yes', parameters: [
                                            string(defaultValue: '', description: 'Release version:', name: 'release'),
                                            string(defaultValue: '', description: 'Development version:', name: 'dev'),
                                            string(defaultValue: '', description: 'Commit:', name: 'commit')
                                        ], submitterParameter: 'user'
                                        env.RELEASE_VERSION = inputVar['release']
                                        env.BUILD_COMMIT     = inputVar['commit']
                                        env.DEV_VERSION = inputVar['dev']
                                    }
                                }
                            }
                        }
                        stage('scm') {
                            agent any
                            steps {
                                echo "Skip scm is ${config.SKIP.SCM}"
                                deleteDir()
                                checkout scm
                                script {
                                    ........
                                }
                                
                                
                            }
                        }
            
                        
                        stage('Release') {
                            agent any
                            steps {
                                 withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: "${env.GIT_CREDENTIALS}",
                                        usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                                    ............
                                    stash name:"jars", includes:"**/target/*${env.RELEASE_VERSION}*.jar"
                                    
                                    script{........
                                    }
                                }
                                
                            
                            
                                
                            }
                        }
                    }
                    post {
                        failure {
                            script{ mail.send(config.MAILS)}
                        }
                        unstable {
                            script{ mail.send(config.MAILS) }
                        }
                    }
                }
            }
            • In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy

             

            def call(body) {
            
            
                env.DEPLOY_ENV            = "test"
                
                // evaluate the body block, and collect configuration into the object
                def config = [:]
                body.resolveStrategy = Closure.DELEGATE_FIRST
                body.delegate = config
                body()
                
                if(env.JOB_NAME.contains('deploy')) {
                    ........
                }else if(env.JOB_NAME.contains('release')) {
                    // here we call the previous defined 'shared' pipeline
                    ReleaseBoot {
                        MAILS     = config.MAILS
                        DEPLOYMENTS = config.DEPLOYMENTS
                    }
                } else {
                   .............
                }
                
            }
            
            • And  in each project's Jekinsfile:
            #!groovy
            @Library('xyz-pipelines') _
            def OPTIONAL_MAILS = []
            def deployTest     = [user:"targetuser", host:"host", credentialsId:"jboss-dev",
                               // config values
                               checkUrl:"http://xyz:8096/health", infoUrl:"http://xyz:8096/info"]
            // defined in shared library vars dir            
            BootPipeline{
                MAILS = OPTIONAL_MAILS
                DEPLOYMENTS = deployTest
            }        
               
            

            We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working

             

            Please keep this working

             

             NIco

             

             

             

             

            Show
            nicolasus Nico Navarrete added a comment - Hi,    actually we're reusing bits of declarative pipelines doing like this:   In a shared library, inside the vars dir web declare the common code, ReleaseBoot .groovy: import x.y.z.jenkins.spring.boot.DeployConfig; def call(body) {     def mail = new x.y.z.jenkins.notifications.MailSender( this )      // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()     pipeline {         options{                skipDefaultCheckout()            }         agent none          // node         stages {             stage( 'Input Parameters' ){                 agent any                 steps{                     timeout(time:1, unit: 'DAYS' ) {                         script {                             def inputVar = input id: 'RELEASE_INPUT' , message: '¿Release new version?' , ok: 'Yes' , parameters: [                                 string(defaultValue: '', description: ' Release version: ', name: ' release'),                                 string(defaultValue: '', description: ' Development version: ', name: ' dev'),                                 string(defaultValue: '', description: ' Commit: ', name: ' commit')                             ], submitterParameter: 'user'                             env.RELEASE_VERSION = inputVar[ 'release' ]                             env.BUILD_COMMIT     = inputVar[ 'commit' ]                             env.DEV_VERSION = inputVar[ 'dev' ]                         }                     }                 }             }             stage( 'scm' ) {                 agent any                 steps {                     echo "Skip scm is ${config.SKIP.SCM}"                     deleteDir()                     checkout scm                     script {                         ........                     }                                                           }             }                          stage( 'Release' ) {                 agent any                 steps {                      withCredentials([[$class: 'UsernamePasswordMultiBinding' , credentialsId: "${env.GIT_CREDENTIALS}" ,                             usernameVariable: 'USERNAME' , passwordVariable: 'PASSWORD' ]]) {                         ............                         stash name: "jars" , includes: "**/target/*${env.RELEASE_VERSION}*.jar"                                                  script{........                         }                     }                                                                                             }             }         }         post {             failure {                 script{ mail.send(config.MAILS)}             }             unstable {                 script{ mail.send(config.MAILS) }             }         }     } } In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy   def call(body) {     env.DEPLOY_ENV            = "test"           // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()           if (env.JOB_NAME.contains( 'deploy' )) {         ........     } else if (env.JOB_NAME.contains( 'release' )) { // here we call the previous defined 'shared' pipeline         ReleaseBoot {             MAILS     = config.MAILS             DEPLOYMENTS = config.DEPLOYMENTS         }     } else {        .............     }      } And  in each project's Jekinsfile: #!groovy @Library( 'xyz-pipelines' ) _ def OPTIONAL_MAILS = [] def deployTest     = [user: "targetuser" , host: "host" , credentialsId: "jboss-dev" ,                    // config values                    checkUrl: "http: //xyz:8096/health" , infoUrl: "http://xyz:8096/info" ] // defined in shared library vars dir             BootPipeline{     MAILS = OPTIONAL_MAILS     DEPLOYMENTS = deployTest }             We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working   Please keep this working    NIco        
            Hide
            abayer Andrew Bayer added a comment -

            Nico Navarrete - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547, FYI.

            Show
            abayer Andrew Bayer added a comment - Nico Navarrete - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547 , FYI.
            Hide
            fxnn Felix Neumann added a comment - - edited

            Andrew Bayer: -JENKINS-46547- was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp.

            Therefore I'd like to see some kind of composable pipelines, where one could have a basic

            pipeline("my-shared-pipeline") {
              agent { /* ... */ }
              tools { /* ... */ }
              options { /* ... */ }
              stages {
                stage('Compile and Test') { /* ... */ }
                stage('Deploy') { /* ... */ }
              }
              post {
                failure { /* ... */ }
                alway { /* ... */ }
              }
            }
            

            and then extend it, using some mechanism to make up the final order of stages:

            pipeline(extends: "my-shared-pipeline") {
              stages {
                stage('Integration Test', after: 'Compile and Test') { /* ... */ }
              }
            }
            

            This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds.

            Btw., loading chunks from resources (as proposed above) also sounds like a good solution.

            Just my 5 Cents – thanks for making all this possible!

            Show
            fxnn Felix Neumann added a comment - - edited Andrew Bayer : - JENKINS-46547 - was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp. Therefore I'd like to see some kind of composable pipelines, where one could have a basic pipeline( "my-shared-pipeline" ) { agent { /* ... */ } tools { /* ... */ } options { /* ... */ } stages { stage( 'Compile and Test' ) { /* ... */ } stage( 'Deploy' ) { /* ... */ } } post { failure { /* ... */ } alway { /* ... */ } } } and then extend it, using some mechanism to make up the final order of stages: pipeline( extends : "my-shared-pipeline" ) { stages { stage( 'Integration Test' , after: 'Compile and Test' ) { /* ... */ } } } This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds. Btw., loading chunks from resources (as proposed above) also sounds like a good solution. Just my 5 Cents – thanks for making all this possible!
            Hide
            tobilarscheid Tobias Larscheid added a comment -

            I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548 for defining reusable stages, maybe it is interesting for you as well.

            Show
            tobilarscheid Tobias Larscheid added a comment - I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548  for defining reusable stages, maybe it is interesting for you as well.
            Hide
            nroose Nick Roosevelt added a comment -

            I would like this as well. But also, it seems like we can make some of those blocks one line, and we could use a preprocessor, if necessary. Just starting to get into this, but seems like groovy should support this already.

            Show
            nroose Nick Roosevelt added a comment - I would like this as well. But also, it seems like we can make some of those blocks one line, and we could use a preprocessor, if necessary. Just starting to get into this, but seems like groovy should support this already.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                bitwiseman Liam Newman
              • Votes:
                67 Vote for this issue
                Watchers:
                79 Start watching this issue

                Dates

                • Created:
                  Updated: