Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42224

Need ability to create reusable chunks of Declarative Pipeline

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      There is no way to write reusable chunks of Declarative Pipeline configuration.
      Jenkinsfile has no facility for this, and
      shared libraries support only scripted groovy (classes, steps, globals).

        Attachments

          Issue Links

            Activity

            Hide
            pleibiger Peter Leibiger added a comment -

            Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.

             

            pipeline {
              libraries {
                lib("mylib@master")
              }
            
              ...
            
              stages {
                stage('Checkout') {
                  steps {
                    gitCheckout()
                  }
                }
            
                stage('Build & Verify') {
                  when {
                     anyOf {
                       branch 'master'
                       branch 'PR**'
                       branch 'feature/**'
                    }
                  }
                  steps {
                     mavenVerify('-Pprod')
                  }
                }
            
                stage('Build & Deploy') {
                  when {
                    branch 'develop'
                  }
                  steps {
                    mavenDeploy('-Pprod')
                  }
                }
            
                stage('Build & Release') {
                  when {
                    branch 'release/**'
                  }
                  steps {
                    mavenRelease('-Pprod,release')
                  }
                }
            
                stage('Sonar Analysis') {
                  when {
                    anyOf {
                      branch 'master'
                      branch 'develop'
                      branch 'PR**'
                    }
                  }
                  steps {
                    sonar()
                  }
                }
            
                stage("Sonar Quality Gate"){
                  agent none
                  when {
                    anyOf {
                      branch 'master'
                      branch 'develop'
                    }
                  }
                  steps {
                    timeout(time: 1, unit: 'HOURS') {
                      script {
                        def qg = waitForQualityGate()
                        echo "Sonar quality gate status: ${qg.status}"
                      }
                    }
                  }  
                }
            
                stage('Post Analysis') {
                  steps {
                    script {
                      echo "Collecting data..."
                    }
                  }
                  post {
                    always {
                      script {
                        collectData()
                      }
                    }
                  }
                }
              }
            
              post {
                changed {
                  script {
                    sendBuildStatusChangedNotifications()
                  }
                }
              }
            }
            

             

            In an ideal world, it would become something like this.

             

            pipeline {
              libraries {
                lib("mylib@master")
              }
            
              ...
            
              stages {
                stage gitCheckout()
                stage mavenVerify('-Pprod')
                stage mavenDeploy('-Pprod')
                stage mavenRelease('-Pprod,release')
                stage sonar()
                stage sonarQualityGate()
                stage collectData()
              }
            
              // or better :)
              stages myCustomMavenBuild(config)
              stages myCustomNpmBuild(config)
            
              post sendBuildStatusChangedNotifications()
            }
            

             

             

            Show
            pleibiger Peter Leibiger added a comment - Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage( 'Checkout' ) { steps { gitCheckout() } } stage( 'Build & Verify' ) { when { anyOf { branch 'master' branch 'PR**' branch 'feature/**' } } steps { mavenVerify( '-Pprod' ) } } stage( 'Build & Deploy' ) { when { branch 'develop' } steps { mavenDeploy( '-Pprod' ) } } stage( 'Build & Release' ) { when { branch 'release/**' } steps { mavenRelease( '-Pprod,release' ) } } stage( 'Sonar Analysis' ) { when { anyOf { branch 'master' branch 'develop' branch 'PR**' } } steps { sonar() } } stage( "Sonar Quality Gate" ){ agent none when { anyOf { branch 'master' branch 'develop' } } steps { timeout(time: 1, unit: 'HOURS' ) { script { def qg = waitForQualityGate() echo "Sonar quality gate status: ${qg.status}" } } } } stage( 'Post Analysis' ) { steps { script { echo "Collecting data..." } } post { always { script { collectData() } } } } } post { changed { script { sendBuildStatusChangedNotifications() } } } }   In an ideal world, it would become something like this.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage gitCheckout() stage mavenVerify( '-Pprod' ) stage mavenDeploy( '-Pprod' ) stage mavenRelease( '-Pprod,release' ) stage sonar() stage sonarQualityGate() stage collectData() } // or better :) stages myCustomMavenBuild(config) stages myCustomNpmBuild(config) post sendBuildStatusChangedNotifications() }    
            Hide
            nicolasus Nico Navarrete added a comment -

            Hi, 

             

            actually we're reusing bits of declarative pipelines doing like this:

             

            • In a shared library, inside the vars dir web declare the common code, ReleaseBoot.groovy:
            import x.y.z.jenkins.spring.boot.DeployConfig;
            
            def call(body) {
            
                def mail = new x.y.z.jenkins.notifications.MailSender(this)
            
                // evaluate the body block, and collect configuration into the object
                def config = [:]
                body.resolveStrategy = Closure.DELEGATE_FIRST
                body.delegate = config
                body()
            
                pipeline {
                    options{   
                        skipDefaultCheckout()   
                    }
                    agent none
                    // node
                    stages {
                        stage('Input Parameters'){
                            agent any
                            steps{
                                timeout(time:1, unit:'DAYS') {
                                    script {
                                        def inputVar = input id: 'RELEASE_INPUT', message: '¿Release new version?', ok: 'Yes', parameters: [
                                            string(defaultValue: '', description: 'Release version:', name: 'release'),
                                            string(defaultValue: '', description: 'Development version:', name: 'dev'),
                                            string(defaultValue: '', description: 'Commit:', name: 'commit')
                                        ], submitterParameter: 'user'
                                        env.RELEASE_VERSION = inputVar['release']
                                        env.BUILD_COMMIT     = inputVar['commit']
                                        env.DEV_VERSION = inputVar['dev']
                                    }
                                }
                            }
                        }
                        stage('scm') {
                            agent any
                            steps {
                                echo "Skip scm is ${config.SKIP.SCM}"
                                deleteDir()
                                checkout scm
                                script {
                                    ........
                                }
                                
                                
                            }
                        }
            
                        
                        stage('Release') {
                            agent any
                            steps {
                                 withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: "${env.GIT_CREDENTIALS}",
                                        usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                                    ............
                                    stash name:"jars", includes:"**/target/*${env.RELEASE_VERSION}*.jar"
                                    
                                    script{........
                                    }
                                }
                                
                            
                            
                                
                            }
                        }
                    }
                    post {
                        failure {
                            script{ mail.send(config.MAILS)}
                        }
                        unstable {
                            script{ mail.send(config.MAILS) }
                        }
                    }
                }
            }
            • In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy

             

            def call(body) {
            
            
                env.DEPLOY_ENV            = "test"
                
                // evaluate the body block, and collect configuration into the object
                def config = [:]
                body.resolveStrategy = Closure.DELEGATE_FIRST
                body.delegate = config
                body()
                
                if(env.JOB_NAME.contains('deploy')) {
                    ........
                }else if(env.JOB_NAME.contains('release')) {
                    // here we call the previous defined 'shared' pipeline
                    ReleaseBoot {
                        MAILS     = config.MAILS
                        DEPLOYMENTS = config.DEPLOYMENTS
                    }
                } else {
                   .............
                }
                
            }
            
            • And  in each project's Jekinsfile:
            #!groovy
            @Library('xyz-pipelines') _
            def OPTIONAL_MAILS = []
            def deployTest     = [user:"targetuser", host:"host", credentialsId:"jboss-dev",
                               // config values
                               checkUrl:"http://xyz:8096/health", infoUrl:"http://xyz:8096/info"]
            // defined in shared library vars dir            
            BootPipeline{
                MAILS = OPTIONAL_MAILS
                DEPLOYMENTS = deployTest
            }        
               
            

            We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working

             

            Please keep this working

             

             NIco

             

             

             

             

            Show
            nicolasus Nico Navarrete added a comment - Hi,    actually we're reusing bits of declarative pipelines doing like this:   In a shared library, inside the vars dir web declare the common code, ReleaseBoot .groovy: import x.y.z.jenkins.spring.boot.DeployConfig; def call(body) {     def mail = new x.y.z.jenkins.notifications.MailSender( this )      // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()     pipeline {         options{                skipDefaultCheckout()            }         agent none          // node         stages {             stage( 'Input Parameters' ){                 agent any                 steps{                     timeout(time:1, unit: 'DAYS' ) {                         script {                             def inputVar = input id: 'RELEASE_INPUT' , message: '¿Release new version?' , ok: 'Yes' , parameters: [                                 string(defaultValue: '', description: ' Release version: ', name: ' release'),                                 string(defaultValue: '', description: ' Development version: ', name: ' dev'),                                 string(defaultValue: '', description: ' Commit: ', name: ' commit')                             ], submitterParameter: 'user'                             env.RELEASE_VERSION = inputVar[ 'release' ]                             env.BUILD_COMMIT     = inputVar[ 'commit' ]                             env.DEV_VERSION = inputVar[ 'dev' ]                         }                     }                 }             }             stage( 'scm' ) {                 agent any                 steps {                     echo "Skip scm is ${config.SKIP.SCM}"                     deleteDir()                     checkout scm                     script {                         ........                     }                                                           }             }                          stage( 'Release' ) {                 agent any                 steps {                      withCredentials([[$class: 'UsernamePasswordMultiBinding' , credentialsId: "${env.GIT_CREDENTIALS}" ,                             usernameVariable: 'USERNAME' , passwordVariable: 'PASSWORD' ]]) {                         ............                         stash name: "jars" , includes: "**/target/*${env.RELEASE_VERSION}*.jar"                                                  script{........                         }                     }                                                                                             }             }         }         post {             failure {                 script{ mail.send(config.MAILS)}             }             unstable {                 script{ mail.send(config.MAILS) }             }         }     } } In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy   def call(body) {     env.DEPLOY_ENV            = "test"           // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()           if (env.JOB_NAME.contains( 'deploy' )) {         ........     } else if (env.JOB_NAME.contains( 'release' )) { // here we call the previous defined 'shared' pipeline         ReleaseBoot {             MAILS     = config.MAILS             DEPLOYMENTS = config.DEPLOYMENTS         }     } else {        .............     }      } And  in each project's Jekinsfile: #!groovy @Library( 'xyz-pipelines' ) _ def OPTIONAL_MAILS = [] def deployTest     = [user: "targetuser" , host: "host" , credentialsId: "jboss-dev" ,                    // config values                    checkUrl: "http: //xyz:8096/health" , infoUrl: "http://xyz:8096/info" ] // defined in shared library vars dir             BootPipeline{     MAILS = OPTIONAL_MAILS     DEPLOYMENTS = deployTest }             We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working   Please keep this working    NIco        
            Hide
            abayer Andrew Bayer added a comment -

            Nico Navarrete - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547, FYI.

            Show
            abayer Andrew Bayer added a comment - Nico Navarrete - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547 , FYI.
            Hide
            fxnn Felix Neumann added a comment - - edited

            Andrew Bayer: -JENKINS-46547- was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp.

            Therefore I'd like to see some kind of composable pipelines, where one could have a basic

            pipeline("my-shared-pipeline") {
              agent { /* ... */ }
              tools { /* ... */ }
              options { /* ... */ }
              stages {
                stage('Compile and Test') { /* ... */ }
                stage('Deploy') { /* ... */ }
              }
              post {
                failure { /* ... */ }
                alway { /* ... */ }
              }
            }
            

            and then extend it, using some mechanism to make up the final order of stages:

            pipeline(extends: "my-shared-pipeline") {
              stages {
                stage('Integration Test', after: 'Compile and Test') { /* ... */ }
              }
            }
            

            This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds.

            Btw., loading chunks from resources (as proposed above) also sounds like a good solution.

            Just my 5 Cents – thanks for making all this possible!

            Show
            fxnn Felix Neumann added a comment - - edited Andrew Bayer : - JENKINS-46547 - was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp. Therefore I'd like to see some kind of composable pipelines, where one could have a basic pipeline( "my-shared-pipeline" ) { agent { /* ... */ } tools { /* ... */ } options { /* ... */ } stages { stage( 'Compile and Test' ) { /* ... */ } stage( 'Deploy' ) { /* ... */ } } post { failure { /* ... */ } alway { /* ... */ } } } and then extend it, using some mechanism to make up the final order of stages: pipeline( extends : "my-shared-pipeline" ) { stages { stage( 'Integration Test' , after: 'Compile and Test' ) { /* ... */ } } } This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds. Btw., loading chunks from resources (as proposed above) also sounds like a good solution. Just my 5 Cents – thanks for making all this possible!
            Hide
            tobilarscheid Tobias Larscheid added a comment -

            I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548 for defining reusable stages, maybe it is interesting for you as well.

            Show
            tobilarscheid Tobias Larscheid added a comment - I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548  for defining reusable stages, maybe it is interesting for you as well.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                bitwiseman Liam Newman
              • Votes:
                56 Vote for this issue
                Watchers:
                67 Start watching this issue

                Dates

                • Created:
                  Updated: