Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50548

Shared Library should be allowed to declare reusable stages

    Details

    • Similar Issues:

      Description

      Currently, we have a lot of reusable steps we use in our Jenkinsfiles like so:

      pipeline {
        stages {
          stage('Build') {
            steps {
              reusableBuild()
            }
          }
          stage('Test') {
            steps {
              reusableTest()
            }
          }
        } 
      }

      I feel it is very repetitive to always replicate the stage definition - the way I look at this, these should be reusable as well. A pipeline would then simply consist of reusable, composable stages:

      pipeline {
        stages {
          reusableBuild()
          reusableTest()
          stage 'Something individual' {
            steps {
              echo 'only for this project'
            }
          }
        } 
      }

      Maybe this is already possible, but at the moment I have no idea how to define reusable stages in a shared library - any hint would be very much appreciated. I would also be willing to provide a PR, if only I had an idea which code to touch

        Attachments

          Issue Links

            Activity

            tobilarscheid Tobias Larscheid created issue -
            tobilarscheid Tobias Larscheid made changes -
            Field Original Value New Value
            Description Currently, we have a lot of reusable steps we use in our Jenkinsfiles like so:
            {code:java}
            pipeline {
              stages {
                stage('Build') {
                  steps {
                    reusableBuild()
                  }
                }
                stage('Test') {
                  steps {
                    reusableTest()
                  }
                }
              }
            }{code}
            I feel it is very repetitive to always replicate the stage definition - the way I look at this, these should be reusable as well. A pipeline would then simply consist of reusable, composable stages:
            {code:java}
            pipeline {
              stages {
                reusableBuild()
                reusableTest()
                stage 'Something individual' {
                  echo 'only for this project'
                }
              }
            }{code}
            Maybe this is already possible, but at the moment I have no idea how to define reusable stages in a shared library - any hint would be very much appreciated. I would also be willing to provide a PR, if only I had an idea which code to touch ;)
            Currently, we have a lot of reusable steps we use in our Jenkinsfiles like so:
            {code:java}
            pipeline {
              stages {
                stage('Build') {
                  steps {
                    reusableBuild()
                  }
                }
                stage('Test') {
                  steps {
                    reusableTest()
                  }
                }
              }
            }{code}
            I feel it is very repetitive to always replicate the stage definition - the way I look at this, these should be reusable as well. A pipeline would then simply consist of reusable, composable stages:
            {code:java}
            pipeline {
              stages {
                reusableBuild()
                reusableTest()
                stage 'Something individual' {
                  steps {
                    echo 'only for this project'
                  }
                }
              }
            }{code}
            Maybe this is already possible, but at the moment I have no idea how to define reusable stages in a shared library - any hint would be very much appreciated. I would also be willing to provide a PR, if only I had an idea which code to touch ;)
            abayer Andrew Bayer made changes -
            Component/s pipeline-model-definition-plugin [ 21706 ]
            Component/s pipeline [ 21692 ]
            Hide
            jalvarez Jesus Alvarez added a comment -

            I spent all day trying do this with Declarative Pipelines and finally had to give up. It was so easy to do things like this with Scripted Pipelines, but Declarative is the way going forward. It would be nice to have this with the shared pipeline library.

            Show
            jalvarez Jesus Alvarez added a comment - I spent all day trying do this with Declarative Pipelines and finally had to give up. It was so easy to do things like this with Scripted Pipelines, but Declarative is the way going forward. It would be nice to have this with the shared pipeline library.
            Hide
            onhate Marcelo Luiz Onhate added a comment - - edited

            Hello,

            As a suggestion you can use the (not well documented) feature that loads scripts on the fly.

            #!groovy
            script {
                library identifier: "setup@$BRANCH_NAME", retriever: legacySCM(scm)
            }
            pipeline {
                stages {
                    stage('Setup & Deploy') {
                        steps {
                            setup()
                            deploy()
                        }
                    }
                }
            }
            

            this will load the libs from the same repository and branch you are checking out, it will go to folder /vars/ and load all .groovy files.

            library identifier: "setup@$BRANCH_NAME", retriever: legacySCM(scm)
            
            <ROOT>/
                - vars/
                    - deploy.groovy
                    - setup.groovy
            

            The name of the .groovy script will become a step on pipeline, as you can see the steps setup and deploy.

            setup.groovy

            #!/usr/bin/env groovy
            def call() {
                sh "echo 'YEAH'"
                sh "some/other/script.sh"
            }
            

            deploy.groovy

            #!/usr/bin/env groovy
            def call() {
                def props = readProperties file: 'environment.env'
                env.PROP_1 = props['PROP_1']
            
                sh "script/that/uses/PROP_1"
            }
            

             

            Show
            onhate Marcelo Luiz Onhate added a comment - - edited Hello, As a suggestion you can use the (not well documented) feature that loads scripts on the fly. #!groovy script { library identifier: "setup@$BRANCH_NAME" , retriever: legacySCM(scm) } pipeline { stages { stage( 'Setup & Deploy' ) { steps { setup() deploy() } } } } this will load the libs from the same repository and branch you are checking out, it will go to folder /vars/ and load all .groovy files. library identifier: "setup@$BRANCH_NAME", retriever: legacySCM(scm) <ROOT>/ - vars/ - deploy.groovy - setup.groovy The name of the .groovy script will become a step on pipeline, as you can see the steps setup and deploy . setup.groovy #!/usr/bin/env groovy def call() { sh "echo 'YEAH' " sh "some/other/script.sh" } deploy.groovy #!/usr/bin/env groovy def call() { def props = readProperties file: 'environment.env' env.PROP_1 = props[ 'PROP_1' ] sh "script/that/uses/PROP_1" }  
            Hide
            stevenfoster Steven Foster added a comment -

            Declarative stages have become so flexible now: they can have their own options, environment, post stages, agent etc.

            This makes the ability to reuse and share them even more powerful. My use case is running tests against a selection of specialised hardware, accessible by a generic build agent. In conjunction with the lockable resources plugin, the typical stage for this in the middle of a pipeline looks something like:

            stage ('Run tests on kit') {
              agent { label 'local-kits' }
              options {
                lock(label: 'kit-type-1', variable: 'KIT_IP', quantity: 1)
              }
              steps {
                runTest('example-test.zip') // existing shared function to download and run previously archived binary on reserved kit
              }
              post {
                always {
                  junit 'results.xml'
                }
              }
            }

            I use this in a ton of places, and help others include it in their pipelines. If I could replace all that (or maybe just the contents of stage { } ) with a call to a library, it would be really handy.

            Show
            stevenfoster Steven Foster added a comment - Declarative stages have become so flexible now: they can have their own options, environment, post stages, agent etc. This makes the ability to reuse and share them even more powerful. My use case is running tests against a selection of specialised hardware, accessible by a generic build agent. In conjunction with the lockable resources plugin, the typical stage for this in the middle of a pipeline looks something like: stage ( 'Run tests on kit' ) { agent { label 'local-kits' } options { lock(label: 'kit-type-1' , variable: 'KIT_IP' , quantity: 1) } steps { runTest( 'example-test.zip' ) // existing shared function to download and run previously archived binary on reserved kit } post { always { junit 'results.xml' } } } I use this in a ton of places, and help others include it in their pipelines. If I could replace all that (or maybe just the contents of stage { } ) with a call to a library, it would be really handy.
            abayer Andrew Bayer made changes -
            Link This issue duplicates JENKINS-49135 [ JENKINS-49135 ]
            abayer Andrew Bayer made changes -
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            Resolution Duplicate [ 3 ]
            Hide
            tom_ghyselinck Tom Ghyselinck added a comment - - edited

            Hi Andrew Bayer,

            Is this really a duplicate of JENKINS-49135?

            In PR #241, you explicitly say that you won't support the directives stage, stages, steps, etc.
            and this is actually what we do want here.

            With best regards,
            Tom.

            Show
            tom_ghyselinck Tom Ghyselinck added a comment - - edited Hi Andrew Bayer , Is this really a duplicate of JENKINS-49135 ? In PR #241 , you explicitly say that you won't support the directives stage , stages , steps , etc. and this is actually what we do want here. With best regards, Tom.
            Hide
            franknarf888 Frank Gen added a comment -

            Hello,

            This fix is marked as "Fixed but Unreleased", can someone add a pointer to the PR implementing this? Or any documentation?

            Thanks

            • Frank
            Show
            franknarf888 Frank Gen added a comment - Hello, This fix is marked as "Fixed but Unreleased", can someone add a pointer to the PR implementing this? Or any documentation? Thanks Frank
            Hide
            colmose Colm O'Shea added a comment -

            Would also love to see if there is docs or something for this?

            Show
            colmose Colm O'Shea added a comment - Would also love to see if there is docs or something for this?
            Hide
            cjharmath CJ Harmath added a comment - - edited

            I would like this as well.

            Use case:

            share build stages across similar projects, to keep things DRY and have a central shared library defining the various build stages per project types.

            i.e.: I might have many many microservices all using the same 1) package restore 2) lint 3) build 4) run unit tests  stages with only parameter differences and it would be awesome if app devs can just include a single line with the necessary parameters to call these 4 steps.

             

            so instead of

             

            stages {
                stage('restore') {
                 steps {
                  echo 'package restore'      
                  restore()     
                 }
            
              }
            stage('build') {
               steps {
                 echo 'building'
                 build()
                }
            
              }
            stage('test') {
                steps {
                  echo 'testing'
                  test()
                 }
            
              }
             }
            

             

            you could do this:

             

            stages {   
               sharedMicroserviceStages()
            }
            

             

             

            Show
            cjharmath CJ Harmath added a comment - - edited I would like this as well. Use case: share build stages across similar projects, to keep things DRY and have a central shared library defining the various build stages per project types. i.e.: I might have many many microservices all using the same 1) package restore 2) lint 3) build 4) run unit tests  stages with only parameter differences and it would be awesome if app devs can just include a single line with the necessary parameters to call these 4 steps.   so instead of   stages {    stage( 'restore' ) {     steps { echo ' package restore'      restore()     }   } stage( 'build' ) {   steps { echo 'building' build() }   } stage( 'test' ) {    steps { echo 'testing'      test() }  } }   you could do this:   stages {       sharedMicroserviceStages() }    
            Hide
            cjharmath CJ Harmath added a comment -

            Okay, so I just re-read the docs and as I understand what I should be doing is to just put the entire pipeline into my shared library.

            https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines

            This kind of makes me wonder if we allow defining entire pipeline, why not allow stages as well ?

            But I guess I am already happy with this as I only need to maintain one pipeline per app type.

            Show
            cjharmath CJ Harmath added a comment - Okay, so I just re-read the docs and as I understand what I should be doing is to just put the entire pipeline into my shared library. https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines This kind of makes me wonder if we allow defining entire pipeline, why not allow stages as well ? But I guess I am already happy with this as I only need to maintain one pipeline per app type.
            Hide
            dalvizu Dan Alvizu added a comment -

            For anyone else who runs into this: Andrew closed this as 'FIxed but unreleased' but with a resolution of 'Duplicate'.

             

            The duplicate issue is JENKINS-49135 and track there

            Show
            dalvizu Dan Alvizu added a comment - For anyone else who runs into this: Andrew closed this as 'FIxed but unreleased' but with a resolution of 'Duplicate'.   The duplicate issue is  JENKINS-49135 and track there

              People

              • Assignee:
                Unassigned
                Reporter:
                tobilarscheid Tobias Larscheid
              • Votes:
                22 Vote for this issue
                Watchers:
                47 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: