Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42079

Allow "local" Shared Library to be loaded from current repo branch

    XMLWordPrintable

    Details

    • Epic Link:
    • Similar Issues:

      Description

      For context see: https://jenkins.io/blog/2017/02/15/declarative-notifications/
      And the linked bugs, in particular JENKINS-41396.

      I've been informed that this is a horrible idea, but I'm going to describe it here just to make sure.

      There have been a number of issues and discussions going around, dealing with how to define variables and methods in Declarative Pipeline.

      Right now, we can define variables and methods outside of the pipeline section but that is not a supported solution and it is not viable long-term.

      As noted in JENKINS-41396, we have a supported way to define variables and methods that we can use in Declarative Pipeline via
      Shared Libraries. However, that means I have to spin up a separate repository just to make a helper method that might be used in only one
      Pipeline.

      JENKINS-41396 and JENKINS-41335 suggest adding sections to the Declarative Pipeline model to allow these things to defined inside
      the same Jenkinsfile. This is also problematic: there are implementation issues, and it means that the distinction between the overall flow
      (defined in Declarative Pipeline) and the implementation details (defined in methods, variables, and classes) becomes more blurry.

      What I'm suggesting, is we reuse the functionality of "Shared libraries" but support loading from a subfolder in the current branch of the currently checked out repository.

      This would have a number of benefits over other solutions, it would:

      • reuse existing (and tested) functionality, limiting risk
      • not require creating a separate repository
      • keep the script code out of the Declarative Pipeline Jenkinsfile
      • use the same file/folder structure as Shared Library, getting users familiar with it for when they decide to use Shared Libraries

      On the downside, it would:

      • depend on additional folder/files beyond Jenkinsfile (method and define would be in Jenkinsfile)
      • require some changes to workflow-cps-global-lib to support loading from local and from subfolder instead of root folder.
      • make users have to understand library folder/file structure even to write simple helper method.

      We had a IRC chat about this, which I've attached.

      Also attached are images from a Pipeline job that I created as a proof-of-concept.
      The code for this POC is in: https://github.com/bitwiseman/hermann/tree/issue/jenkins-library/example

      What is not shown/done in the POC:

      • loading from a subfolder (not currently supported), subfolder name should default to "jenkins" (maybe?) and be configurable to other values.
      • * Worked around by putting the vars folder in the root - this would not work if the project already had a src or vars folder.
      • loading from the currently checked out branch/commit (not currently supported)
      • * Worked around by hard coding the branch name
      • using a library local directive to tell declarative to load a local library
      • * Worked around by adding the configuration to the parent Multibranch pipeline job and loading implicitly.
      • Using the already checked-out code
      • * Shared library simply clones it's own copy. For this small repo, this is fine, for larger projects it would not be acceptable.

        Attachments

          Issue Links

            Activity

            Hide
            bitwiseman Liam Newman added a comment -

            Some have suggested that evaluate or readTrusted would be preferable solutions.
            They are fine for loading individual files, but not for more than one. Also, they behave differently from code in shared libraries, meaning users will have learn different behaviors for each.
            These also have drawbacks in areas such as testing, support, and documentation. If we guide users towards one consistent way to load things (Libraries) we do not have to explain it multiple times and fixes that help Shared Libraries would generally help local libraries as well.

            Show
            bitwiseman Liam Newman added a comment - Some have suggested that evaluate or readTrusted would be preferable solutions. They are fine for loading individual files, but not for more than one. Also, they behave differently from code in shared libraries, meaning users will have learn different behaviors for each. These also have drawbacks in areas such as testing, support, and documentation. If we guide users towards one consistent way to load things (Libraries) we do not have to explain it multiple times and fixes that help Shared Libraries would generally help local libraries as well.
            Hide
            bitwiseman Liam Newman added a comment -

            One other note: even if we implement the method or define sections, I believe we should still consider this option. It provides another helpful interim step between on the way to Shared Libraries.

            Show
            bitwiseman Liam Newman added a comment - One other note: even if we implement the method or define sections, I believe we should still consider this option. It provides another helpful interim step between on the way to Shared Libraries.
            Hide
            jons Jon Sten added a comment -

            We have a similar needs like this, in our case or needs are:

            • We wan't to extract all the "bulk" parts of our Jenkinsfile into a library
            • We have downstream products which wan't to run part of our tests (but with their latest version), running the tests isn't as simple as a single call and requires several lines of groovy code and thus we don't wan't to duplicate this. So ideally they wan't to call one of our functions in the library
            • Generally it feels natural to co-develop and version the library with the product in the same repo (since you usually require updates to both).

             

            Now, I have found a solution where you can load a library in a sub-folder of the repository, this requires that you load the Jenkinsfile through scm so that you have the scm variable. Note, we use SVN but it should be possible to port to GIT or similar:

            library identifier: 'OurLib@JenkinsLIB', retriever: modernSCM([$class: 'SubversionSCMSource', remoteBase: scm.getLocations()[0].remote])

            This loads the folder JenkinsLIB and treats it as a jenkins library. I wouldn't say that this is a nice solution, another problem is that the library is unaware of the repo url that it is associated with (which is necessary if you wan't to checkout the actual repo).

            Show
            jons Jon Sten added a comment - We have a similar needs like this, in our case or needs are: We wan't to extract all the "bulk" parts of our Jenkinsfile into a library We have downstream products which wan't to run part of our tests (but with their latest version), running the tests isn't as simple as a single call and requires several lines of groovy code and thus we don't wan't to duplicate this. So ideally they wan't to call one of our functions in the library Generally it feels natural to co-develop and version the library with the product in the same repo (since you usually require updates to both).   Now, I have found a solution where you can load a library in a sub-folder of the repository, this requires that you load the Jenkinsfile through scm so that you have the scm variable. Note, we use SVN but it should be possible to port to GIT or similar: library identifier: 'OurLib@JenkinsLIB', retriever: modernSCM([$class: 'SubversionSCMSource', remoteBase: scm.getLocations()[0].remote]) This loads the folder JenkinsLIB and treats it as a jenkins library. I wouldn't say that this is a nice solution, another problem is that the library is unaware of the repo url that it is associated with (which is necessary if you wan't to checkout the actual repo).
            Hide
            medianick Nick Jones added a comment -

            In my case I handle reusable notification code via a .groovy file committed into the project's source control (managed through a Jenkins-independent packaging mechanism like npm or NuGet, so versioned and updated there), which allows a single "library" to be loaded from the local branch via readTrusted:

            def lib = evaluate readTrusted('build/Library.groovy')
            def slackChannel = '#foo'
            
            pipeline {
              ...
              stages {
                stage('Build') {
                  script {
                    lib.notifySlack(slackChannel, 'STARTED')
                  }
                }
              }
              ...
              post {
                always {
                  script {
                    lib.notifySlack(slackChannel, currentBuild.result)
                  }
                }
              }
            }
            

            The script blocks are necessary to avoid Method calls on objects not allowed outside "script" blocks errors (JENKINS-42360, Won't Fix), and to enable the use of a slackChannel variable to avoid hard-coding the channel name in multiple places.

            My fear is that this loophole of using def outside the pipeline block will get closed, in as much as it's ostensibly unsupported behavior. But I don't currently have another way that I know of to have the syntactic goodness of Declarative while still leveraging code reuse. (And I should note that the exact same Library.groovy file in question is used in Scripted Pipeline jobs too.)

            Show
            medianick Nick Jones added a comment - In my case I handle reusable notification code via a .groovy file committed into the project's source control (managed through a Jenkins-independent packaging mechanism like npm or NuGet, so versioned and updated there), which allows a single "library" to be loaded from the local branch via readTrusted : def lib = evaluate readTrusted( 'build/Library.groovy' ) def slackChannel = '#foo' pipeline { ... stages { stage( 'Build' ) { script { lib.notifySlack(slackChannel, 'STARTED' ) } } } ... post { always { script { lib.notifySlack(slackChannel, currentBuild.result) } } } } The script blocks are necessary to avoid Method calls on objects not allowed outside "script" blocks errors ( JENKINS-42360 , Won't Fix), and to enable the use of a slackChannel variable to avoid hard-coding the channel name in multiple places. My fear is that this loophole of using def outside the pipeline block will get closed, in as much as it's ostensibly unsupported behavior. But I don't currently have another way that I know of to have the syntactic goodness of Declarative while still leveraging code reuse. (And I should note that the exact same  Library.groovy file in question is used in Scripted Pipeline jobs too.)
            Hide
            aarondmarasco_vsi Aaron D. Marasco added a comment -

            I'm kind of new to this pipeline stuff, but I think I'm doing this now?

            // To bring in the snippets, we need to be running on a real node
            // TODO: There are Jenkins plugins that can fix this.
            node('master') {
              checkout scm
              def import_config = load("releng/jenkins/runtime/groovy/import_config.groovy")
              branch_config = import_config()
              def find_last_success = load("releng/jenkins/runtime/groovy/find_last_success.groovy")
              previous_changesets = find_last_success()
            }
            
            Show
            aarondmarasco_vsi Aaron D. Marasco added a comment - I'm kind of new to this pipeline stuff, but I think I'm doing this now? // To bring in the snippets, we need to be running on a real node // TODO: There are Jenkins plugins that can fix this . node( 'master' ) { checkout scm def import_config = load( "releng/jenkins/runtime/groovy/import_config.groovy" ) branch_config = import_config() def find_last_success = load( "releng/jenkins/runtime/groovy/find_last_success.groovy" ) previous_changesets = find_last_success() }
            Show
            mkobit Mike Kobit added a comment - Is https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/37  related to this ?
            Hide
            jons Jon Sten added a comment -

            Mike Kobit It sure looks like that, that is great news!

            Show
            jons Jon Sten added a comment - Mike Kobit It sure looks like that, that is great news!

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                bitwiseman Liam Newman
              • Votes:
                13 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated: