Details

    • Similar Issues:

      Description

      It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

      Mock framework

      Inspired by Job DSL (example).

      We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.

      Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).

      Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.

      Quick and flexible, but low fidelity to real behavior.

      JenkinsRule-style

      Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.

      Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.

      Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.

        Attachments

          Issue Links

            Activity

            Hide
            sparshev Sergei Parshev added a comment -

            Hi Jesse Glick,

            I just completed my research & implementation of JenkinsRule+Groovy-CPS interceptor Jenkinsfile unit testing framework (link) - right now it's integrated to MPL and it's working, but looks not so good as I thought:

            • It uses java path overrides to setup a couple of files to use my invokers: link - not sure it's a good idea... Maybe there is a way to provide some hooks from Jenkins side, wdyt?
            • JenkinsRule produces too much logs - there is a way how to disable the logging? My attempts were unsuccessful: link

            If you could provide some advice - that will be really great!

            Thank you

            Show
            sparshev Sergei Parshev added a comment - Hi Jesse Glick , I just completed my research & implementation of JenkinsRule+Groovy-CPS interceptor Jenkinsfile unit testing framework ( link ) - right now it's integrated to MPL and it's working, but looks not so good as I thought: It uses java path overrides to setup a couple of files to use my invokers: link - not sure it's a good idea... Maybe there is a way to provide some hooks from Jenkins side, wdyt? JenkinsRule produces too much logs - there is a way how to disable the logging? My attempts were unsuccessful: link If you could provide some advice - that will be really great! Thank you
            Hide
            jglick Jesse Glick added a comment -

            Sergei Parshev Interesting but I doubt maintainers of groovy-sandbox / groovy-cps can take on any testability enhancements—keeping production code running is more than enough work.

            Show
            jglick Jesse Glick added a comment - Sergei Parshev Interesting but I doubt maintainers of groovy-sandbox / groovy-cps can take on any testability enhancements—keeping production code running is more than enough work.
            Hide
            sparshev Sergei Parshev added a comment -

            Jesse Glick Ok, got it. Will work on preparing the changes for groovy-cps unit testing in this intercepting way, if you think it's a good idea.

            Show
            sparshev Sergei Parshev added a comment - Jesse Glick Ok, got it. Will work on preparing the changes for groovy-cps unit testing in this intercepting way, if you think it's a good idea.
            Hide
            jglick Jesse Glick added a comment -

            I am not maintaining this code so I cannot comment.

            Show
            jglick Jesse Glick added a comment - I am not maintaining this code so I cannot comment.
            Hide
            sparshev Sergei Parshev added a comment -

            Ok, just added issue and PR with a sample realization to groovy-cps. Hopefully we will find some way to simplify the execution of JenkinsRule-based unit tests for the shared libraries.

            Show
            sparshev Sergei Parshev added a comment - Ok, just added issue and PR with a sample realization to groovy-cps. Hopefully we will find some way to simplify the execution of JenkinsRule-based unit tests for the shared libraries.

              People

              • Assignee:
                Unassigned
                Reporter:
                jglick Jesse Glick
              • Votes:
                110 Vote for this issue
                Watchers:
                129 Start watching this issue

                Dates

                • Created:
                  Updated: