Details

    • Similar Issues:

      Description

      It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

      Mock framework

      Inspired by Job DSL (example).

      We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.

      Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).

      Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.

      Quick and flexible, but low fidelity to real behavior.

      JenkinsRule-style

      Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.

      Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.

      Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            it might be a good idea to mention it in the docs

            If you have field experience doing this successfully, you probably know as much as anyone about the topic, so contributions are welcomed.

            Show
            jglick Jesse Glick added a comment - it might be a good idea to mention it in the docs If you have field experience doing this successfully, you probably know as much as anyone about the topic, so contributions are welcomed .
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Stefan Rademacher I suggest bringing up the framework comparison topics in https://jenkins.io/sigs/pipeline-authoring/ . It this the venue with the ongoing discussion about Pipeline development tools. CC Andrew Bayer Liam Newman

             

             

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - Stefan Rademacher I suggest bringing up the framework comparison topics in https://jenkins.io/sigs/pipeline-authoring/  . It this the venue with the ongoing discussion about Pipeline development tools. CC Andrew Bayer Liam Newman      
            Hide
            sparshev Sergei Parshev added a comment - - edited

            Guys, why we're not unit-testing pipelines directly using JenkinsRule?
            I don't think it's worth to support some separated engine to run the pipelines, because they always will have some deviation from the actual engine (for example JenkinsPipelineEngine CPS issue and related CPS issue of find/findAll and the other closure functions that was fixed in workflow-cps-plugin years ago).

            I'm checking a way to test like in JenkinsPipelineEngine (same interface) but using JenkinsRule and workflow-cps-plugin and seems it's possible, but probably will require some hooks from CpsScript.invokeMethod (at least I did not found a way without copy to groovy src). When it will work somehow - I will share the changes and hopefully they will work well)

            Show
            sparshev Sergei Parshev added a comment - - edited Guys, why we're not unit-testing pipelines directly using JenkinsRule? I don't think it's worth to support some separated engine to run the pipelines, because they always will have some deviation from the actual engine (for example JenkinsPipelineEngine CPS issue and related CPS issue of find/findAll and the other closure functions that was fixed in workflow-cps-plugin years ago). I'm checking a way to test like in JenkinsPipelineEngine (same interface) but using JenkinsRule and workflow-cps-plugin and seems it's possible, but probably will require some hooks from CpsScript.invokeMethod (at least I did not found a way without copy to groovy src). When it will work somehow - I will share the changes and hopefully they will work well)
            Hide
            jglick Jesse Glick added a comment -

            Tip: if a script defines a method named, say, node, it will take precedence over a Step of the same name.

            Show
            jglick Jesse Glick added a comment - Tip: if a script defines a method named, say, node , it will take precedence over a Step of the same name.
            Hide
            sparshev Sergei Parshev added a comment -

            Jesse Glick I think the good thing of JenkinsPipelineUnit - user specifically chooses what kind of steps is allowed. Unfortunately just redefine still leaves a way to execute some not described step. I think it's a potential issue, so without this strict rule proper unit tests will be impossible.

            Show
            sparshev Sergei Parshev added a comment - Jesse Glick I think the good thing of JenkinsPipelineUnit - user specifically chooses what kind of steps is allowed. Unfortunately just redefine still leaves a way to execute some not described step. I think it's a potential issue, so without this strict rule proper unit tests will be impossible.

              People

              • Assignee:
                Unassigned
                Reporter:
                jglick Jesse Glick
              • Votes:
                108 Vote for this issue
                Watchers:
                127 Start watching this issue

                Dates

                • Created:
                  Updated: