Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33925

Test framework for Jenkinsfile

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

      Mock framework

      Inspired by Job DSL (example).

      We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.

      Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).

      Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.

      Quick and flexible, but low fidelity to real behavior.

      JenkinsRule-style

      Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.

      Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.

      Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.

        Attachments

          Issue Links

            Activity

            Hide
            sradi81 Stefan Rademacher added a comment - - edited

            The Jenkins Pipeline docs only refer to JenkinsPipelineUnit.

            On Jenkins World 2018 there was a talk about jenkins-spock.
            (see https://www.youtube.com/watch?v=4PZ-UFBexIE )

            I would like to find out, which library is the best choice, as I am starting to unittest my Jenkins Pipelines just now.

            Show
            sradi81 Stefan Rademacher added a comment - - edited The Jenkins Pipeline docs only refer to  JenkinsPipelineUnit . On Jenkins World 2018 there was a talk about  jenkins-spock . (see https://www.youtube.com/watch?v=4PZ-UFBexIE  ) I would like to find out, which library is the best choice, as I am starting to unittest my Jenkins Pipelines just now.
            Hide
            jglick Jesse Glick added a comment -

            jenkinsfile-runner-test-framework can be used for this purpose, again if you are interested in something closer to an integration test than a unit test.

            Show
            jglick Jesse Glick added a comment - jenkinsfile-runner-test-framework can be used for this purpose, again if you are interested in something closer to an integration test than a unit test.
            Hide
            sradi81 Stefan Rademacher added a comment -

            Thanks! I think unit tests are the best way to test the behavior of my custom steps. I just want to check, if all mocked steps are called as expected.
            For that scenario 'jenkins-spock' seems to be a notable alternative to JenkinsPipelineUnit.

            If you agree, it might be a good idea to mention it in the docs. Might be interesting, especially for Spock users.

            Show
            sradi81 Stefan Rademacher added a comment - Thanks! I think unit tests are the best way to test the behavior of my custom steps. I just want to check, if all mocked steps are called as expected. For that scenario 'jenkins-spock' seems to be a notable alternative to JenkinsPipelineUnit. If you agree, it might be a good idea to mention it in the docs. Might be interesting, especially for Spock users.
            Hide
            jglick Jesse Glick added a comment -

            it might be a good idea to mention it in the docs

            If you have field experience doing this successfully, you probably know as much as anyone about the topic, so contributions are welcomed.

            Show
            jglick Jesse Glick added a comment - it might be a good idea to mention it in the docs If you have field experience doing this successfully, you probably know as much as anyone about the topic, so contributions are welcomed .
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Stefan Rademacher I suggest bringing up the framework comparison topics in https://jenkins.io/sigs/pipeline-authoring/ . It this the venue with the ongoing discussion about Pipeline development tools. CC Andrew Bayer Liam Newman

             

             

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - Stefan Rademacher I suggest bringing up the framework comparison topics in https://jenkins.io/sigs/pipeline-authoring/  . It this the venue with the ongoing discussion about Pipeline development tools. CC Andrew Bayer Liam Newman      

              People

              • Assignee:
                Unassigned
                Reporter:
                jglick Jesse Glick
              • Votes:
                104 Vote for this issue
                Watchers:
                121 Start watching this issue

                Dates

                • Created:
                  Updated: