Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43083

Improve unit and integration tests and avoid relying on E2E (ATH) for front end code

    Details

    • Similar Issues:
    • Sprint:
      Blue Ocean - 1.1-beta-1, Blue Ocean - 1.1-beta2, Blue Ocean 1.1-beta4, Blue Ocean 1.1, Blue Ocean 1.1

      Description

      Currently we have some unit tests for UI, a lot of unit and integration tests for API, but rely mostly on ATH for feature and integration testing. 

      This is perhaps not optimal, there must be ways where we can unit and even integration tests components without end to end testing with full browsers. Eg, could we aim for a pyramid of tests like: 

      https://2.bp.blogspot.com/-YTzv_O4TnkA/VTgexlumP1I/AAAAAAAAAJ8/57-rnwyvP6g/s1600/image02.png from: https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html 

      In scope: 

      • Look at a way of reporting test coverage (so we can report on it) and know where we stand
      • Consolidate test runners used (karma or mocha?) across modules
      • Improve code coverage (even if small improvement) for unit testable areas in javascript code where it makes sense, using mocks, component testing, whatever
        • Add unit tests for Activity, Branches and Pagers tab to ensure that data doesn't "blink" when pressing "Show More" (see JENKINS-43702). Note: this is currently a bug on Pull Requests tab.

        Attachments

          Issue Links

            Activity

            Hide
            michaelneale Michael Neale added a comment -

            Cliff Meyers yes - by "unit tests" really means "not end to end" - it may mean "integration" tests, for sure. 

            Show
            michaelneale Michael Neale added a comment - Cliff Meyers yes - by "unit tests" really means "not end to end" - it may mean "integration" tests, for sure. 
            Hide
            jhill John Hill added a comment - - edited

            Without a deep understanding of React testing (I come from an angularJS 1.x shop) I'd say that it's always a safe bet to chose the framework which maintained by the principal contributors to the project. In this case Facebook -> Jest.

            The Jest team has a healthy community which continually addresses issues and has specifically solved the issue which affected the user from feb 2016 in addition to a bunch of other first-time-user issues.

            Show
            jhill John Hill added a comment - - edited Without a deep understanding of React testing (I come from an angularJS 1.x shop) I'd say that it's always a safe bet to chose the framework which maintained by the principal contributors to the project. In this case Facebook -> Jest. The Jest team has a healthy community which continually addresses issues and has specifically solved the issue which affected the user from feb 2016 in addition to a bunch of other first-time-user issues .
            Hide
            michaelneale Michael Neale added a comment -

            Jest looks good - but I will let Cliff Meyers comment. When I looked it was about snapshot based testing which seems "odd" to me and not sure if it really helps people that much in practice (it just tells you that "things have changed") and encourages people to churn the tests all the time, vs writing things that are less variable with small changes. 

            Show
            michaelneale Michael Neale added a comment - Jest looks good - but I will let Cliff Meyers comment. When I looked it was about snapshot based testing which seems "odd" to me and not sure if it really helps people that much in practice (it just tells you that "things have changed") and encourages people to churn the tests all the time, vs writing things that are less variable with small changes. 
            Hide
            cliffmeyers Cliff Meyers added a comment -

            John Hill thanks for that feedback. There are a few parts to Jest really, that need to be considered independently:

            • Jest is a test runner, which would replace the mocha runner (used by jenkins-js-builder) or the karma runner (used in some libs like JDL or blueocean-core-js). Jest runs in a browserless environment which has pros and cons. We don't have too many issues with the test runner itself, although it would be good to standardize on one across the board.
            • Just has some auto-mocking behavior which is interesting but also very complex under the hood.
            • Jest offers an API to do snapshotting which is optional; you can also use Enzyme w/ react-test-utils like we do today in most unit tests. The thrust of this ticket was mostly on this point here, as to whether it would better than Enzyme shallow render. Enzyme's html() method should also be explored.
            Show
            cliffmeyers Cliff Meyers added a comment - John Hill thanks for that feedback. There are a few parts to Jest really, that need to be considered independently: Jest is a test runner, which would replace the mocha runner (used by jenkins-js-builder) or the karma runner (used in some libs like JDL or blueocean-core-js). Jest runs in a browserless environment which has pros and cons. We don't have too many issues with the test runner itself, although it would be good to standardize on one across the board. Just has some auto-mocking behavior which is interesting but also very complex under the hood. Jest offers an API to do snapshotting which is optional; you can also use Enzyme w/ react-test-utils like we do today in most unit tests. The thrust of this ticket was mostly on this point here, as to whether it would better than Enzyme shallow render. Enzyme's html() method should also be explored.
            Hide
            cliffmeyers Cliff Meyers added a comment - - edited

            Added link to the PR. As it stands now it accomplishes the following:

            • Switch to Jest test runner for core-js standalone gulp build and dashboard / personalization via js-builder plugin.
            • Outputs coverage reports for all
            • Outputs junit test reports for all
            • Allows unit tests to be easily debugged

            Does not address improvement of test coverage in any particular place, but this should give us the tools to do that.

            Show
            cliffmeyers Cliff Meyers added a comment - - edited Added link to the PR. As it stands now it accomplishes the following: Switch to Jest test runner for core-js standalone gulp build and dashboard / personalization via js-builder plugin. Outputs coverage reports for all Outputs junit test reports for all Allows unit tests to be easily debugged Does not address improvement of test coverage in any particular place, but this should give us the tools to do that.

              People

              • Assignee:
                cliffmeyers Cliff Meyers
                Reporter:
                michaelneale Michael Neale
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: