Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45014

Developer is able to see which tests are intermittent

    Details

    • Similar Issues:

      Description

      Feature request: 

      If tests in a test suite often flick between failure and pass, it would be nice to report on this right in the test results screen in blue ocean. 

       

      https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin shows some of the functionality, but this is not about rerunning bad things, but just identifying that "often this test fails, but then passes in a subsequent run when there were no code changes. 

      A fairly simple heuristic could be used - doens't necessarily need to use stats over many builds. If any test failed, but then passed due a subsequent run when there were NO CODE changes, mark it as flaky. This could also be applicable across branches (ie use the default branch mainly). 

       

      This would mean when you see failures, you know that you can run it again, but also that you really should get your shit together and fix that test somehow. 

        Attachments

          Issue Links

            Activity

            Hide
            michaelneale Michael Neale added a comment -

            James Dumay I think this is something that could just be baked in - any time the tests tab is used. 

            Show
            michaelneale Michael Neale added a comment - James Dumay I think this is something that could just be baked in - any time the tests tab is used. 
            Hide
            jamesdumay James Dumay added a comment -

            Yes, this would be most helpful. Not sure how this would work with the 10,000+ tests that some of our users have. Another case where this problem could be easily solved with a DB query.

            Show
            jamesdumay James Dumay added a comment - Yes, this would be most helpful. Not sure how this would work with the 10,000+ tests that some of our users have. Another case where this problem could be easily solved with a DB query.

              People

              • Assignee:
                Unassigned
                Reporter:
                michaelneale Michael Neale
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: