Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36759

MSTest Plugin reports failed data driven tests as passed in certain situations.

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: mstest-plugin
    • Labels:
      None
    • Environment:
      Windows 2012, Jenkins 1.653, mstest-plugin 0.19
    • Similar Issues:

      Description

      If a data driven test fails before ever running the test - for example, if the data file cannot be found - then the "failed" outcome in the .trx output file is not reported at all. It is not even included in the test counts. Also, the Jenkins Job passes as Stable, so it appears to everybody that everything is OK, whereas at best the test didn't run at all.
      The attached file UnitTest1.cs is a minimal test which reproduces the issue.
      The attached file TestResults.trx is the mstest output showing the test failure.
      The attached file Results.jpg is a screenshot of the Jenkins results as displayed by the mstest plugin.
      This is pretty serious, we had a product failure but when we looked at the Jenkins tests all appeared fine. The problem only showed up when we ran the tests manually and they failed.

        Attachments

        1. Results.jpg
          Results.jpg
          91 kB
        2. TestResults.trx
          5 kB
        3. UnitTest1.cs
          0.4 kB

          Activity

          Show
          nilleb Ivo Bellin Salarin added a comment - Fixed in  https://github.com/jenkinsci/mstest-plugin/commit/eb2cd6c07520ae2a5d4fd9e1f2d3d7569d445bb9

            People

            • Assignee:
              nilleb Ivo Bellin Salarin
              Reporter:
              jonbowen Jon Bowen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: