Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13113

xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.455
      xunit 1.40
    • Similar Issues:

      Description

      We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

      One example from the TRX file:

          <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
          </UnitTestResult>
      

      The transformation in the junitResult.xml file:

        <case>
        <duration>NaN</duration> 
        <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className> 
        <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName> 
        <skipped>false</skipped> 
        <failedSince>0</failedSince> 
        </case>
      

        Attachments

          Activity

          bruce Dirk Kuypers created issue -
          gbois Gregory Boissinot made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          gbois Gregory Boissinot made changes -
          Status In Progress [ 3 ] Closed [ 6 ]
          Resolution Incomplete [ 4 ]
          bruce Dirk Kuypers made changes -
          Resolution Incomplete [ 4 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 143574 ] JNJira + In-Review [ 186047 ]
          nfalco Nikolas Falco made changes -
          Description We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

          One example from the TRX file:

              <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
              </UnitTestResult>

          The transformation in the junitResult.xml file:

            <case>
            <duration>NaN</duration>
            <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className>
            <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName>
            <skipped>false</skipped>
            <failedSince>0</failedSince>
            </case>
          We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

          One example from the TRX file:

          {code:xml}
              <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
              </UnitTestResult>
          {code}
          The transformation in the junitResult.xml file:
          {code:xml}
            <case>
            <duration>NaN</duration>
            <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className>
            <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName>
            <skipped>false</skipped>
            <failedSince>0</failedSince>
            </case>
          {code}
          nfalco Nikolas Falco made changes -
          Assignee Gregory Boissinot [ gbois ] Nikolas Falco [ nfalco ]
          nfalco Nikolas Falco made changes -
          Summary Xunit Plugin detects MSTEST "NotExecuted" as "Passed" instead of "Skipped" xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped
          nfalco Nikolas Falco made changes -
          Status Reopened [ 4 ] In Progress [ 3 ]
          nfalco Nikolas Falco made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              bruce Dirk Kuypers
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: