Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13113

xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.455
      xunit 1.40
    • Similar Issues:

      Description

      We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

      One example from the TRX file:

          <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
          </UnitTestResult>
      

      The transformation in the junitResult.xml file:

        <case>
        <duration>NaN</duration> 
        <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className> 
        <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName> 
        <skipped>false</skipped> 
        <failedSince>0</failedSince> 
        </case>
      

        Attachments

          Activity

          bruce Dirk Kuypers created issue -
          Hide
          gbois Gregory Boissinot added a comment -

          Do you have the same issue with the independent MSTest Jenkins plugins?
          Thanks

          Show
          gbois Gregory Boissinot added a comment - Do you have the same issue with the independent MSTest Jenkins plugins? Thanks
          Hide
          bruce Dirk Kuypers added a comment -

          Didn't try that, I am only using the xunit plugin. Should I? Are using the MSTest plugin under the hood? But I am not sure I can reproduce the problem because it is possible that our developers fixed the bug. Or is it somehow possible to run the MSTest stuff without executing the job in Jenkins?

          Show
          bruce Dirk Kuypers added a comment - Didn't try that, I am only using the xunit plugin. Should I? Are using the MSTest plugin under the hood? But I am not sure I can reproduce the problem because it is possible that our developers fixed the bug. Or is it somehow possible to run the MSTest stuff without executing the job in Jenkins?
          Hide
          gbois Gregory Boissinot added a comment -

          MSTest plugin is an independent plugin, written before xUnit. xUnit plugin is aimed at supporting all xUnit tools.
          Each plugin delegated his process to an XSL. And xUnit plugin uses a copy of the MSTest XSL.
          Maybe, the xsl has been fixed in MSTest plugin but not in the xUnit plugin.

          Therefore, could you try with MSTest?

          Thanks

          Show
          gbois Gregory Boissinot added a comment - MSTest plugin is an independent plugin, written before xUnit. xUnit plugin is aimed at supporting all xUnit tools. Each plugin delegated his process to an XSL. And xUnit plugin uses a copy of the MSTest XSL. Maybe, the xsl has been fixed in MSTest plugin but not in the xUnit plugin. Therefore, could you try with MSTest? Thanks
          gbois Gregory Boissinot made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          gbois Gregory Boissinot added a comment -

          Please reopen the issue when you could have more inputs.
          Thanks

          Show
          gbois Gregory Boissinot added a comment - Please reopen the issue when you could have more inputs. Thanks
          gbois Gregory Boissinot made changes -
          Status In Progress [ 3 ] Closed [ 6 ]
          Resolution Incomplete [ 4 ]
          Hide
          bruce Dirk Kuypers added a comment -

          Hi Gregory,

          I came accross those two issues for mstest plugin. As the mstest plugin shows the same errors and there even exists a pull request with a fix in 11332, couldn't you just implement the fix for the xunit plugin? I could not see any progress in mstest plugin for ages now...

          https://issues.jenkins-ci.org/browse/JENKINS-8435
          https://issues.jenkins-ci.org/browse/JENKINS-11332

          Thanks
          Dirk

          Show
          bruce Dirk Kuypers added a comment - Hi Gregory, I came accross those two issues for mstest plugin. As the mstest plugin shows the same errors and there even exists a pull request with a fix in 11332, couldn't you just implement the fix for the xunit plugin? I could not see any progress in mstest plugin for ages now... https://issues.jenkins-ci.org/browse/JENKINS-8435 https://issues.jenkins-ci.org/browse/JENKINS-11332 Thanks Dirk
          bruce Dirk Kuypers made changes -
          Resolution Incomplete [ 4 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 143574 ] JNJira + In-Review [ 186047 ]
          nfalco Nikolas Falco made changes -
          Description We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

          One example from the TRX file:

              <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
              </UnitTestResult>

          The transformation in the junitResult.xml file:

            <case>
            <duration>NaN</duration>
            <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className>
            <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName>
            <skipped>false</skipped>
            <failedSince>0</failedSince>
            </case>
          We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

          One example from the TRX file:

          {code:xml}
              <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
              </UnitTestResult>
          {code}
          The transformation in the junitResult.xml file:
          {code:xml}
            <case>
            <duration>NaN</duration>
            <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className>
            <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName>
            <skipped>false</skipped>
            <failedSince>0</failedSince>
            </case>
          {code}
          nfalco Nikolas Falco made changes -
          Assignee Gregory Boissinot [ gbois ] Nikolas Falco [ nfalco ]
          nfalco Nikolas Falco made changes -
          Summary Xunit Plugin detects MSTEST "NotExecuted" as "Passed" instead of "Skipped" xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped
          nfalco Nikolas Falco made changes -
          Status Reopened [ 4 ] In Progress [ 3 ]
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Nikolas Falco
          Path:
          src/main/resources/org/jenkinsci/plugins/xunit/types/ctest-to-junit.xsl
          src/test/java/org/jenkinsci/plugins/xunit/types/MSTestTest.java
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase1/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase2/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase3/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase4/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase5/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase6/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/input.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/result.xml
          http://jenkins-ci.org/commit/xunit-plugin/5100d71b47639be65dbf916e467d181e9e11e9d9
          Log:
          JENKINS-13113 xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped

          Update XSL to count NotExecuted test as skipped.
          Minor improvements to testsuite time duration.

          Compare: https://github.com/jenkinsci/xunit-plugin/compare/dd79be01562d...5100d71b4763
          *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

          Functionality will be removed from GitHub.com on January 31st, 2019.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nikolas Falco Path: src/main/resources/org/jenkinsci/plugins/xunit/types/ctest-to-junit.xsl src/test/java/org/jenkinsci/plugins/xunit/types/MSTestTest.java src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase1/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase2/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase3/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase4/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase5/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase6/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/input.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/result.xml http://jenkins-ci.org/commit/xunit-plugin/5100d71b47639be65dbf916e467d181e9e11e9d9 Log: JENKINS-13113 xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped Update XSL to count NotExecuted test as skipped. Minor improvements to testsuite time duration. Compare: https://github.com/jenkinsci/xunit-plugin/compare/dd79be01562d...5100d71b4763 * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.
          nfalco Nikolas Falco made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              bruce Dirk Kuypers
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: