[JENKINS-13113] xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped - Jenkins JIRA

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Major
Resolution: Fixed
Component/s: xunit-plugin
Labels:
None
Environment:
Jenkins 1.455
xunit 1.40

Similar Issues:

Show

Description

One example from the TRX file:

    <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
    </UnitTestResult>

The transformation in the junitResult.xml file:

  <case>
  <duration>NaN</duration> 
  <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className> 
  <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName> 
  <skipped>false</skipped> 
  <failedSince>0</failedSince> 
  </case>

Attachments

Activity

Ascending order - Click to sort in descending order

Dirk Kuypers created issue - 2012-03-16 14:42

Hide

Permalink

Gregory Boissinot added a comment - 2012-03-18 23:28

Do you have the same issue with the independent MSTest Jenkins plugins?
Thanks

Show

Gregory Boissinot added a comment - 2012-03-18 23:28 Do you have the same issue with the independent MSTest Jenkins plugins? Thanks

Hide

Permalink

Dirk Kuypers added a comment - 2012-03-19 09:11

Didn't try that, I am only using the xunit plugin. Should I? Are using the MSTest plugin under the hood? But I am not sure I can reproduce the problem because it is possible that our developers fixed the bug. Or is it somehow possible to run the MSTest stuff without executing the job in Jenkins?

Show

Dirk Kuypers added a comment - 2012-03-19 09:11 Didn't try that, I am only using the xunit plugin. Should I? Are using the MSTest plugin under the hood? But I am not sure I can reproduce the problem because it is possible that our developers fixed the bug. Or is it somehow possible to run the MSTest stuff without executing the job in Jenkins?

Hide

Permalink

Gregory Boissinot added a comment - 2012-03-19 22:50

MSTest plugin is an independent plugin, written before xUnit. xUnit plugin is aimed at supporting all xUnit tools.
Each plugin delegated his process to an XSL. And xUnit plugin uses a copy of the MSTest XSL.
Maybe, the xsl has been fixed in MSTest plugin but not in the xUnit plugin.

Therefore, could you try with MSTest?

Thanks

Show

Gregory Boissinot added a comment - 2012-03-19 22:50 MSTest plugin is an independent plugin, written before xUnit. xUnit plugin is aimed at supporting all xUnit tools. Each plugin delegated his process to an XSL. And xUnit plugin uses a copy of the MSTest XSL. Maybe, the xsl has been fixed in MSTest plugin but not in the xUnit plugin. Therefore, could you try with MSTest? Thanks

Gregory Boissinot made changes - 2012-03-19 22:50

Field	Original Value	New Value
Status	Open [ 1 ]	In Progress [ 3 ]

Hide

Permalink

Gregory Boissinot added a comment - 2012-03-24 22:37

Please reopen the issue when you could have more inputs.
Thanks

Show

Gregory Boissinot added a comment - 2012-03-24 22:37 Please reopen the issue when you could have more inputs. Thanks

Gregory Boissinot made changes - 2012-03-24 22:37

Status	In Progress [ 3 ]	Closed [ 6 ]
Resolution		Incomplete [ 4 ]

Hide

Permalink

Dirk Kuypers added a comment - 2012-08-07 09:02

Hi Gregory,

I came accross those two issues for mstest plugin. As the mstest plugin shows the same errors and there even exists a pull request with a fix in 11332, couldn't you just implement the fix for the xunit plugin? I could not see any progress in mstest plugin for ages now...

https://issues.jenkins-ci.org/browse/JENKINS-8435
https://issues.jenkins-ci.org/browse/JENKINS-11332

Thanks
Dirk

Show

Dirk Kuypers added a comment - 2012-08-07 09:02 Hi Gregory, I came accross those two issues for mstest plugin. As the mstest plugin shows the same errors and there even exists a pull request with a fix in 11332, couldn't you just implement the fix for the xunit plugin? I could not see any progress in mstest plugin for ages now... https://issues.jenkins-ci.org/browse/JENKINS-8435 https://issues.jenkins-ci.org/browse/JENKINS-11332 Thanks Dirk

Dirk Kuypers made changes - 2012-08-07 09:02

Resolution	Incomplete [ 4 ]
Status	Closed [ 6 ]	Reopened [ 4 ]

R. Tyler Croy made changes - 2016-07-25 23:53

Workflow

JNJira [ 143574 ]

JNJira + In-Review [ 186047 ]

Nikolas Falco made changes - 2018-05-15 19:31

Description

We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

One example from the TRX file:

    <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
    </UnitTestResult>

The transformation in the junitResult.xml file:

  <case>
  <duration>NaN</duration>
  <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className>
  <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName>
  <skipped>false</skipped>
  <failedSince>0</failedSince>
  </case>

We had an out of memory exception while running our MSTEST unit tests which caused all subsequent tests to be NotExecuted. Unfortunately those "NotExecuted" tests were counted as passed, so the test job succeeded instead of failing.

One example from the TRX file:

{code:xml}
    <UnitTestResult executionId="88518b81-226a-4fe9-9896-774a00c13e8e" testId="3509a64f-6214-eb24-6628-bd431f93997c" testName="TestcaseWcdmaTxIntermod_5_12__FDD9" computerName="1SP1-SLAVE2" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="88518b81-226a-4fe9-9896-774a00c13e8e">
    </UnitTestResult>
{code}
The transformation in the junitResult.xml file:
{code:xml}
  <case>
  <duration>NaN</duration>
  <className>ConformanceWcdmaCompleteTest.BandSpecificTests</className>
  <testName>TestcaseWcdmaTxIntermod_5_12__FDD9</testName>
  <skipped>false</skipped>
  <failedSince>0</failedSince>
  </case>
{code}

Nikolas Falco made changes - 2018-05-15 21:45

Assignee

Gregory Boissinot [ gbois ]

Nikolas Falco [ nfalco ]

Nikolas Falco made changes - 2018-05-15 21:46

Summary

Xunit Plugin detects MSTEST "NotExecuted" as "Passed" instead of "Skipped"

xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped

Nikolas Falco made changes - 2018-05-15 21:46

Status

Reopened [ 4 ]

In Progress [ 3 ]

Hide

Permalink

SCM/JIRA link daemon added a comment - 2018-05-15 21:56

Code changed in jenkins
User: Nikolas Falco
Path:
src/main/resources/org/jenkinsci/plugins/xunit/types/ctest-to-junit.xsl
src/test/java/org/jenkinsci/plugins/xunit/types/MSTestTest.java
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase1/result.xml
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase2/result.xml
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase3/result.xml
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase4/result.xml
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase5/result.xml
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase6/result.xml
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/input.xml
src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/result.xml
http://jenkins-ci.org/commit/xunit-plugin/5100d71b47639be65dbf916e467d181e9e11e9d9
Log:
~~JENKINS-13113~~ xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped

Update XSL to count NotExecuted test as skipped.
Minor improvements to testsuite time duration.

Compare: https://github.com/jenkinsci/xunit-plugin/compare/dd79be01562d...5100d71b4763
*NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

Functionality will be removed from GitHub.com on January 31st, 2019.

Show

SCM/JIRA link daemon added a comment - 2018-05-15 21:56 Code changed in jenkins User: Nikolas Falco Path: src/main/resources/org/jenkinsci/plugins/xunit/types/ctest-to-junit.xsl src/test/java/org/jenkinsci/plugins/xunit/types/MSTestTest.java src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase1/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase2/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase3/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase4/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase5/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase6/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/input.xml src/test/resources/org/jenkinsci/plugins/xunit/types/mstest/testcase7/result.xml http://jenkins-ci.org/commit/xunit-plugin/5100d71b47639be65dbf916e467d181e9e11e9d9 Log: JENKINS-13113 xUnit plugins detects MSTEST "NotExecuted" as successed instead of skipped Update XSL to count NotExecuted test as skipped. Minor improvements to testsuite time duration. Compare: https://github.com/jenkinsci/xunit-plugin/compare/dd79be01562d...5100d71b4763 * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.

Nikolas Falco made changes - 2018-05-15 21:57

Status	In Progress [ 3 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]

People

Assignee:

Nikolas Falco

Reporter:

Dirk Kuypers

Votes:

0 Vote for this issue

Watchers:

2 Start watching this issue

Dates

Created:

2012-03-16 14:42

Updated:

2018-05-15 21:57

Resolved:

2018-05-15 21:57