Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18098

xunit does not recognize failed tests anymore when there is a delay between testing and processing of test result files (because of coverage post processing in my case)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      xunit 1.58, Jenkins 1.515
    • Similar Issues:

      Description

      (I posted this question on the user's mailinglist before)

      I recently started to use Opencover with my MSTest unit tests to get code coverage results. Since then the xunit plugin does not recognize errors as a Failure anymore. The TRX file is parsed and you can see at the summary page that there are tests failed and since when, but the job is counted as success and no email is triggered.

      Could it be that there is some delay now introduced inbetween writing the MSTest results file and processing by xunit that causes this behaviour? When I rerun the tests without coverage after processing the coverage results everything works like it should. What I can see from the last versions of xunit plugin and a short look into the source code I think there should be a mechanism that if the test result file is older than 3 seconds there should be an error which can be ignored if configured. But even when the error because of an outdated file is ignored (because I make sure that the workspace is always cleaned after the running the tests) the Jenkins test job should fail if one or more of the unit tests fail.

      I paste a log and if more info is needed like job config xml etc I will be happy to provide it.

      18:44:02 D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw>"C:\Program Files\OpenCover\OpenCover.Console.exe" -target:"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\..\IDE\MSTEST.EXE" -output:coverage.xml -targetargs:"/testcontainer:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\Bin\Debug\CommonFunctionsTests.dll /testsettings:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\localtestrun.testsettings /resultsfile:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\TestResults\CommonFunctionsTests\CommonFunctionsTests.trx" -mergebyhash -targetdir:"Bin\Debug" -filter:+[CommonFunctions*]*
      18:44:03 Executing: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTEST.EXE
      18:44:06 Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1
      18:44:06 Copyright (c) Microsoft Corporation. All rights reserved.
      18:44:06
      18:44:06 Loading D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\localtestrun.testsettings...
      18:44:07 Loading D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\Bin\Debug\CommonFunctionsTests.dll...
      18:44:08 Starting execution...
      18:44:18
      18:44:18 Results Top Level Tests
      18:44:18 ------- ---------------
      18:44:18 Passed CommonFunctionsTest.BufferedTraceListenerTest.BufferedTraceListenerConstructorTest
      [... Passes removed ...]
      18:44:31 Passed CommonFunctionsTest.CommonFunctionsTest.ResizeTwoDimensionalArrayTest2
      18:44:31 Failed CommonFunctionsTest.ContestConfigurationParametersTest.ContestConfigurationParametersFileTest
      18:44:31 Passed CommonFunctionsTest.ContestSecurityStreamTest.ContestSecurityStreamConstructorTest
      [... Passes removed ...]
      18:44:59 Passed TestcaseInformationTest.TestcaseXmlElementTest.TestcaseXmlElementConstructorTest2
      18:45:00 216/217 test(s) Passed, 1 Failed
      18:45:00
      18:45:00 Summary
      18:45:00 -------
      18:45:00 Test Run Failed.
      18:45:00 Passed 216
      18:45:00 Failed 1
      18:45:00 -----------
      18:45:00 Total 217
      18:45:00 Results file: D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\TestResults\CommonFunctionsTests\CommonFunctionsTests.trx
      18:45:00 Test Settings: localtestrun
      18:45:00 Committing...
      18:45:00 Visited Classes 70 of 114 (61.4)
      18:45:00 Visited Methods 588 of 1049 (56.05)
      18:45:00 Visited Points 5763 of 12043 (47.85)
      18:45:00 Visited Branches 1629 of 4073 (40.00)
      18:45:00
      18:45:00 ==== Alternative Results (includes all methods including those without corresponding source) ====
      18:45:00 Alternative Visited Classes 70 of 127 (55.12)
      18:45:00 Alternative Visited Methods 676 of 1266 (53.4)
      18:45:00
      18:45:00 D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw>"C:\Program Files\ReportGenerator_1.8.1.0.\bin\ReportGenerator.exe" -reports:coverage.xml -targetdir:coverage
      18:45:01 Loading report 'D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\coverage.xml'
      18:45:01 Preprocessing report
      18:45:02 Indexing classes in directory 'c:\Snapshots\ContestContinuous\contest_sw\CommonFunctions\CommonFunctions\'
      18:45:03 Added coverage information of 86/90 auto properties to module 'CommonFunctions'
      18:45:03 Initiating parser for OpenCover
      18:45:03 Current Assembly: CommonFunctions
      18:45:03 Current Assembly: CommonFunctionsTests
      18:45:03 Analyzing 89 classes
      18:45:03 Creating report 1/89 (Assembly: CommonFunctionsTests, Class: CommonFunctionsTest.CommonFunctionsTest)
      [....]
      18:45:04 Creating report 84/89 (Assembly: CommonFunctions, Class: TSTraceOrigin)
      18:45:04 Creating summary
      18:45:04 Report generation took 3 seconds
      18:45:04 Archiving artifacts
      18:45:04 [htmlpublisher] Archiving HTML reports...
      18:45:04 [htmlpublisher] Archiving at BUILD level D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\coverage to D:\Jenkins\jobs\CommonFunctionsTests\builds\2013-05-24_18-43-55\htmlreports\Coverage_Report
      18:45:06 [xUnit] [INFO] - Starting to record.
      18:45:06 [xUnit] [INFO] - Processing MSTest-Version N/A (default)
      18:45:06 [xUnit] [INFO] - [MSTest-Version N/A (default)] - 1 test report file(s) were found with the pattern 'ContestContinuous\contest_sw\TestResults\CommonFunctionsTests\CommonFunctionsTests.trx' relative to 'D:\WS\CommonFunctionsTests' for the testing framework 'MSTest-Version N/A (default)'.
      18:45:06 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      18:45:06 [xUnit] [INFO] - Check 'Skipped Tests' threshold.
      18:45:06 [xUnit] [INFO] - Setting the build status to SUCCESS
      18:45:06 [xUnit] [INFO] - Stopping recording.
      18:45:06 Recording fingerprints
      18:45:06 No emails were triggered.
      18:45:06
      18:45:06 Deleting project workspace...
      18:45:18 done
      18:45:18
      18:45:18 Notifying upstream projects of job completion
      18:45:18 Finished: SUCCESS

        Attachments

          Activity

            People

            • Assignee:
              gbois Gregory Boissinot
              Reporter:
              bruce Dirk Kuypers
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: