Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33450

Support for NUnit3 which uses a different XML format by NUnit2

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.625.3
      xUnit plugin 1.101
    • Similar Issues:

      Description

      xUnit appears to be incompatible with the test results report generated by NUnit 3. This is a snippet of the log produced when it tries to process the results file:

      10:00:44 [xUnit] [INFO] - Starting to record.
      10:00:44 [xUnit] [INFO] - Processing NUnit-Version N/A (default)
      10:00:45 [xUnit] [INFO] - [NUnit-Version N/A (default)] - 1 test report file(s) were found with the pattern 'UnitTestResults.xml' relative to 'E:\JPPSlave\workspace\Complete.Build.UT.NUnit3' for the testing framework 'NUnit-Version N/A (default)'.
      10:01:02 [xUnit] [ERROR] - The converted file for the result file 'E:\JPPSlave\workspace\Complete.Build.UT.NUnit3\UnitTestResults.xml' (during conversion process for the metric 'NUnit') is not valid. The report file has been skipped.
      10:01:02 [xUnit] [ERROR] - Content is not allowed in prolog.
      10:01:02 [xUnit] [ERROR] - Content is not allowed in prolog.
      10:01:02 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated.
      10:01:02 [xUnit] [INFO] - There are errors when processing test results.
      10:01:02 [xUnit] [INFO] - Skipping tests recording.
      10:01:02 [xUnit] [INFO] - Stop build.
      10:01:02 Build step 'Publish xUnit test result report' changed build result to FAILURE

      Initially we had this issue using quite an old version of xUnit (1.60) but the issue still persists after upgrading to the latest version (1.101). This is a blocker for us upgrading our code-base to NUnit 3 because we rely on the xUnit plugin processing our report file in order to pass or fail builds based on unit test success.

      The most recent comment on JENKINS-27043 seems to relate to the same issue but as that bug wasn't reported for the same reason I thought it would be best to create this as a stand-alone issue as it's quite specifically a compatibility problem with NUnit 3.

        Attachments

          Activity

          Hide
          teognost Adi Adrian added a comment -

          Is there any plan to make xUnit plugin to support NUnit 3.x?

          Show
          teognost Adi Adrian added a comment - Is there any plan to make xUnit plugin to support NUnit 3.x?
          Hide
          lsolano Lorenzo Solano Martinez added a comment -

          Hi Will Appleby and Jason Carlberg,

          I did a simmilar trick, as follow:

          From https://github.com/nunit/nunit-console on branch release/3.5 I found the following test scenario

          • Name-space: NUnit.ConsoleRunner.Tests
            • Test Class: CommandLineTests
              • Scenario: ResultOptionWithFilePathAndFormat()
                • Options: ConsoleOptions options = new ConsoleOptions("tests.dll", "-result:results.xml;format=nunit2");

          After changing my NUnit run step on Jenkins from old v2 console runner to V3
          "%NUNIT3_HOME%\nunit3-console.exe" NUnitTests.nunit /config=%MSBuildConfiguration% --framework=net-4.5 -result:TestResult.xml;format=nunit2

          The last option is what did the trick: -result:TestResult.xml;format=nunit2 , that way you tell the console runner to write the output in the V2 format and also to use the legacy (v2 default) name TestResult.xml.

          I'm also waiting to use a future release of the xUnit plugin with support for NUnit 3.x. I believe that the conversion process is a little bit slower for the v3 console runner.

          Environment:

          • Windows 2012 R2
          • Jenkins: 1.651.1
          • xUnit Plugin: 1.102
          • NUnit: 3.5 (framework and console runner with v2 driver / plugin)
          Show
          lsolano Lorenzo Solano Martinez added a comment - Hi Will Appleby and Jason Carlberg , I did a simmilar trick, as follow: From https://github.com/nunit/nunit-console on branch release/3.5 I found the following test scenario Name-space: NUnit.ConsoleRunner.Tests Test Class: CommandLineTests Scenario: ResultOptionWithFilePathAndFormat() Options: ConsoleOptions options = new ConsoleOptions("tests.dll", "-result:results.xml;format=nunit2"); After changing my NUnit run step on Jenkins from old v2 console runner to V3 "%NUNIT3_HOME%\nunit3-console.exe" NUnitTests.nunit /config=%MSBuildConfiguration% --framework=net-4.5 -result:TestResult.xml;format=nunit2 The last option is what did the trick: -result:TestResult.xml;format=nunit2 , that way you tell the console runner to write the output in the V2 format and also to use the legacy (v2 default) name TestResult.xml . I'm also waiting to use a future release of the xUnit plugin with support for NUnit 3.x. I believe that the conversion process is a little bit slower for the v3 console runner. Environment: Windows 2012 R2 Jenkins: 1.651.1 xUnit Plugin: 1.102 NUnit: 3.5 (framework and console runner with v2 driver / plugin)
          Hide
          bjornr Bjørn Reppen added a comment -

          The new dotnet-test-nunit vtest runner version 3.4.0-beta-2 doesn't seem to support nunit2 format anymore which can be troublesome if you want to test ASP.NET Core applications.

          Show
          bjornr Bjørn Reppen added a comment - The new dotnet-test-nunit vtest runner version 3.4.0-beta-2 doesn't seem to support nunit2 format anymore which can be troublesome if you want to test ASP.NET Core applications.
          Hide
          willapp Will Appleby added a comment -

          Hi Jason Carlberg,

          I hadn't actually seen that format argument in the NUnit docs, so you're comment is really handy as we can now upgrade to NUnit 3 and still use xUnit by making it output the results file in compatibility mode with NUnit 2. Obviously it would be great if xUnit could be upgraded to support the newer format but this is actually a really helpful solution.

          https://github.com/nunit/docs/wiki/Console-Command-Line

          Thanks,
          Will.

          Show
          willapp Will Appleby added a comment - Hi Jason Carlberg , I hadn't actually seen that format argument in the NUnit docs, so you're comment is really handy as we can now upgrade to NUnit 3 and still use xUnit by making it output the results file in compatibility mode with NUnit 2. Obviously it would be great if xUnit could be upgraded to support the newer format but this is actually a really helpful solution. https://github.com/nunit/docs/wiki/Console-Command-Line Thanks, Will.
          Hide
          carlberg Jason Carlberg added a comment -

          We are experiencing the same issue with Jenkins 1.625.2, xUnit plugin 1.101. Currently working around by appending ";format=nunit2" to nunit3-console.exe build step but would like to have NUnit 3 format supported.

          Show
          carlberg Jason Carlberg added a comment - We are experiencing the same issue with Jenkins 1.625.2, xUnit plugin 1.101. Currently working around by appending ";format=nunit2" to nunit3-console.exe build step but would like to have NUnit 3 format supported.

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              willapp Will Appleby
            • Votes:
              13 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: