Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53186

NUnit 2.x test cases with result Error are reported as Success

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      xunit-plugin 2.2.3
      nunit3-console.exe 3.7.0
    • Similar Issues:

      Description

      We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner.

      With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported as failures anymore. Even worse, they are reported as success!

      TestResults.xml report file says:

      Jenkins / xunit-plugin 2.2.3 says:

      See the full example XML report attached

        Attachments

          Activity

          tknerr Torben Knerr created issue -
          tknerr Torben Knerr made changes -
          Field Original Value New Value
          Attachment Nunit3-in-nunit2-format-with-failures-TestReport.xml [ 43819 ]
          tknerr Torben Knerr made changes -
          Hide
          tknerr Torben Knerr added a comment -

          Nikolas Falco my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted. However, I manually changed this to "2.5.0.0" but still the same issue, failures were not reported correctly

          Note: we use "NUnitJunitHudsonTestType" assuming this will handle the NUnit2 format correctly (which it did in the xunit-plugin 1.102 version at least)

          Show
          tknerr Torben Knerr added a comment - Nikolas Falco my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted. However, I manually changed this to "2.5.0.0" but still the same issue, failures were not reported correctly Note: we use "NUnitJunitHudsonTestType" assuming this will handle the NUnit2 format correctly (which it did in the xunit-plugin 1.102 version at least)
          tknerr Torben Knerr made changes -
          Attachment screenshot-1.png [ 43821 ]
          tknerr Torben Knerr made changes -
          Attachment screenshot-2.png [ 43822 ]
          tknerr Torben Knerr made changes -
          Description We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner.

          With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported anymore.

          See the example XML report attached
          We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner.

          With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported as failures anymore. Even worse, they are reported as success!

          TestResults.xml report file says:

           !screenshot-2.png|thumbnail!

          Jenkins / xunit-plugin 2.2.3 says:

           !screenshot-1.png|thumbnail!

          See the full example XML report attached
          tknerr Torben Knerr made changes -
          Summary Test Failures not reported when using nunit3-console with nunit2 output formatter Test Failures reported as Success when using nunit3-console with nunit2 output formatter
          tknerr Torben Knerr made changes -
          Issue Type Improvement [ 4 ] Bug [ 1 ]
          tknerr Torben Knerr made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Hide
          nfalco Nikolas Falco added a comment -

          If you produce the NUnit3 why do you convert to NUni2 instead of configure the tool type to NUnit 3.x?

          Show
          nfalco Nikolas Falco added a comment - If you produce the NUnit3 why do you convert to NUni2 instead of configure the tool type to NUnit 3.x?
          Hide
          nfalco Nikolas Falco added a comment -

          Anyway let me have a look if result Error is a valid kind in NUnit2 and in case handle properly or it is a status in NUnit3 that is handled in wrong manner in the converter.

          I suggest if you have NUni3 to do not convert and use the tool type set to NUnit 3.x

          Show
          nfalco Nikolas Falco added a comment - Anyway let me have a look if result Error is a valid kind in NUnit2 and in case handle properly or it is a status in NUnit3 that is handled in wrong manner in the converter. I suggest if you have NUni3 to do not convert and use the tool type set to NUnit 3.x
          Hide
          nfalco Nikolas Falco added a comment -

          "my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted"

          This is a bug of adapter https://github.com/nunit/nunit-v2-result-writer/issues/11

          Show
          nfalco Nikolas Falco added a comment - "my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted" This is a bug of adapter https://github.com/nunit/nunit-v2-result-writer/issues/11
          nfalco Nikolas Falco made changes -
          Summary Test Failures reported as Success when using nunit3-console with nunit2 output formatter NUnit 2.x test cases with result Error are reported as Success
          nfalco Nikolas Falco made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          tknerr Torben Knerr added a comment - - edited

          Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3.

          Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here.

          Show
          tknerr Torben Knerr added a comment - - edited Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3. Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here.
          nfalco Nikolas Falco made changes -
          Status In Progress [ 3 ] Fixed but Unreleased [ 10203 ]
          Resolution Fixed [ 1 ]
          Hide
          tknerr Torben Knerr added a comment -

          Thanks Nikolas Falco, just saw you already fixed it in master!

          Any plans for pushing out a 2.2.4 release with that included anytime soon?

          Show
          tknerr Torben Knerr added a comment - Thanks Nikolas Falco , just saw you already fixed it in master! Any plans for pushing out a 2.2.4 release with that included anytime soon?
          Hide
          tknerr Torben Knerr added a comment -

          Hi Nikolas Falco, any news yet about a potential 2.2.4 release or expected timeline?

          Cheers, Torben

          Show
          tknerr Torben Knerr added a comment - Hi Nikolas Falco , any news yet about a potential 2.2.4 release or expected timeline? Cheers, Torben
          Hide
          nfalco Nikolas Falco added a comment -

          released in 2.2.4

          Show
          nfalco Nikolas Falco added a comment - released in 2.2.4
          nfalco Nikolas Falco made changes -
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              tknerr Torben Knerr
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: