Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53186

NUnit 2.x test cases with result Error are reported as Success

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      xunit-plugin 2.2.3
      nunit3-console.exe 3.7.0
    • Similar Issues:

      Description

      We are running `nunit3-console -result:TestResult.xml;format=nunit2` to produce NUnit2 compatible test reports with the NUnit3 runner.

      With the earlier xunit-plugin 1.102 version this worked just fine. Now with xunit-plugin 2.2.3, if we have errors / failures in the test results, these are not reported as failures anymore. Even worse, they are reported as success!

      TestResults.xml report file says:

      Jenkins / xunit-plugin 2.2.3 says:

      See the full example XML report attached

        Attachments

        1. Nunit3-in-nunit2-format-with-failures-TestReport.xml
          17 kB
          Torben Knerr
        2. screenshot-1.png
          114 kB
          Torben Knerr
        3. screenshot-2.png
          334 kB
          Torben Knerr

          Activity

          Hide
          tknerr Torben Knerr added a comment -

          Nikolas Falco my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted. However, I manually changed this to "2.5.0.0" but still the same issue, failures were not reported correctly

          Note: we use "NUnitJunitHudsonTestType" assuming this will handle the NUnit2 format correctly (which it did in the xunit-plugin 1.102 version at least)

          Show
          tknerr Torben Knerr added a comment - Nikolas Falco my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted. However, I manually changed this to "2.5.0.0" but still the same issue, failures were not reported correctly Note: we use "NUnitJunitHudsonTestType" assuming this will handle the NUnit2 format correctly (which it did in the xunit-plugin 1.102 version at least)
          Hide
          nfalco Nikolas Falco added a comment -

          If you produce the NUnit3 why do you convert to NUni2 instead of configure the tool type to NUnit 3.x?

          Show
          nfalco Nikolas Falco added a comment - If you produce the NUnit3 why do you convert to NUni2 instead of configure the tool type to NUnit 3.x?
          Hide
          nfalco Nikolas Falco added a comment -

          Anyway let me have a look if result Error is a valid kind in NUnit2 and in case handle properly or it is a status in NUnit3 that is handled in wrong manner in the converter.

          I suggest if you have NUni3 to do not convert and use the tool type set to NUnit 3.x

          Show
          nfalco Nikolas Falco added a comment - Anyway let me have a look if result Error is a valid kind in NUnit2 and in case handle properly or it is a status in NUnit3 that is handled in wrong manner in the converter. I suggest if you have NUni3 to do not convert and use the tool type set to NUnit 3.x
          Hide
          nfalco Nikolas Falco added a comment -

          "my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted"

          This is a bug of adapter https://github.com/nunit/nunit-v2-result-writer/issues/11

          Show
          nfalco Nikolas Falco added a comment - "my first impression was that the "nunit-version" field (which is set to "3.5.0.0" in the attached example) might get interpreted" This is a bug of adapter https://github.com/nunit/nunit-v2-result-writer/issues/11
          Hide
          tknerr Torben Knerr added a comment - - edited

          Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3.

          Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here.

          Show
          tknerr Torben Knerr added a comment - - edited Nikolas Falco the reason we are converting to nunit2 format is actually that the previous version of xunit-plugin we used (1.102) did not support nunit3 yet. So while we are using nunit2 format in other places, we could indeed try to use nunit3 format here to work around that issue with xunit-plugin 2.2.3. Regarding the nunit-version attribute, I suspected that the "3.5.0.0" version was the culprit. However, I changed that to "2.5.0.0" before the report publishing and still got the same misbehaviour. So I concluded that even though this is a bug in nunit-v2-result-writer (thanks for digging out the link to the github issue!) it would not resolve the issue reported here.
          Hide
          tknerr Torben Knerr added a comment -

          Thanks Nikolas Falco, just saw you already fixed it in master!

          Any plans for pushing out a 2.2.4 release with that included anytime soon?

          Show
          tknerr Torben Knerr added a comment - Thanks Nikolas Falco , just saw you already fixed it in master! Any plans for pushing out a 2.2.4 release with that included anytime soon?
          Hide
          tknerr Torben Knerr added a comment -

          Hi Nikolas Falco, any news yet about a potential 2.2.4 release or expected timeline?

          Cheers, Torben

          Show
          tknerr Torben Knerr added a comment - Hi Nikolas Falco , any news yet about a potential 2.2.4 release or expected timeline? Cheers, Torben
          Hide
          nfalco Nikolas Falco added a comment -

          released in 2.2.4

          Show
          nfalco Nikolas Falco added a comment - released in 2.2.4

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              tknerr Torben Knerr
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: