Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50635

NUnit plugin does not fail build on test-runner error

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: nunit-plugin
    • Labels:
    • Environment:
      Jenkins ver. 2.89.4
      NUnit Plugin
    • Similar Issues:

      Description

      When there is an error in unit tests that causes the test runner to crash (running nunit3-console with --process=Separate) the test report does not contain any test cases but reports an error on test-suite level. (see attached report0.xml)

      The nunit-plugin should fail the build if such a report file is found. Currently it seems to silently ignore the error and our build job succeeded although there was a massive bug in one of our tests.

        Attachments

          Activity

          Hide
          slide_o_mix Alex Earl added a comment -

          I don't think failing the build would be correct for everyone, I would propose treating Error status as a failing test and mark the build as unstable as is done for failing tests.

          Show
          slide_o_mix Alex Earl added a comment - I don't think failing the build would be correct for everyone, I would propose treating Error status as a failing test and mark the build as unstable as is done for failing tests.
          Hide
          exomo Kai Bublitz added a comment -

          I guess that would be ok for us as well, we have configured the build job to fail on unstable anyway.
           

          Show
          exomo Kai Bublitz added a comment - I guess that would be ok for us as well, we have configured the build job to fail on unstable anyway.  
          Hide
          slide_o_mix Alex Earl added a comment -

          Would it be possible for you to enable the option to save the JUnit files that are generated and then upload the resulting JUnit file? The current XSLT transform should actually work for the case of setting it to unstable, so I want to see what the resulting transform creates.

          Show
          slide_o_mix Alex Earl added a comment - Would it be possible for you to enable the option to save the JUnit files that are generated and then upload the resulting JUnit file? The current XSLT transform should actually work for the case of setting it to unstable, so I want to see what the resulting transform creates.
          Hide
          exomo Kai Bublitz added a comment -

          We could do this if you can tell me how. This might look like a stupid questions but I have no idea how to enable keeping JUnit files. There is no option for this, so what should I do?

          Show
          exomo Kai Bublitz added a comment - We could do this if you can tell me how. This might look like a stupid questions but I have no idea how to enable keeping JUnit files. There is no option for this, so what should I do?
          Hide
          slide_o_mix Alex Earl added a comment -

          The option is in the global configuration area for NUnit.

          Manage Jenkins > Global Configuration

          Show
          slide_o_mix Alex Earl added a comment - The option is in the global configuration area for NUnit. Manage Jenkins > Global Configuration
          Hide
          exomo Kai Bublitz added a comment -

          I looked through all the "Manage Jenkins" but I couldn't find anything related to nunit settings. There is no "Global Configuration" in "Manage Jenkins" on our server.
           

          Show
          exomo Kai Bublitz added a comment - I looked through all the "Manage Jenkins" but I couldn't find anything related to nunit settings. There is no "Global Configuration" in "Manage Jenkins" on our server.  
          Hide
          slide_o_mix Alex Earl added a comment -

          It looks like debug mode is only available if you connect via a debugger, so that doesn't help much. I'll just try running your file through the xslt transform locally.

          Show
          slide_o_mix Alex Earl added a comment - It looks like debug mode is only available if you connect via a debugger, so that doesn't help much. I'll just try running your file through the xslt transform locally.
          Hide
          slide_o_mix Alex Earl added a comment -

          So, what looks like is happening is the NUnit XML output supports a failure at the test-suite level, which I am not sure the JUnit format supports (which is used to display the results). The schema for the JUnit format, and what is supported by the JUnit plugin is not very well documented, so I will need to determine if its possible to support this sort of thing.

          Show
          slide_o_mix Alex Earl added a comment - So, what looks like is happening is the NUnit XML output supports a failure at the test-suite level, which I am not sure the JUnit format supports (which is used to display the results). The schema for the JUnit format, and what is supported by the JUnit plugin is not very well documented, so I will need to determine if its possible to support this sort of thing.

            People

            • Assignee:
              Unassigned
              Reporter:
              exomo Kai Bublitz
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: