Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49344

Add support for categorizing tests to xunit plugin

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: xunit-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      It would be nice to be able to categorize tests by providing an optional "category" parameter to the xunit publisher. 

      The specific use case I had in mind was to provide some ability to group test results based on stages in a Jenkins pipeline build. If the xunit test publisher accepted a "category" parameter, one could provide some unique name - like the name of the current build stage - as the category, allowing all the tests reported by that publisher to be grouped together and kept separate from the results produced by other stages.

      In general, I think this feature would be handy for legacy and single-stage test suites as well. Suppose several teams are writing tests for a project and you want to group the tests from team 1 together, team 2, etc. You could then have spearate publish operations for the different xUnit report files, each with a custom "team" based category, so their results can be grouped together.

      As for the visual representation, I was thinking that maybe a small row of tabs above or below the test report table would work, with one tab labelled "all" and each of the others labelled with the associated category name. In this way, if you wanted to see information on a specific group of tests you could just select the appropriate tab / category.

        Attachments

          Issue Links

            Activity

            Hide
            leedega Kevin Phillips added a comment -

            As I had mentioned in JENKINS-49322, the reason I've set the initial priority of this improvement request to major is because at the moment if one has a multi-stage pipeline build that produces multiple xUnit test reports, the behavior is essentially broken. Any 2 test suites that happen to have the same test names (which has happened to us in production btw) run by the same pipeline build have name clashes in the xunit publisher. Similarly, when one has multiple test stages in a single pipeline, and each of which has hundreds or thousands of tests in them, it becomes prohibitively difficult to sift through the test reports to find out which parts of the test reports apply to which build stages, making debugging difficult if not impossible. This becomes even more of a problem in practice when multiple teams are working on different test suites, because then it is hard to assign blame to a specific team when certain tests fail.

            Show
            leedega Kevin Phillips added a comment - As I had mentioned in JENKINS-49322 , the reason I've set the initial priority of this improvement request to major is because at the moment if one has a multi-stage pipeline build that produces multiple xUnit test reports, the behavior is essentially broken. Any 2 test suites that happen to have the same test names (which has happened to us in production btw) run by the same pipeline build have name clashes in the xunit publisher. Similarly, when one has multiple test stages in a single pipeline, and each of which has hundreds or thousands of tests in them, it becomes prohibitively difficult to sift through the test reports to find out which parts of the test reports apply to which build stages, making debugging difficult if not impossible. This becomes even more of a problem in practice when multiple teams are working on different test suites, because then it is hard to assign blame to a specific team when certain tests fail.

              People

              • Assignee:
                nfalco Nikolas Falco
                Reporter:
                leedega Kevin Phillips
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: