Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49344

Add support for categorizing tests to xunit plugin

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: xunit-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      It would be nice to be able to categorize tests by providing an optional "category" parameter to the xunit publisher. 

      The specific use case I had in mind was to provide some ability to group test results based on stages in a Jenkins pipeline build. If the xunit test publisher accepted a "category" parameter, one could provide some unique name - like the name of the current build stage - as the category, allowing all the tests reported by that publisher to be grouped together and kept separate from the results produced by other stages.

      In general, I think this feature would be handy for legacy and single-stage test suites as well. Suppose several teams are writing tests for a project and you want to group the tests from team 1 together, team 2, etc. You could then have spearate publish operations for the different xUnit report files, each with a custom "team" based category, so their results can be grouped together.

      As for the visual representation, I was thinking that maybe a small row of tabs above or below the test report table would work, with one tab labelled "all" and each of the others labelled with the associated category name. In this way, if you wanted to see information on a specific group of tests you could just select the appropriate tab / category.

        Attachments

          Issue Links

            Activity

            Hide
            leedega Kevin Phillips added a comment -

            As I had mentioned in JENKINS-49322, the reason I've set the initial priority of this improvement request to major is because at the moment if one has a multi-stage pipeline build that produces multiple xUnit test reports, the behavior is essentially broken. Any 2 test suites that happen to have the same test names (which has happened to us in production btw) run by the same pipeline build have name clashes in the xunit publisher. Similarly, when one has multiple test stages in a single pipeline, and each of which has hundreds or thousands of tests in them, it becomes prohibitively difficult to sift through the test reports to find out which parts of the test reports apply to which build stages, making debugging difficult if not impossible. This becomes even more of a problem in practice when multiple teams are working on different test suites, because then it is hard to assign blame to a specific team when certain tests fail.

            Show
            leedega Kevin Phillips added a comment - As I had mentioned in JENKINS-49322 , the reason I've set the initial priority of this improvement request to major is because at the moment if one has a multi-stage pipeline build that produces multiple xUnit test reports, the behavior is essentially broken. Any 2 test suites that happen to have the same test names (which has happened to us in production btw) run by the same pipeline build have name clashes in the xunit publisher. Similarly, when one has multiple test stages in a single pipeline, and each of which has hundreds or thousands of tests in them, it becomes prohibitively difficult to sift through the test reports to find out which parts of the test reports apply to which build stages, making debugging difficult if not impossible. This becomes even more of a problem in practice when multiple teams are working on different test suites, because then it is hard to assign blame to a specific team when certain tests fail.
            Hide
            quas Jakub Pawlinski added a comment -

            My use case would be to separate x86 and x64 test results.

            Doing them in same stage (due to missing dynamic stages in declarative) via dotCover.

            Currently I get 2 results for each and cannot tell if it is x64 or x86

            Show
            quas Jakub Pawlinski added a comment - My use case would be to separate x86 and x64 test results. Doing them in same stage (due to missing dynamic stages in declarative) via dotCover. Currently I get 2 results for each and cannot tell if it is x64 or x86
            Hide
            nfalco Nikolas Falco added a comment -

            How you distinguish the test cases x86 from x64? Based on report path ?

            Show
            nfalco Nikolas Falco added a comment - How you distinguish the test cases x86 from x64? Based on report path ?
            Hide
            quas Jakub Pawlinski added a comment -

            Only by reading log at the moment I assumed I will find solution when I will reach it, but found   JENKINS-45826 instead and that leaded me here

            Show
            quas Jakub Pawlinski added a comment - Only by reading log at the moment I assumed I will find solution when I will reach it, but found    JENKINS-45826  instead and that leaded me here

              People

              • Assignee:
                nfalco Nikolas Falco
                Reporter:
                leedega Kevin Phillips
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: