Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47194

xUnit should not consider report generated by other xUnit step

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      node('master') {
          stage('1') {
              step([$class: 'XUnitBuilder',
                            thresholds: [[$class: 'FailedThreshold', failureThreshold: "1"]],
                             tools: [[$class: 'JUnitType', pattern: "tmp.xml"]]])
          }
          stage('2') {
              step([$class: 'XUnitBuilder',
                            thresholds: [[$class: 'FailedThreshold', failureThreshold: "0"]],
                             tools: [[$class: 'JUnitType', pattern: "tmp2.xml"]]])       
          }
      }
      

      tmp.xml contains one failure and tmp2.xml contains no failures.

       

      The result is as followed:

      [test_pipeline] Running shell script
      [Pipeline] step
      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing JUnit
      [xUnit] [INFO] - [JUnit] - 1 test report file(s) were found with the pattern 'tmp.xml' relative to '/var/lib/jenkins/workspace/test_pipeline' for the testing framework 'JUnit'.
      [xUnit] [INFO] - Check 'Failed Tests' threshold.
      [xUnit] [INFO] - Setting the build status to SUCCESS
      [xUnit] [INFO] - Stopping recording.
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (2)
      [Pipeline] sh
      [test_pipeline] Running shell script
      [Pipeline] step
      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing JUnit
      [xUnit] [INFO] - [JUnit] - 1 test report file(s) were found with the pattern 'tmp2.xml' relative to '/var/lib/jenkins/workspace/test_pipeline' for the testing framework 'JUnit'.
      [xUnit] [INFO] - Check 'Failed Tests' threshold.
      [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
      [xUnit] [INFO] - Setting the build status to FAILURE
      [xUnit] [INFO] - Stopping recording.
      

      Xunit keeps track of previous recorded failures, while it, in my opinion, should not

        Attachments

          Issue Links

            Activity

            Hide
            abayer Andrew Bayer added a comment -

            JENKINS-27395 may be relevant here.

            Show
            abayer Andrew Bayer added a comment - JENKINS-27395 may be relevant here.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Nikolas Falco
            Path:
            src/main/java/org/jenkinsci/plugins/xunit/XUnitProcessor.java
            src/main/java/org/jenkinsci/plugins/xunit/service/XUnitTransformer.java
            src/test/java/org/jenkinsci/plugins/xunit/XUnitPublisherTest.java
            src/test/java/org/jenkinsci/plugins/xunit/XUnitWorkflowTest.java
            src/test/java/org/jenkinsci/plugins/xunit/service/TempWorkspace.java
            src/test/java/org/jenkinsci/plugins/xunit/service/XUnitTransformerTest.java
            src/test/java/org/jenkinsci/plugins/xunit/service/XUnitValidationServiceTest.java
            src/test/resources/org/jenkinsci/plugins/xunit/XUnitPublisherTest/workspace/JENKINS-47194/input.xml
            src/test/resources/org/jenkinsci/plugins/xunit/config.xml
            http://jenkins-ci.org/commit/xunit-plugin/10d52804cbf03c855fc0b9208ff6d8a62cf0efa0
            Log:
            JENKINS-47194 xUnit should not consider report generated by other xUnit step

            XUnit build step or publisher use an instance of processor than generates output report to a shared common folder. This cause on latest step thresholds consider also reports produced by previous steps. Processor now generate an identifier that permits to generate reports into an unique output folder.

            *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

            Functionality will be removed from GitHub.com on January 31st, 2019.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nikolas Falco Path: src/main/java/org/jenkinsci/plugins/xunit/XUnitProcessor.java src/main/java/org/jenkinsci/plugins/xunit/service/XUnitTransformer.java src/test/java/org/jenkinsci/plugins/xunit/XUnitPublisherTest.java src/test/java/org/jenkinsci/plugins/xunit/XUnitWorkflowTest.java src/test/java/org/jenkinsci/plugins/xunit/service/TempWorkspace.java src/test/java/org/jenkinsci/plugins/xunit/service/XUnitTransformerTest.java src/test/java/org/jenkinsci/plugins/xunit/service/XUnitValidationServiceTest.java src/test/resources/org/jenkinsci/plugins/xunit/XUnitPublisherTest/workspace/ JENKINS-47194 /input.xml src/test/resources/org/jenkinsci/plugins/xunit/config.xml http://jenkins-ci.org/commit/xunit-plugin/10d52804cbf03c855fc0b9208ff6d8a62cf0efa0 Log: JENKINS-47194 xUnit should not consider report generated by other xUnit step XUnit build step or publisher use an instance of processor than generates output report to a shared common folder. This cause on latest step thresholds consider also reports produced by previous steps. Processor now generate an identifier that permits to generate reports into an unique output folder. * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.
            Hide
            stefanthurnherr Stefan Thurnherr added a comment -

            I'm still experiencing this problem with xUnit 2.0.0 on jenkins 2.126. I just posted to jenkinsci-users mailing list: https://groups.google.com/d/msg/jenkinsci-users/Wk2QfWdsRFo/8pNpN3jDBAAJ

            Does the fix in 2.0.0 work for anyone else?

            Show
            stefanthurnherr Stefan Thurnherr added a comment - I'm still experiencing this problem with xUnit 2.0.0 on jenkins 2.126. I just posted to jenkinsci-users mailing list: https://groups.google.com/d/msg/jenkinsci-users/Wk2QfWdsRFo/8pNpN3jDBAAJ Does the fix in 2.0.0 work for anyone else?
            Hide
            nfalco Nikolas Falco added a comment - - edited

            This bug is not still fixed completely. Generated report files does not interfere with ones generated by other but thresholds are calculated against the global TestResult object in the TestResultAction for the current job.

            Each step must generates single TestResult and use it for threasholds, but we need hack the TestResult object to use merge method.

            Show
            nfalco Nikolas Falco added a comment - - edited This bug is not still fixed completely. Generated report files does not interfere with ones generated by other but thresholds are calculated against the global TestResult object in the TestResultAction for the current job. Each step must generates single TestResult and use it for threasholds, but we need hack the TestResult object to use merge method.
            Hide
            stefanthurnherr Stefan Thurnherr added a comment -

            Nikolas Falco ok good to know, thanks for the info!

            Also when testing a bit more I found that the same problem occurs if one stage uses xUnit and the other stage uses the basic junit() step. Will your fix work only if all test result analysis uses xUnit?

            Show
            stefanthurnherr Stefan Thurnherr added a comment - Nikolas Falco ok good to know, thanks for the info! Also when testing a bit more I found that the same problem occurs if one stage uses xUnit and the other stage uses the basic junit() step. Will your fix work only if all test result analysis uses xUnit?
            Hide
            nfalco Nikolas Falco added a comment -

            The threshold will be always applied against the sum of the tools you had specify in the step, not anymore against the global sum of all gathered tests in the job.

            Show
            nfalco Nikolas Falco added a comment - The threshold will be always applied against the sum of the tools you had specify in the step, not anymore against the global sum of all gathered tests in the job.

              People

              • Assignee:
                nfalco Nikolas Falco
                Reporter:
                roel0 roel postelmans
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: