Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37598

Inaccurate aggregation of multiple xmls containing case information about the same testsuite

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: junit-plugin
    • Labels:
      None
    • Environment:
      junit plugin 1.18
    • Similar Issues:

      Description

      When the same testsuite appears in multiple result xmls; the current aggregation logic fails to aggregate neither the sum of the testcases - and neither the total testclass runtime - instead it reports only 1 testclass'es runtime.

      example xml:

      <?xml version="1.0" encoding="UTF-8"?>
      <testsuites name="Automation Tests" tests="2" errors="0" failures="0" ignored="0">
          <testsuite name="test.fs.FileSystemTests" time="50">
            <testcase name="testPrefix1" classname="test.fs.FileSystemTest1" time="10"/>
          </testsuite>
          <testsuite name="test.fs.FileSystemTests" time="50">
            <testcase name="testPrefix2" classname="test.fs.FileSystemTest1" time="10"/>
          </testsuite>
      </testsuites>
      

      in the above example the runtime of test.fs.FileSystemTests should be 100

      Encountered during hive / maven 2.19.1 upgrade..details:
      https://issues.apache.org/jira/browse/HIVE-14572

        Attachments

          Issue Links

            Activity

            kirk Zoltán Haindrich created issue -
            kirk Zoltán Haindrich made changes -
            Field Original Value New Value
            Summary Inaccurate affregation of multiple xmls containing case information about the same testsuite Inaccurate aggregation of multiple xmls containing case information about the same testsuite
            kirk Zoltán Haindrich made changes -
            Remote Link This issue links to "pull request (Web Link)" [ 14752 ]
            kirk Zoltán Haindrich made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            kirk Zoltán Haindrich made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            kirk Zoltán Haindrich made changes -
            Status In Review [ 10005 ] In Progress [ 3 ]
            Hide
            kirk Zoltán Haindrich added a comment -

            I'm not sure which stats should I leave this ticket in...i've looked into a few...and haven't seen any in IN REVIEW
            I think the code in the pull request should take care of the issue

            Show
            kirk Zoltán Haindrich added a comment - I'm not sure which stats should I leave this ticket in...i've looked into a few...and haven't seen any in IN REVIEW I think the code in the pull request should take care of the issue
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            kirk Zoltán Haindrich added a comment -

            Jesse Glick can you take a look at the update changes?

            Show
            kirk Zoltán Haindrich added a comment - Jesse Glick can you take a look at the update changes?
            Hide
            kirk Zoltán Haindrich added a comment -

            Oleg Nenashev, Jesse Glick Can you please take a look at the changes...it would be great to have this fixed:

            I would like to propose a new way to run tests in parallel which would not need any effort on the user side - by extending the parallel test executor plugin with a different approach (which would not be prone to lose testcases)....this problem would arise in that scenario as well because that will possibly "split" a junit4 testcase in 2 or more pieces...which would be affected by the same timing inconsistency issues which this issue solves.

            Show
            kirk Zoltán Haindrich added a comment - Oleg Nenashev , Jesse Glick Can you please take a look at the changes...it would be great to have this fixed: I would like to propose a new way to run tests in parallel which would not need any effort on the user side - by extending the parallel test executor plugin with a different approach (which would not be prone to lose testcases)....this problem would arise in that scenario as well because that will possibly "split" a junit4 testcase in 2 or more pieces...which would be affected by the same timing inconsistency issues which this issue solves.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Haindrich Zoltán (kirk)
            Path:
            src/main/java/hudson/tasks/junit/ClassResult.java
            src/main/java/hudson/tasks/junit/SuiteResult.java
            src/main/java/hudson/tasks/junit/TestResult.java
            src/test/java/hudson/tasks/junit/TestResultTest.java
            src/test/resources/hudson/tasks/junit/junit-report-time-aggregation-invalid.xml
            src/test/resources/hudson/tasks/junit/junit-report-time-aggregation.xml
            src/test/resources/hudson/tasks/junit/junit-report-time-aggregation2.xml
            http://jenkins-ci.org/commit/junit-plugin/8f3473b215130ea3e537ca41a97df9a5572d1c2d
            Log:
            JENKINS-37598: Inaccurate aggregation of multiple xmls containing case information about the same testsuite

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Haindrich Zoltán (kirk) Path: src/main/java/hudson/tasks/junit/ClassResult.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/resources/hudson/tasks/junit/junit-report-time-aggregation-invalid.xml src/test/resources/hudson/tasks/junit/junit-report-time-aggregation.xml src/test/resources/hudson/tasks/junit/junit-report-time-aggregation2.xml http://jenkins-ci.org/commit/junit-plugin/8f3473b215130ea3e537ca41a97df9a5572d1c2d Log: JENKINS-37598 : Inaccurate aggregation of multiple xmls containing case information about the same testsuite
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Zoltan Haindrich
            Path:
            src/main/java/hudson/tasks/junit/SuiteResult.java
            src/test/java/hudson/tasks/junit/TestResultTest.java
            src/test/resources/hudson/tasks/junit/junit-report-time-aggregation-invalid.xml
            http://jenkins-ci.org/commit/junit-plugin/319280f67394b2b0eda1fba589912f30bbd3ebcc
            Log:
            JENKINS-37598: address review comments; put back time field, doesn't throw exception

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Zoltan Haindrich Path: src/main/java/hudson/tasks/junit/SuiteResult.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/resources/hudson/tasks/junit/junit-report-time-aggregation-invalid.xml http://jenkins-ci.org/commit/junit-plugin/319280f67394b2b0eda1fba589912f30bbd3ebcc Log: JENKINS-37598 : address review comments; put back time field, doesn't throw exception
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/main/java/hudson/tasks/junit/ClassResult.java
            src/main/java/hudson/tasks/junit/SuiteResult.java
            src/main/java/hudson/tasks/junit/TestResult.java
            src/test/java/hudson/tasks/junit/TestResultTest.java
            src/test/resources/hudson/tasks/junit/junit-report-time-aggregation.xml
            src/test/resources/hudson/tasks/junit/junit-report-time-aggregation2.xml
            http://jenkins-ci.org/commit/junit-plugin/d083744e64c6f1a084bc032f9ef8f8a9e45cf38e
            Log:
            Merge pull request #54 from kgyrtkirk/JENKINS-37598

            JENKINS-37598 Inaccurate aggregation of multiple xmls containing case information about the same testsuite

            Compare: https://github.com/jenkinsci/junit-plugin/compare/6243fa595ab3...d083744e64c6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/main/java/hudson/tasks/junit/ClassResult.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/resources/hudson/tasks/junit/junit-report-time-aggregation.xml src/test/resources/hudson/tasks/junit/junit-report-time-aggregation2.xml http://jenkins-ci.org/commit/junit-plugin/d083744e64c6f1a084bc032f9ef8f8a9e45cf38e Log: Merge pull request #54 from kgyrtkirk/ JENKINS-37598 JENKINS-37598 Inaccurate aggregation of multiple xmls containing case information about the same testsuite Compare: https://github.com/jenkinsci/junit-plugin/compare/6243fa595ab3...d083744e64c6
            Hide
            kirk Zoltán Haindrich added a comment -

            merged; will be fixed in >=1.22

            Show
            kirk Zoltán Haindrich added a comment - merged; will be fixed in >=1.22
            kirk Zoltán Haindrich made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]

              People

              • Assignee:
                kirk Zoltán Haindrich
                Reporter:
                kirk Zoltán Haindrich
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: