Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12417

xUnit problems with CppTest reports generated with default options

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: xunit-plugin
    • Labels:
      None
    • Environment:
      xUnit 1.32
      Jenkins 1.424.1
      C++test 7.3.2.97
    • Similar Issues:

      Description

      We are encountering some problems of xUnit when parsing Parasoft C++test results.

      • When parsing results output with C++test default options, xUnit fails when all tests succeed.
      • xUnit miscounts the number of failures and total, when multiple assertions fail in one test case.
      • When a certain C++test report option enabled, xUnit outputs "ERROR" to console log.

      To help you reproduce above problems, I attach a zip file,
      which includes some report.xml files output by C++test, (and also report.html)

      Below are expected results and obtained ones by xUnit,
      using each report.xml file in each folder of the zip as an input for xUnit.

      A) report_with_no_coverage_detail/

                               Expected   xUnit result
      a1) all_success_no_opt/    0/20      ERROR
      a2) all_success/           0/20      OK (0/20)
      a3) mix/                   1/20      OK (1/20)
      a4) 2assertion_failure/    1/20      NG (2/21)
                                (M/N means failures/total of tests)
      

      a1) is the case when all tests succeeded and report.xml was output with C++test default options.
      This case caused an xUnit error below:

      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing CppTest-7.3 (default)
      [xUnit] [INFO] - [CppTest-7.3 (default)] - 1 test report file(s) were found with the pattern 'repout/report.xml' relative to '/home/user/slave01/workspace/test-cpptest-unit' for the testing framework 'CppTest-7.3 (default)'.
      [xUnit] [ERROR] - The plugin hasn't been performed correctly: None of the test reports contained any result
      Build step 'Publish testing tools result report' changed build result to FAILURE
      Build step 'Publish testing tools result report' marked build as failure
      Notifying upstream projects of job completion
      Finished: FAILURE
      

      When we used xUnit 1.34 insted of 1.32, xUnit 1.34 didn't cause an error,
      but the result was "no tests" though 20 tests existed.

      [xUnit] [WARNING] - All test reports are empty
      

      a2) is same to a1), but report.xml was output with C++test report option
      "Overview of checked files and executed tests" enabled.
      This case looks OK.
      (In this case, report.xml looks to have additional <ExecutedTestsDetails> elements)

      a3) is the case when one test failed.
      This looks OK.

      a4) is the case when one test failed, but two assertions failed in that one failed test.
      xUnit seems to wrongly increase the number of failures and total for each assertion failure in one failed test.

      B) report_with_coverage_detail/

      This folder contains the cases same to A),
      but report.xml files were output with C++test report option "Detailed report for Line coverage" enabled.

      Each report.xml file with this option looks to contain an additional <Files> element,
      as the last child of Exec/Coverage element.

      The results are same to A), but xUnit output ERROR message to console log:

      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing CppTest-7.3 (default)
      [xUnit] [INFO] - [CppTest-7.3 (default)] - 1 test report file(s) were found with the pattern 'repout/report.xml' relative to '/home/baba/slave01/workspace/test-cpptest-unit' for the testing framework 'CppTest-7.3 (default)'.
      [xUnit] [ERROR] - The converted file for the result file '/home/user/slave01/workspace/test-cpptest-unit/repout/report.xml' (during conversion process for the metric 'CppTest') is not valid. The report file has been skipped.
      [xUnit] [INFO] - Check 'Failed Tests' threshold.
      [xUnit] [INFO] - Check 'Skipped Tests' threshold.
      [xUnit] [INFO] - Setting the build status to SUCCESS
      [xUnit] [INFO] - Stopping recording.
      

      xUnit seems to have failed to parse report.xml, but continue processing regardless of "skipped" and "ERROR".


      If you need some more information, please let me know.
      Thanks in advance.

        Attachments

          Activity

          Hide
          egsejenkins Reece Johnston added a comment -

          I am having the same issue except with version 9.1.0.214 of the C++test tool.

          Show
          egsejenkins Reece Johnston added a comment - I am having the same issue except with version 9.1.0.214 of the C++test tool.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Nikolas Falco
          Path:
          src/main/resources/org/jenkinsci/plugins/xunit/types/cpptestunit-2.0-to-junit-1.0.xsl
          src/test/java/org/jenkinsci/plugins/xunit/types/AbstractTest.java
          src/test/java/org/jenkinsci/plugins/xunit/types/CppTestTest.java
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase1/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase4/input.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase4/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase5/input.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase5/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase6/input.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase6/result.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase7/input.xml
          src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase7/result.xml
          http://jenkins-ci.org/commit/xunit-plugin/c9cecd58f640ed8dda0c8859c3fcbae5a063c4d5
          Log:
          JENKINS-12417 xUnit problems with CppTest reports generated with
          default options

          The default CLI options seems not produce the ExecutedTestsDetails
          element in the output report where are described test suites. In absense
          of ExecutedTestsDetails we try to do our best to generate all testcase
          elements from ExecViols. The number of testcase could differs from the
          total count of executed test.

          Compare: https://github.com/jenkinsci/xunit-plugin/compare/6aed82bc92a5...c9cecd58f640
          *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

          Functionality will be removed from GitHub.com on January 31st, 2019.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nikolas Falco Path: src/main/resources/org/jenkinsci/plugins/xunit/types/cpptestunit-2.0-to-junit-1.0.xsl src/test/java/org/jenkinsci/plugins/xunit/types/AbstractTest.java src/test/java/org/jenkinsci/plugins/xunit/types/CppTestTest.java src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase1/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase4/input.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase4/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase5/input.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase5/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase6/input.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase6/result.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase7/input.xml src/test/resources/org/jenkinsci/plugins/xunit/types/cpptest/testcase7/result.xml http://jenkins-ci.org/commit/xunit-plugin/c9cecd58f640ed8dda0c8859c3fcbae5a063c4d5 Log: JENKINS-12417 xUnit problems with CppTest reports generated with default options The default CLI options seems not produce the ExecutedTestsDetails element in the output report where are described test suites. In absense of ExecutedTestsDetails we try to do our best to generate all testcase elements from ExecViols. The number of testcase could differs from the total count of executed test. Compare: https://github.com/jenkinsci/xunit-plugin/compare/6aed82bc92a5...c9cecd58f640 * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              yamabb Yama Tksh
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: