Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8713

Display skipped test messages in results

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: junit-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      JUnit report XML can contain a message for skipped tests, which often contains information regarding why the test is being skipped. It would be very useful if this message could be output in the test results in a similar way to how the failure are shown.

      I've attached an example JUnit style report with a skipped test.

        Attachments

          Activity

          Hide
          nofinator Gregory Nofi added a comment - - edited

          The issue type is Improvement but I think this might be a bug because this used to work quite well.

          Any skipped tests were listed as such in the Test Report, and the number of skipped tests was represented by a nice yellow color in the Test Result Trend chart. Even now the Test Result still has a column for number of Skipped tests, but it is always set to 0 in each row even when there are skipped tests in the JUnit report.

          One theory: this may have started happening to our team after the Performance Plugin was installed, which I think has its own JUnit parser. However, we have this problem even in jobs that aren't using the Performance Plugin.

          Update: Skipped tests started showing up again in the Test Result Trend Chart for us again. Please disregard my comment, which should have been logged as a separate, new bug report instead.

          Show
          nofinator Gregory Nofi added a comment - - edited The issue type is Improvement but I think this might be a bug because this used to work quite well. Any skipped tests were listed as such in the Test Report, and the number of skipped tests was represented by a nice yellow color in the Test Result Trend chart. Even now the Test Result still has a column for number of Skipped tests, but it is always set to 0 in each row even when there are skipped tests in the JUnit report. One theory: this may have started happening to our team after the Performance Plugin was installed, which I think has its own JUnit parser. However, we have this problem even in jobs that aren't using the Performance Plugin. Update: Skipped tests started showing up again in the Test Result Trend Chart for us again. Please disregard my comment, which should have been logged as a separate, new bug report instead.
          Hide
          redsolo redsolo added a comment -

          Hi gregory, this issue should be marked as an improvement as it describes a feature that does not yet exist. Your problem sounds like a bug in an existing feature, so please create a new bug-report describing the problem and what plugins you are using (and version). Plus if you can, add a sample report that contains skipped tests to make easy to reproduce the issue.

          Show
          redsolo redsolo added a comment - Hi gregory, this issue should be marked as an improvement as it describes a feature that does not yet exist. Your problem sounds like a bug in an existing feature, so please create a new bug-report describing the problem and what plugins you are using (and version). Plus if you can, add a sample report that contains skipped tests to make easy to reproduce the issue.
          Hide
          yfyf Ignas Vyšnu added a comment -

          Bumping this and would like to clear some confusion: skipped tests are counted and the total number is displayed, I think there is no "bug" here, but the feature improvement would be to actually list/display (or at least have an option to do so) the skipped tests the same way failed tests are being listed under Test Result view.

          Show
          yfyf Ignas Vyšnu added a comment - Bumping this and would like to clear some confusion: skipped tests are counted and the total number is displayed, I think there is no "bug" here, but the feature improvement would be to actually list/display (or at least have an option to do so) the skipped tests the same way failed tests are being listed under Test Result view.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Michael Clarke
          Path:
          core/src/main/java/hudson/tasks/junit/CaseResult.java
          core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly
          http://jenkins-ci.org/commit/jenkins/3b58cd294f8266f2e4dc80356bf172c37769030e
          Log:
          [FIXED JENKINS-8713] Display test skipped messages in test results

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Michael Clarke Path: core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly http://jenkins-ci.org/commit/jenkins/3b58cd294f8266f2e4dc80356bf172c37769030e Log: [FIXED JENKINS-8713] Display test skipped messages in test results
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Michael Clarke
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/2aa0293858d50b064128a9a74f4d25d3f5c4a5c8
          Log:
          JENKINS-8713 Add changelog entry

          Compare: https://github.com/jenkinsci/jenkins/compare/fe9f67602877...2aa0293858d5


          You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
          For more options, visit https://groups.google.com/groups/opt_out.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Michael Clarke Path: changelog.html http://jenkins-ci.org/commit/jenkins/2aa0293858d50b064128a9a74f4d25d3f5c4a5c8 Log: JENKINS-8713 Add changelog entry Compare: https://github.com/jenkinsci/jenkins/compare/fe9f67602877...2aa0293858d5 – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #2369
          [FIXED JENKINS-8713] Display test skipped messages in test results (Revision 3b58cd294f8266f2e4dc80356bf172c37769030e)
          JENKINS-8713 Add changelog entry (Revision 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8)

          Result = UNSTABLE
          michael.m.clarke : 3b58cd294f8266f2e4dc80356bf172c37769030e
          Files :

          • core/src/main/java/hudson/tasks/junit/CaseResult.java
          • core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly

          michael.m.clarke : 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8
          Files :

          • changelog.html
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2369 [FIXED JENKINS-8713] Display test skipped messages in test results (Revision 3b58cd294f8266f2e4dc80356bf172c37769030e) JENKINS-8713 Add changelog entry (Revision 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8) Result = UNSTABLE michael.m.clarke : 3b58cd294f8266f2e4dc80356bf172c37769030e Files : core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly michael.m.clarke : 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8 Files : changelog.html
          Hide
          sahumada sahumada added a comment -

          Hi,

          I've been trying to show my skipped tests without much success. My XML looks like:

          <?xml version="1.0" encoding="UTF-8"?>
          <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000">
             <properties>
                <property name="QtVersion" value="5.2.2"/>
                <property name="QTestVersion" value="5.2.2"/>
             </properties>
             <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000">
                <skipped>tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run.</skipped>
             </testcase>
             <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/>
             <system-out/>
             <system-err/>
          </testsuite>
          

          but it doesn't show anything.

          Looking at the source code, I see that you added an attribute called "message", so if I change my XML to

          <?xml version="1.0" encoding="UTF-8"?>
          <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000">
             <properties>
                <property name="QtVersion" value="5.2.2"/>
                <property name="QTestVersion" value="5.2.2"/>
             </properties>
             <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000">
                <skipped message="tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run."/>
             </testcase>
             <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/>
             <system-out/>
             <system-err/>
          </testsuite>
          

          then it fails because the XML is not valid according to junit-4.xsd for the xUnit plugin I am using.

          The error from the xUnit plugin I get is:

          13:03:18 [xUnit] [INFO] - Starting to record.
          13:03:18 [xUnit] [INFO] - Processing Custom Tool
          13:03:18 [xUnit] [INFO] - [Custom Tool] - 29 test report file(s) were found with the pattern 'test-results/tst_*.xml' relative to '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM' for the testing framework 'Custom Tool'.
          13:03:18 [xUnit] [ERROR] - The converted file for the result file '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM/test-results/tst_xmlpatterns.xml' (during conversion process for the metric 'Custom Tool') is not valid. The report file has been skipped.
          13:03:18 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated.
          13:03:18 [xUnit] [INFO] - There are errors when processing test results.
          13:03:18 [xUnit] [INFO] - Skipping tests recording.
          13:03:18 [xUnit] [INFO] - Stop build.
          

          So, is this change incorrect/incomplete or does junit-4.xsd/xUnit need to be updated ?

          Show
          sahumada sahumada added a comment - Hi, I've been trying to show my skipped tests without much success. My XML looks like: <?xml version="1.0" encoding="UTF-8"?> <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000"> <properties> <property name="QtVersion" value="5.2.2"/> <property name="QTestVersion" value="5.2.2"/> </properties> <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000"> <skipped>tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run.</skipped> </testcase> <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/> <system-out/> <system-err/> </testsuite> but it doesn't show anything. Looking at the source code, I see that you added an attribute called "message", so if I change my XML to <?xml version="1.0" encoding="UTF-8"?> <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000"> <properties> <property name="QtVersion" value="5.2.2"/> <property name="QTestVersion" value="5.2.2"/> </properties> <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000"> <skipped message="tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run."/> </testcase> <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/> <system-out/> <system-err/> </testsuite> then it fails because the XML is not valid according to junit-4.xsd for the xUnit plugin I am using. The error from the xUnit plugin I get is: 13:03:18 [xUnit] [INFO] - Starting to record. 13:03:18 [xUnit] [INFO] - Processing Custom Tool 13:03:18 [xUnit] [INFO] - [Custom Tool] - 29 test report file(s) were found with the pattern 'test-results/tst_*.xml' relative to '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM' for the testing framework 'Custom Tool'. 13:03:18 [xUnit] [ERROR] - The converted file for the result file '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM/test-results/tst_xmlpatterns.xml' (during conversion process for the metric 'Custom Tool') is not valid. The report file has been skipped. 13:03:18 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated. 13:03:18 [xUnit] [INFO] - There are errors when processing test results. 13:03:18 [xUnit] [INFO] - Skipping tests recording. 13:03:18 [xUnit] [INFO] - Stop build. So, is this change incorrect/incomplete or does junit-4.xsd/xUnit need to be updated ?

            People

            • Assignee:
              mc1arke Michael Clarke
              Reporter:
              davehunt Dave Hunt
            • Votes:
              10 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: