Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8713

Display skipped test messages in results

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: junit-plugin
    • Labels:
      None

      Description

      JUnit report XML can contain a message for skipped tests, which often contains information regarding why the test is being skipped. It would be very useful if this message could be output in the test results in a similar way to how the failure are shown.

      I've attached an example JUnit style report with a skipped test.

        Activity

        Hide
        nofinator Gregory Nofi added a comment - - edited

        The issue type is Improvement but I think this might be a bug because this used to work quite well.

        Any skipped tests were listed as such in the Test Report, and the number of skipped tests was represented by a nice yellow color in the Test Result Trend chart. Even now the Test Result still has a column for number of Skipped tests, but it is always set to 0 in each row even when there are skipped tests in the JUnit report.

        One theory: this may have started happening to our team after the Performance Plugin was installed, which I think has its own JUnit parser. However, we have this problem even in jobs that aren't using the Performance Plugin.

        Update: Skipped tests started showing up again in the Test Result Trend Chart for us again. Please disregard my comment, which should have been logged as a separate, new bug report instead.

        Show
        nofinator Gregory Nofi added a comment - - edited The issue type is Improvement but I think this might be a bug because this used to work quite well. Any skipped tests were listed as such in the Test Report, and the number of skipped tests was represented by a nice yellow color in the Test Result Trend chart. Even now the Test Result still has a column for number of Skipped tests, but it is always set to 0 in each row even when there are skipped tests in the JUnit report. One theory: this may have started happening to our team after the Performance Plugin was installed, which I think has its own JUnit parser. However, we have this problem even in jobs that aren't using the Performance Plugin. Update: Skipped tests started showing up again in the Test Result Trend Chart for us again. Please disregard my comment, which should have been logged as a separate, new bug report instead.
        Hide
        redsolo redsolo added a comment -

        Hi gregory, this issue should be marked as an improvement as it describes a feature that does not yet exist. Your problem sounds like a bug in an existing feature, so please create a new bug-report describing the problem and what plugins you are using (and version). Plus if you can, add a sample report that contains skipped tests to make easy to reproduce the issue.

        Show
        redsolo redsolo added a comment - Hi gregory, this issue should be marked as an improvement as it describes a feature that does not yet exist. Your problem sounds like a bug in an existing feature, so please create a new bug-report describing the problem and what plugins you are using (and version). Plus if you can, add a sample report that contains skipped tests to make easy to reproduce the issue.
        Hide
        yfyf Ignas Vyšnu added a comment -

        Bumping this and would like to clear some confusion: skipped tests are counted and the total number is displayed, I think there is no "bug" here, but the feature improvement would be to actually list/display (or at least have an option to do so) the skipped tests the same way failed tests are being listed under Test Result view.

        Show
        yfyf Ignas Vyšnu added a comment - Bumping this and would like to clear some confusion: skipped tests are counted and the total number is displayed, I think there is no "bug" here, but the feature improvement would be to actually list/display (or at least have an option to do so) the skipped tests the same way failed tests are being listed under Test Result view.
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Michael Clarke
        Path:
        core/src/main/java/hudson/tasks/junit/CaseResult.java
        core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly
        http://jenkins-ci.org/commit/jenkins/3b58cd294f8266f2e4dc80356bf172c37769030e
        Log:
        [FIXED JENKINS-8713] Display test skipped messages in test results

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Michael Clarke Path: core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly http://jenkins-ci.org/commit/jenkins/3b58cd294f8266f2e4dc80356bf172c37769030e Log: [FIXED JENKINS-8713] Display test skipped messages in test results
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Michael Clarke
        Path:
        changelog.html
        http://jenkins-ci.org/commit/jenkins/2aa0293858d50b064128a9a74f4d25d3f5c4a5c8
        Log:
        JENKINS-8713 Add changelog entry

        Compare: https://github.com/jenkinsci/jenkins/compare/fe9f67602877...2aa0293858d5


        You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
        For more options, visit https://groups.google.com/groups/opt_out.

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Michael Clarke Path: changelog.html http://jenkins-ci.org/commit/jenkins/2aa0293858d50b064128a9a74f4d25d3f5c4a5c8 Log: JENKINS-8713 Add changelog entry Compare: https://github.com/jenkinsci/jenkins/compare/fe9f67602877...2aa0293858d5 – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
        Hide
        dogfood dogfood added a comment -

        Integrated in jenkins_main_trunk #2369
        [FIXED JENKINS-8713] Display test skipped messages in test results (Revision 3b58cd294f8266f2e4dc80356bf172c37769030e)
        JENKINS-8713 Add changelog entry (Revision 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8)

        Result = UNSTABLE
        michael.m.clarke : 3b58cd294f8266f2e4dc80356bf172c37769030e
        Files :

        • core/src/main/java/hudson/tasks/junit/CaseResult.java
        • core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly

        michael.m.clarke : 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8
        Files :

        • changelog.html
        Show
        dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2369 [FIXED JENKINS-8713] Display test skipped messages in test results (Revision 3b58cd294f8266f2e4dc80356bf172c37769030e) JENKINS-8713 Add changelog entry (Revision 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8) Result = UNSTABLE michael.m.clarke : 3b58cd294f8266f2e4dc80356bf172c37769030e Files : core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly michael.m.clarke : 2aa0293858d50b064128a9a74f4d25d3f5c4a5c8 Files : changelog.html
        Hide
        sahumada sahumada added a comment -

        Hi,

        I've been trying to show my skipped tests without much success. My XML looks like:

        <?xml version="1.0" encoding="UTF-8"?>
        <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000">
           <properties>
              <property name="QtVersion" value="5.2.2"/>
              <property name="QTestVersion" value="5.2.2"/>
           </properties>
           <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000">
              <skipped>tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run.</skipped>
           </testcase>
           <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/>
           <system-out/>
           <system-err/>
        </testsuite>
        

        but it doesn't show anything.

        Looking at the source code, I see that you added an attribute called "message", so if I change my XML to

        <?xml version="1.0" encoding="UTF-8"?>
        <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000">
           <properties>
              <property name="QtVersion" value="5.2.2"/>
              <property name="QTestVersion" value="5.2.2"/>
           </properties>
           <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000">
              <skipped message="tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run."/>
           </testcase>
           <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/>
           <system-out/>
           <system-err/>
        </testsuite>
        

        then it fails because the XML is not valid according to junit-4.xsd for the xUnit plugin I am using.

        The error from the xUnit plugin I get is:

        13:03:18 [xUnit] [INFO] - Starting to record.
        13:03:18 [xUnit] [INFO] - Processing Custom Tool
        13:03:18 [xUnit] [INFO] - [Custom Tool] - 29 test report file(s) were found with the pattern 'test-results/tst_*.xml' relative to '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM' for the testing framework 'Custom Tool'.
        13:03:18 [xUnit] [ERROR] - The converted file for the result file '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM/test-results/tst_xmlpatterns.xml' (during conversion process for the metric 'Custom Tool') is not valid. The report file has been skipped.
        13:03:18 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated.
        13:03:18 [xUnit] [INFO] - There are errors when processing test results.
        13:03:18 [xUnit] [INFO] - Skipping tests recording.
        13:03:18 [xUnit] [INFO] - Stop build.
        

        So, is this change incorrect/incomplete or does junit-4.xsd/xUnit need to be updated ?

        Show
        sahumada sahumada added a comment - Hi, I've been trying to show my skipped tests without much success. My XML looks like: <?xml version="1.0" encoding="UTF-8"?> <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000"> <properties> <property name="QtVersion" value="5.2.2"/> <property name="QTestVersion" value="5.2.2"/> </properties> <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000"> <skipped>tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run.</skipped> </testcase> <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/> <system-out/> <system-err/> </testsuite> but it doesn't show anything. Looking at the source code, I see that you added an attribute called "message", so if I change my XML to <?xml version="1.0" encoding="UTF-8"?> <testsuite name="tst_XmlPatterns" tests="2" failures="0" errors="0" skipped="1" time="0.000"> <properties> <property name="QtVersion" value="5.2.2"/> <property name="QTestVersion" value="5.2.2"/> </properties> <testcase classname="tst_XmlPatterns" name="initTestCase" time="0.000"> <skipped message="tst_xmlpatterns.cpp:120 :: [] Device &#x9;IP:169.254.0.13&#xA;The command line tool (/usr/lib/qt5/bin/xmlpatterns) could not be run, possibly because Qt was incompletely built or installed. No tests will be run."/> </testcase> <testcase classname="tst_XmlPatterns" name="cleanupTestCase" time="0.000"/> <system-out/> <system-err/> </testsuite> then it fails because the XML is not valid according to junit-4.xsd for the xUnit plugin I am using. The error from the xUnit plugin I get is: 13:03:18 [xUnit] [INFO] - Starting to record. 13:03:18 [xUnit] [INFO] - Processing Custom Tool 13:03:18 [xUnit] [INFO] - [Custom Tool] - 29 test report file(s) were found with the pattern 'test-results/tst_*.xml' relative to '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM' for the testing framework 'Custom Tool'. 13:03:18 [xUnit] [ERROR] - The converted file for the result file '/var/lib/jenkins/jobs/qt-5-qtxmlpatterns-dev/workspace/ARCH/ARM/test-results/tst_xmlpatterns.xml' (during conversion process for the metric 'Custom Tool') is not valid. The report file has been skipped. 13:03:18 [xUnit] [INFO] - Failing BUILD because 'set build failed if errors' option is activated. 13:03:18 [xUnit] [INFO] - There are errors when processing test results. 13:03:18 [xUnit] [INFO] - Skipping tests recording. 13:03:18 [xUnit] [INFO] - Stop build. So, is this change incorrect/incomplete or does junit-4.xsd/xUnit need to be updated ?

          People

          • Assignee:
            mc1arke Michael Clarke
            Reporter:
            davehunt Dave Hunt
          • Votes:
            10 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: