Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23480

Test Report shows test passed that actually did not run due to initialization errors

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • junit-plugin
    • None
    • Windows XP

      When running JUnit4 tests (using buckminster in our case, but I guess this is not relevant), failing class initializations (@BeforeClass) report the initialization as a failed additional test. All test cases located in this class however are reported as "passed", while "skipped" would be more appropriate.

      I'll attach

      • the test job definition
      • a small demo test suite which five test classes (each of them having a different behavior: success, ignored, assertionfailure, exception, exception in class setup)
      • screenshots showing the test reports generated (the uploaded test suite gives produced run#27)
      • the JUnit xml output generated by buckminster (calling junit) and used to produce the report (de.logiball.ci.testfailure.test_JUnit.xml)
      • the junitResult.xml generated from that

      The reports shown are misleading at least in case of "exception in class setup", I would guess also in case of "ignored" (while I am not sure where to report those two cases; mixing intentionally skipped tests with those skipped by an initialization error may also be a bad idea).

        1. config.xml
          3 kB
        2. JenkinsAllTestResults.png
          JenkinsAllTestResults.png
          20 kB
        3. JenkinsClassTestResults.png
          JenkinsClassTestResults.png
          11 kB
        4. JenkinsPackageTestResults.png
          JenkinsPackageTestResults.png
          22 kB
        5. JenkinsTestResultTrend.png
          JenkinsTestResultTrend.png
          55 kB
        6. de.logiball.ci.testfailure.test_JUnit.xml
          12 kB
        7. workspace.jenkins-ci.test.zip
          11 kB
        8. junitResult.xml
          14 kB

            Unassigned Unassigned
            bsailer Benjamin Sailer
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: