Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31258

Jenkins Maven plug-in ignores JUnit-format test results from unknown Maven plug-ins

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: maven-plugin
    • Labels:
      None
    • Environment:
      Windows 8.1. JavaSE-1.8. Jenkins 1.635.
    • Similar Issues:

      Description

      The Jenkins Maven plug-in ignores JUnit format files in the test results directory (${project.build.directory}/surefire-reports by default). Debugging revealed the cause to be that maven-plugin knows about certain* test-capable MOJOs and attempts to support unknown MOJOs by making assumptions which may or may not be justified, depending on the MOJO in question:

      1. that the MOJO will target the 'test' goal or similar
      2. that the MOJO configuration will have a 'reportsDirectory' property

      * maven testing plug-ins known to the Jenkins Maven plug-in / project type as of 2014-12-04 13:40:29:

      • org.apache.maven.plugins:maven-surefire-plugin
      • net.kennychua:phantomjs-qunit-runner
      • org.apache.maven.plugins:maven-failsafe-plugin
      • com.sun.maven:maven-junit-plugin
      • org.sonatype.flexmojos:flexmojos-maven-plugin
      • org.sonatype.tycho:maven-osgi-test-plugin
      • org.eclipse.tycho:tycho-surefire-plugin
      • com.jayway.maven.plugins.android.generation2:maven-android-plugin
      • com.jayway.maven.plugins.android.generation2:android-maven-plugin
      • org.codehaus.mojo:gwt-maven-plugin
      • com.smartbear.soapui:soapui-maven-plugin
      • com.smartbear.soapui:soapui-pro-maven-plugin
      • com.github.redfish4ktc.soapui:maven-soapui-extension-plugin
      • com.github.searls:jasmine-maven-plugin
      • org.terracotta.maven.plugins:toolkit-resolver-plugin
      • org.scalatest:scalatest-maven-plugin
      • org.nanoko.playframework:play2-maven-plugin

      Examples of unknown/unsupported Maven test-capable plug-ins:

      • com.github.eirslett:frontend-maven-plugin:karma
      • com.kelveden:maven-karma-plugin:start
      • com.github.greengerong:maven-ng-protractor:run
      • org.codehaus.mojo:exec-maven-plugin:exec (can launch any arbitrary testing process)
      • others, doubtless...

      In the general 'unknown Maven plug-in' case neither of these assumptions are likely to be valid e.g., com.github.eirslett:frontend-maven-plugin running Karma tests, the goal is 'karma' (phase = 'test' or 'integration-test', typically) and the test output directory is configured in a separate karma.conf.js file.

      Overall, this means there is no general solution for reporting standard JUnit-format results that have been placed in the Maven-standard location for such results, other than to use only plug-ins that the Jenkins Maven plugin actually knows about! This is inflexible and an unreasonable limitation for a general-purpose CI tool and a highly extensible build tool.

      I see three possible solutions:

      1. The Jenkins Job configuration for Maven projects provides some declarative means to identify additional test reports.
        • PROs:
          • straightforward to configure
          • no Jenkins-specific POM changes required
          • only one set of changes required
        • CONs:
          • requires explicit configuration in every Jenkins Maven job
      2. Develop a Maven plug-in that could be incorporated into a POM's pluginManagement section purely as the means to indicate that a given plug-in execution (specified as the groupdId, artifactId, version, goal and phase of the otherwise unknown-to-Jenkins Maven test plug-in) produces test results in the specified directory).
        • PROs:
          • should work OOTB in any up-to-date Jenkins installation
          • the Maven plug-in could be extended to cover additional capabilities where Jenkins' built-in Maven integration is inadquate
        • CONs:
          • additional POM complexity purely to support Jenkins integration
          • requires changes to the Jenkins-Maven integration plus a new Maven plug-in
      3. Support a Maven property name format that can supply the equivalent information. Something like:
        • jenkins.<goal>/<group-id>:<artifact-id>[:<version>].reportsDirectory = <test-reports-path>
          • e.g. jenkins.integration-test/org.codehaus.mojo:exec-maven-plugin:1.4.0.reportsDirectory = target/surefire-reports (inexpressible in XML)
        • jenkins.<plugin-execution-id>.reportsDirectory = <test-reports-path>
          • e.g. jenkins.run-protractor-tests.reportsDirectory = target/surefire-reports
        • PROs:
          • minimal changes required to implement
          • should work OOTB in any up-to-date Jenkins installation
        • CONs:
          • somewhat arcane syntax, not readily discoverable
          • sets a sub-optimal precedent for other such extension metadata

      I have prepared a simple implementation of Option 3 for which I am awaiting legal approval to submit a Pull Request.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                adrianp Adrian Price
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: