Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31258

Jenkins Maven plug-in ignores JUnit-format test results from unknown Maven plug-ins

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: maven-plugin
    • Labels:
      None
    • Environment:
      Windows 8.1. JavaSE-1.8. Jenkins 1.635.
    • Similar Issues:

      Description

      The Jenkins Maven plug-in ignores JUnit format files in the test results directory (${project.build.directory}/surefire-reports by default). Debugging revealed the cause to be that maven-plugin knows about certain* test-capable MOJOs and attempts to support unknown MOJOs by making assumptions which may or may not be justified, depending on the MOJO in question:

      1. that the MOJO will target the 'test' goal or similar
      2. that the MOJO configuration will have a 'reportsDirectory' property

      * maven testing plug-ins known to the Jenkins Maven plug-in / project type as of 2014-12-04 13:40:29:

      • org.apache.maven.plugins:maven-surefire-plugin
      • net.kennychua:phantomjs-qunit-runner
      • org.apache.maven.plugins:maven-failsafe-plugin
      • com.sun.maven:maven-junit-plugin
      • org.sonatype.flexmojos:flexmojos-maven-plugin
      • org.sonatype.tycho:maven-osgi-test-plugin
      • org.eclipse.tycho:tycho-surefire-plugin
      • com.jayway.maven.plugins.android.generation2:maven-android-plugin
      • com.jayway.maven.plugins.android.generation2:android-maven-plugin
      • org.codehaus.mojo:gwt-maven-plugin
      • com.smartbear.soapui:soapui-maven-plugin
      • com.smartbear.soapui:soapui-pro-maven-plugin
      • com.github.redfish4ktc.soapui:maven-soapui-extension-plugin
      • com.github.searls:jasmine-maven-plugin
      • org.terracotta.maven.plugins:toolkit-resolver-plugin
      • org.scalatest:scalatest-maven-plugin
      • org.nanoko.playframework:play2-maven-plugin

      Examples of unknown/unsupported Maven test-capable plug-ins:

      • com.github.eirslett:frontend-maven-plugin:karma
      • com.kelveden:maven-karma-plugin:start
      • com.github.greengerong:maven-ng-protractor:run
      • org.codehaus.mojo:exec-maven-plugin:exec (can launch any arbitrary testing process)
      • others, doubtless...

      In the general 'unknown Maven plug-in' case neither of these assumptions are likely to be valid e.g., com.github.eirslett:frontend-maven-plugin running Karma tests, the goal is 'karma' (phase = 'test' or 'integration-test', typically) and the test output directory is configured in a separate karma.conf.js file.

      Overall, this means there is no general solution for reporting standard JUnit-format results that have been placed in the Maven-standard location for such results, other than to use only plug-ins that the Jenkins Maven plugin actually knows about! This is inflexible and an unreasonable limitation for a general-purpose CI tool and a highly extensible build tool.

      I see three possible solutions:

      1. The Jenkins Job configuration for Maven projects provides some declarative means to identify additional test reports.
        • PROs:
          • straightforward to configure
          • no Jenkins-specific POM changes required
          • only one set of changes required
        • CONs:
          • requires explicit configuration in every Jenkins Maven job
      2. Develop a Maven plug-in that could be incorporated into a POM's pluginManagement section purely as the means to indicate that a given plug-in execution (specified as the groupdId, artifactId, version, goal and phase of the otherwise unknown-to-Jenkins Maven test plug-in) produces test results in the specified directory).
        • PROs:
          • should work OOTB in any up-to-date Jenkins installation
          • the Maven plug-in could be extended to cover additional capabilities where Jenkins' built-in Maven integration is inadquate
        • CONs:
          • additional POM complexity purely to support Jenkins integration
          • requires changes to the Jenkins-Maven integration plus a new Maven plug-in
      3. Support a Maven property name format that can supply the equivalent information. Something like:
        • jenkins.<goal>/<group-id>:<artifact-id>[:<version>].reportsDirectory = <test-reports-path>
          • e.g. jenkins.integration-test/org.codehaus.mojo:exec-maven-plugin:1.4.0.reportsDirectory = target/surefire-reports (inexpressible in XML)
        • jenkins.<plugin-execution-id>.reportsDirectory = <test-reports-path>
          • e.g. jenkins.run-protractor-tests.reportsDirectory = target/surefire-reports
        • PROs:
          • minimal changes required to implement
          • should work OOTB in any up-to-date Jenkins installation
        • CONs:
          • somewhat arcane syntax, not readily discoverable
          • sets a sub-optimal precedent for other such extension metadata

      I have prepared a simple implementation of Option 3 for which I am awaiting legal approval to submit a Pull Request.

        Attachments

          Issue Links

            Activity

            Hide
            adrianp Adrian Price added a comment -

            Fixed in Pull Request #56.

            Show
            adrianp Adrian Price added a comment - Fixed in Pull Request #56 .
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: demonfiddler
            Path:
            src/main/java/hudson/maven/reporters/TestMojo.java
            src/main/resources/hudson/maven/Messages.properties
            src/test/java/hudson/maven/MojoInfoBuilder.java
            src/test/java/hudson/maven/reporters/TestMojoTest.java
            http://jenkins-ci.org/commit/maven-plugin/bb25b8524cbaa2f5c11226140617c994672b3ce5
            Log:
            [FIXED JENKINS-31258]

            • Unknown test plug-ins can now specify the path to their reports directory via the Maven project property jenkins.<execution-id>.reportsDirectory. Relative paths are resolved against the project's base directory.
            • Unit test for the above capability.
            • Fixed a compilation failure that was apparently caused an extraneous (and inconsistent!) Unicode right-apostrophe character instead of a single quote in Messages.properties
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: demonfiddler Path: src/main/java/hudson/maven/reporters/TestMojo.java src/main/resources/hudson/maven/Messages.properties src/test/java/hudson/maven/MojoInfoBuilder.java src/test/java/hudson/maven/reporters/TestMojoTest.java http://jenkins-ci.org/commit/maven-plugin/bb25b8524cbaa2f5c11226140617c994672b3ce5 Log: [FIXED JENKINS-31258] Unknown test plug-ins can now specify the path to their reports directory via the Maven project property jenkins.<execution-id>.reportsDirectory. Relative paths are resolved against the project's base directory. Unit test for the above capability. Fixed a compilation failure that was apparently caused an extraneous (and inconsistent!) Unicode right-apostrophe character instead of a single quote in Messages.properties
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Adrian Price
            Path:
            src/main/java/hudson/maven/reporters/SurefireArchiver.java
            src/main/java/hudson/maven/reporters/TestMojo.java
            src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
            src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
            http://jenkins-ci.org/commit/maven-plugin/45712ad8938580c1ae1a8c76095917f119d3f0be
            Log:
            Implement Oleg's code review suggestions:

            • replace extraneous tab characters with spaces
            • follow Oracle coding standards
            • remove unnecessary null check
            • add test for JENKINS-31258 'detect test Mojo identified by Jenkins Maven Property'
            • add test for JENKINS-31524 'ensure updated existing results are counted'
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Adrian Price Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java src/main/java/hudson/maven/reporters/TestMojo.java src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java http://jenkins-ci.org/commit/maven-plugin/45712ad8938580c1ae1a8c76095917f119d3f0be Log: Implement Oleg's code review suggestions: replace extraneous tab characters with spaces follow Oracle coding standards remove unnecessary null check add test for JENKINS-31258 'detect test Mojo identified by Jenkins Maven Property' add test for JENKINS-31524 'ensure updated existing results are counted'
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Olivier Lamy
            Path:
            src/main/java/hudson/maven/reporters/SurefireArchiver.java
            src/main/java/hudson/maven/reporters/TestMojo.java
            src/main/resources/hudson/maven/Messages.properties
            src/test/java/hudson/maven/MojoInfoBuilder.java
            src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java
            src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
            src/test/java/hudson/maven/reporters/TestMojoTest.java
            http://jenkins-ci.org/commit/maven-plugin/3ba74bd9564ce7af7300825341e4d392a2fa61bc
            Log:
            Merge pull request #56 from demonfiddler/master

            Merge fixes for JENKINS-31258 and JENKINS-31524.
            As it's merged now @demonfiddler please document the new expression in the wiki page.

            Compare: https://github.com/jenkinsci/maven-plugin/compare/0223184e413c...3ba74bd9564c

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Olivier Lamy Path: src/main/java/hudson/maven/reporters/SurefireArchiver.java src/main/java/hudson/maven/reporters/TestMojo.java src/main/resources/hudson/maven/Messages.properties src/test/java/hudson/maven/MojoInfoBuilder.java src/test/java/hudson/maven/reporters/SurefireArchiverDetectTestMojosTest.java src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java src/test/java/hudson/maven/reporters/TestMojoTest.java http://jenkins-ci.org/commit/maven-plugin/3ba74bd9564ce7af7300825341e4d392a2fa61bc Log: Merge pull request #56 from demonfiddler/master Merge fixes for JENKINS-31258 and JENKINS-31524 . As it's merged now @demonfiddler please document the new expression in the wiki page. Compare: https://github.com/jenkinsci/maven-plugin/compare/0223184e413c...3ba74bd9564c
            Hide
            aheritier Arnaud Héritier added a comment -

            Adrian Price could update the documentation please (wiki) ?

            Show
            aheritier Arnaud Héritier added a comment - Adrian Price could update the documentation please (wiki) ?
            Hide
            adrianp Adrian Price added a comment -

            Arnaud, I documented the new Maven property and related matters (see the new paragraph headed 'Maven Surefire Test Results') on Mar 02, 2016 10:12.

            Show
            adrianp Adrian Price added a comment - Arnaud, I documented the new Maven property and related matters (see the new paragraph headed 'Maven Surefire Test Results') on Mar 02, 2016 10:12.
            Hide
            aheritier Arnaud Héritier added a comment -

            thanks adrianprice
            Fixed in 2.13

            Show
            aheritier Arnaud Héritier added a comment - thanks adrianprice Fixed in 2.13

              People

              • Assignee:
                Unassigned
                Reporter:
                adrianp Adrian Price
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: