Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24655

Maven Job use of testFailureIgnore option should be configurable

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Following the Jenkins Terminology, when (surefire or failsafe) tests fail, the Jenkins build status must be UNSTABLE:
      << A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable. >>

      Maven job

      That is the default case on Maven jobs; the code of https://github.com/jenkinsci/maven-plugin/blob/maven-plugin-2.6/src/main/java/hudson/maven/reporters/SurefireArchiver.java#L88 uses the -Dmaven.test.failure.ignore=true option:

      • the Maven execution is not interrupted by a test failure
      • Maven build is SUCCESS
      • Jenkins build is UNSTABLE

      However, that is a few error-prone and inconsistent with the command line or Freestyle job Maven execution.
      That would be highlighted by a Jenkins option to switch on or off that feature, rather than the need to know one must pass -Dmaven.test.failure.ignore=false to deactivate it.

      The error-prone aspect comes from the build continuation, which will for instance install the artifacts:

      [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
      [ERROR] There are test failures.
      [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
      [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
      [ERROR] There are test failures.
      [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
      [INFO] Installing ...
      [INFO] BUILD SUCCESS
      Finished: UNSTABLE

      So, it is recommended in a Maven job to call the package or verify phases rather than install or deploy and use the post-build steps to deploy the artifacts at the end (the new deployAtEnd Maven option is an alternative).

      It would be nice if there was a way to alternatively use for instance the -Dmaven.test.failure.ignore=false parameter and the -fae Maven 3 option ("Only fail the build afterwards; allow all non-impacted builds to continue") but get the final Maven build FAILED translated to Jenkins build UNSTABLE. For consistency with the terminology.

      Freestyle job

      About the Maven build step in Freestyle jobs, they do not behave that way by default. The result is the same as an execution from command line:

      • Maven execution is interrupted by tests failures
      • Maven build is FAILURE
      • Jenkins build is FAILURE

      Reproducing the Maven job behavior is possible with the -Dmaven.test.failure.ignore=true option and the use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).

      As for the Maven job, it would be great if the 'Publish JUnit test result report' post build step would be able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven test failures.

        Attachments

          Activity

          jcarsique Julien Carsique created issue -
          jcarsique Julien Carsique made changes -
          Field Original Value New Value
          Description I understood from various emails, issues, discussions and source code that it is a wanted behavior to set a build UNSTABLE when (surefire or failsafe) tests fail. That is consistent with https://wiki.jenkins-ci.org/display/JENKINS/Terminology

          However, the current behavior is error-prone and inconsistent with a command line or Freestyle job Maven execution.

          I guess that if the UNSTABLE status was that much important, it would have been implemented on Freestyle jobs too. Especially since Freestyle are so often recommended rather than the useful Maven job.

          h3. Problem

          First main issue is the build continuation.
          Second issue is the behavior difference between Maven and Freestyle jobs.

          h4. Freestyle job
          The result is the same as an execution from command line:
          - Maven execution is interrupted
          - Maven build is FAILURE
          - Jenkins build is FAILURE

          Reproducing the Maven job behavior is partially possible with {{-Dmaven.test.failure.ignore=true}} option and the use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).


          h4. Maven job
          The result is similar to using the {{-Dmaven.test.failure.ignore=true}} option:
          - *the Maven execution is not interrupted*
          - Maven build is SUCCESS
          - Jenkins build is UNSTABLE

          Responsible code is there: https://github.com/jenkinsci/maven-plugin/blob/maven-plugin-2.6/src/main/java/hudson/maven/reporters/SurefireArchiver.java#L88

          The main issue is the build continuation, which will for instance install the artifacts:
          {noformat}
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
          [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
          [INFO] Installing ...
          [INFO] BUILD SUCCESS
          Finished: UNSTABLE{noformat}

          ḧ3. Solutions

          h4. Add an option for ignoring or not the test failures during the build.

          An option would highlight that behavior and allow to easily switch from a testFailureIgnore mode to a standard mode.

          h4. Rather use {{-fae}} Maven 3 option, or improve current implementation not to use the testFailureIgnore property hack

          {noformat}-fae,--fail-at-end Only fail the build afterwards; allow all non-impacted builds to continue{noformat}

          That would not be the exact equivalent to the current implementation but would still allow to get a status on multiple modules and would only require to "catch" the Maven failure at the end of the build.

          Having a final Maven build FAILED translated to Jenkins build UNSTABLE would avoid the unwanted execution of other plugins after surefire or failsafe detected the failure.

          h4. Consistency with Freestyle job

          For consistency purpose, the 'Publish JUnit test result report' post build step should be able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven tests failure.
          I understood from various emails, issues, discussions and source code that it is a wanted behavior to set a build UNSTABLE when (surefire or failsafe) tests fail. That is consistent with https://wiki.jenkins-ci.org/display/JENKINS/Terminology

          However, the current behavior is error-prone and inconsistent with a command line or Freestyle job Maven execution.

          I guess that if the UNSTABLE status was that much important, it would have been implemented on Freestyle jobs too. Especially since Freestyle are so often recommended rather than the useful Maven job.

          h3. Problem

          First main issue is the build continuation.
          Second issue is the behavior difference between Maven and Freestyle jobs.

          h4. Freestyle job
          The result is the same as an execution from command line:
          - Maven execution is interrupted
          - Maven build is FAILURE
          - Jenkins build is FAILURE

          Reproducing the Maven job behavior is partially possible with {{-Dmaven.test.failure.ignore=true}} option and the use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).


          h4. Maven job
          The result is similar to using the {{-Dmaven.test.failure.ignore=true}} option:
          - *the Maven execution is not interrupted*
          - Maven build is SUCCESS
          - Jenkins build is UNSTABLE

          Responsible code is there: https://github.com/jenkinsci/maven-plugin/blob/maven-plugin-2.6/src/main/java/hudson/maven/reporters/SurefireArchiver.java#L88

          The main issue is the build continuation, which will for instance install the artifacts:
          {noformat}
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
          [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
          [INFO] Installing ...
          [INFO] BUILD SUCCESS
          Finished: UNSTABLE{noformat}

          h3. Solutions

          h4. Add an option for ignoring or not the test failures during the build.

          An option would highlight that behavior and allow to easily switch from a testFailureIgnore mode to a standard mode.

          h4. Rather use {{-fae}} Maven 3 option, or improve current implementation not to use the testFailureIgnore property hack

          {noformat}-fae,--fail-at-end Only fail the build afterwards; allow all non-impacted builds to continue{noformat}

          That would not be the exact equivalent to the current implementation but would still allow to get a status on multiple modules and would only require to "catch" the Maven failure at the end of the build.

          Having a final Maven build FAILED translated to Jenkins build UNSTABLE would avoid the unwanted execution of other plugins after surefire or failsafe detected the failure.

          h4. Consistency with Freestyle job

          For consistency purpose, the 'Publish JUnit test result report' post build step should be able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven tests failure.
          Hide
          jcarsique Julien Carsique added a comment -

          EDIT
          Changed from bug to improvement: that is not a bug. Changed the title and description. My mistake came from the following output in the console after having encountered false-positive errors due to the TASK and DRY plugins upgrade (I first though it was another similar issue):

          [ERROR] There are test failures.
          [INFO] BUILD SUCCESS
          Finished: UNSTABLE
          Show
          jcarsique Julien Carsique added a comment - EDIT Changed from bug to improvement: that is not a bug. Changed the title and description. My mistake came from the following output in the console after having encountered false-positive errors due to the TASK and DRY plugins upgrade (I first though it was another similar issue): [ERROR] There are test failures. [INFO] BUILD SUCCESS Finished: UNSTABLE
          jcarsique Julien Carsique made changes -
          Summary Maven Job use of testFailureIgnore option should be configurable or avoided Maven Job use of testFailureIgnore option should be configurable
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Description I understood from various emails, issues, discussions and source code that it is a wanted behavior to set a build UNSTABLE when (surefire or failsafe) tests fail. That is consistent with https://wiki.jenkins-ci.org/display/JENKINS/Terminology

          However, the current behavior is error-prone and inconsistent with a command line or Freestyle job Maven execution.

          I guess that if the UNSTABLE status was that much important, it would have been implemented on Freestyle jobs too. Especially since Freestyle are so often recommended rather than the useful Maven job.

          h3. Problem

          First main issue is the build continuation.
          Second issue is the behavior difference between Maven and Freestyle jobs.

          h4. Freestyle job
          The result is the same as an execution from command line:
          - Maven execution is interrupted
          - Maven build is FAILURE
          - Jenkins build is FAILURE

          Reproducing the Maven job behavior is partially possible with {{-Dmaven.test.failure.ignore=true}} option and the use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).


          h4. Maven job
          The result is similar to using the {{-Dmaven.test.failure.ignore=true}} option:
          - *the Maven execution is not interrupted*
          - Maven build is SUCCESS
          - Jenkins build is UNSTABLE

          Responsible code is there: https://github.com/jenkinsci/maven-plugin/blob/maven-plugin-2.6/src/main/java/hudson/maven/reporters/SurefireArchiver.java#L88

          The main issue is the build continuation, which will for instance install the artifacts:
          {noformat}
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
          [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
          [INFO] Installing ...
          [INFO] BUILD SUCCESS
          Finished: UNSTABLE{noformat}

          h3. Solutions

          h4. Add an option for ignoring or not the test failures during the build.

          An option would highlight that behavior and allow to easily switch from a testFailureIgnore mode to a standard mode.

          h4. Rather use {{-fae}} Maven 3 option, or improve current implementation not to use the testFailureIgnore property hack

          {noformat}-fae,--fail-at-end Only fail the build afterwards; allow all non-impacted builds to continue{noformat}

          That would not be the exact equivalent to the current implementation but would still allow to get a status on multiple modules and would only require to "catch" the Maven failure at the end of the build.

          Having a final Maven build FAILED translated to Jenkins build UNSTABLE would avoid the unwanted execution of other plugins after surefire or failsafe detected the failure.

          h4. Consistency with Freestyle job

          For consistency purpose, the 'Publish JUnit test result report' post build step should be able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven tests failure.
          Following the Jenkins [Terminology|https://wiki.jenkins-ci.org/display/JENKINS/Terminology], when (surefire or failsafe) tests fail, the Jenkins build status must be UNSTABLE:
          << A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable. >>

          h4. Maven job

          That is the default case on Maven jobs; the code of https://github.com/jenkinsci/maven-plugin/blob/maven-plugin-2.6/src/main/java/hudson/maven/reporters/SurefireArchiver.java#L88 uses the {{-Dmaven.test.failure.ignore=true}} option:
          - *the Maven execution is not interrupted by a test failure*
          - Maven build is SUCCESS
          - Jenkins build is UNSTABLE

          However, that is a few error-prone and inconsistent with the command line or Freestyle job Maven execution.
          That would be highlighted by a Jenkins option to switch on or off that feature, rather than the need to know one must pass {{-Dmaven.test.failure.ignore=false}} to deactivate it.

          The error-prone aspect comes from the build continuation, which will for instance install the artifacts:
          {noformat}
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
          [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
          [INFO] Installing ...
          [INFO] BUILD SUCCESS
          Finished: UNSTABLE{noformat}
          So, it is recommended in a Maven job to call the {{package} phase rather than {{install}} or {{deploy}} and use the post-build steps to deploy the artifacts at the end (the new deployAtEnd Maven option is an alternative).

          It would be nice if there was a way to alternatively use for instance the {{-fae}} Maven 3 option ("Only fail the build afterwards; allow all non-impacted builds to continue") and get the final Maven build FAILED translated to Jenkins build UNSTABLE...

          h4. Freestyle job

          About the Maven build step in Freestyle jobs, they do not behave that way by default. The result is the same as an execution from command line:
          - Maven execution is interrupted by tests failures
          - Maven build is FAILURE
          - Jenkins build is FAILURE

          Reproducing the Maven job behavior is possible with the {{-Dmaven.test.failure.ignore=true}} option and the use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).

          As for the Maven job, it would be great if the 'Publish JUnit test result report' post build step would be able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven test failures.
          jcarsique Julien Carsique made changes -
          Description Following the Jenkins [Terminology|https://wiki.jenkins-ci.org/display/JENKINS/Terminology], when (surefire or failsafe) tests fail, the Jenkins build status must be UNSTABLE:
          << A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable. >>

          h4. Maven job

          That is the default case on Maven jobs; the code of https://github.com/jenkinsci/maven-plugin/blob/maven-plugin-2.6/src/main/java/hudson/maven/reporters/SurefireArchiver.java#L88 uses the {{-Dmaven.test.failure.ignore=true}} option:
          - *the Maven execution is not interrupted by a test failure*
          - Maven build is SUCCESS
          - Jenkins build is UNSTABLE

          However, that is a few error-prone and inconsistent with the command line or Freestyle job Maven execution.
          That would be highlighted by a Jenkins option to switch on or off that feature, rather than the need to know one must pass {{-Dmaven.test.failure.ignore=false}} to deactivate it.

          The error-prone aspect comes from the build continuation, which will for instance install the artifacts:
          {noformat}
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
          [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
          [INFO] Installing ...
          [INFO] BUILD SUCCESS
          Finished: UNSTABLE{noformat}
          So, it is recommended in a Maven job to call the {{package} phase rather than {{install}} or {{deploy}} and use the post-build steps to deploy the artifacts at the end (the new deployAtEnd Maven option is an alternative).

          It would be nice if there was a way to alternatively use for instance the {{-fae}} Maven 3 option ("Only fail the build afterwards; allow all non-impacted builds to continue") and get the final Maven build FAILED translated to Jenkins build UNSTABLE...

          h4. Freestyle job

          About the Maven build step in Freestyle jobs, they do not behave that way by default. The result is the same as an execution from command line:
          - Maven execution is interrupted by tests failures
          - Maven build is FAILURE
          - Jenkins build is FAILURE

          Reproducing the Maven job behavior is possible with the {{-Dmaven.test.failure.ignore=true}} option and the use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).

          As for the Maven job, it would be great if the 'Publish JUnit test result report' post build step would be able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven test failures.
          Following the Jenkins [Terminology|https://wiki.jenkins-ci.org/display/JENKINS/Terminology], when (surefire or failsafe) tests fail, the Jenkins build status must be UNSTABLE:
          << A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable. >>

          h4. Maven job

          That is the default case on Maven jobs; the code of https://github.com/jenkinsci/maven-plugin/blob/maven-plugin-2.6/src/main/java/hudson/maven/reporters/SurefireArchiver.java#L88 uses the {{-Dmaven.test.failure.ignore=true}} option:
          - *the Maven execution is not interrupted by a test failure*
          - Maven build is SUCCESS
          - Jenkins build is UNSTABLE

          However, that is a few error-prone and inconsistent with the command line or Freestyle job Maven execution.
          That would be highlighted by a Jenkins option to switch on or off that feature, rather than the need to know one must pass {{-Dmaven.test.failure.ignore=false}} to deactivate it.

          The error-prone aspect comes from the build continuation, which will for instance install the artifacts:
          {noformat}
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ test-jenkins-failures ---
          [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ test-jenkins-failures ---
          [ERROR] There are test failures.
          [INFO] --- maven-install-plugin:2.5.1:install (default-install) @ test-jenkins-failures ---
          [INFO] Installing ...
          [INFO] BUILD SUCCESS
          Finished: UNSTABLE{noformat}
          So, it is recommended in a Maven job to call the {{package}} or {{verify}} phases rather than {{install}} or {{deploy}} and use the post-build steps to deploy the artifacts at the end (the new deployAtEnd Maven option is an alternative).

          It would be nice if there was a way to alternatively use for instance the {{-Dmaven.test.failure.ignore=false}} parameter and the {{-fae}} Maven 3 option ("Only fail the build afterwards; allow all non-impacted builds to continue") but get the final Maven build FAILED translated to Jenkins build UNSTABLE. For consistency with the terminology.

          h4. Freestyle job

          About the Maven build step in Freestyle jobs, they do not behave that way by default. The result is the same as an execution from command line:
          - Maven execution is interrupted by tests failures
          - Maven build is FAILURE
          - Jenkins build is FAILURE

          Reproducing the Maven job behavior is possible with the {{-Dmaven.test.failure.ignore=true}} option and the use of 'Publish JUnit test result report' post build step (to change the build result from SUCCESS to UNSTABLE).

          As for the Maven job, it would be great if the 'Publish JUnit test result report' post build step would be able to switch the Jenkins build result from FAILURE to UNSTABLE when the BUILD failure is only due to Maven test failures.
          Hide
          jordansilva Jordan Silva added a comment -

          I think that for consistence both Maven and Freestyle jobs should behave the same way. Freestyle job behavior is the most appropriated one.

          Show
          jordansilva Jordan Silva added a comment - I think that for consistence both Maven and Freestyle jobs should behave the same way. Freestyle job behavior is the most appropriated one.
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 157682 ] JNJira + In-Review [ 179650 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              jcarsique Julien Carsique
            • Votes:
              6 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated: