Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35169

Jenkins "Job status" shows success even if it has many / all test cases failed

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Not A Defect
    • Component/s: testng-plugin
    • Labels:
      None
    • Environment:
      Windows 7, Chrome 50.0.2661.102, Jenkins 2.5,2.6
    • Similar Issues:

      Description

      Hi All,

      We recently upgraded Jenkins from 2.1 to 2.6. After which Job Status is always success once the execution is completed.

      For example : if a job as 24 test cases and 24 of them also failed, in dashboard the status is still shown as success.

      Please refer to attached screen shot.

      in which test result clearly says that there are 9 failures out of 30 test cases, but still the job status is showing as success. which was not the case with Previous version of Jenkins 2.1.

      Thanks,
      pavan

        Attachments

          Issue Links

            Activity

            Hide
            ppothnis pavan pothnis added a comment -

            we have downgraded Jenkins version to 2.5 still no help.

            Show
            ppothnis pavan pothnis added a comment - we have downgraded Jenkins version to 2.5 still no help.
            Hide
            t_westling Tomas Westling added a comment -

            In which way is this problem related to the build failure analyzer plugin?

            Show
            t_westling Tomas Westling added a comment - In which way is this problem related to the build failure analyzer plugin?
            Hide
            ppothnis pavan pothnis added a comment -

            I was under impression that build failure analyzer is supposed analyze the result and show the status.

            If not, could you please let me know what could be the issue for which Jenkins Shows Job Status as Success, even when many test cases have failed.

            Thanks,
            Pavan

            Show
            ppothnis pavan pothnis added a comment - I was under impression that build failure analyzer is supposed analyze the result and show the status. If not, could you please let me know what could be the issue for which Jenkins Shows Job Status as Success, even when many test cases have failed. Thanks, Pavan
            Hide
            t_westling Tomas Westling added a comment -

            The build failure analyzer plugin is used to quickly tell a user what went wrong with a build when it failed, by matching regular expressions
            with the build log. It doesn't have any power over which builds fail or not.

            Since TestNG tells you that tests failed, I would start there, with the testng-plugin.

            Br Tomas

            Show
            t_westling Tomas Westling added a comment - The build failure analyzer plugin is used to quickly tell a user what went wrong with a build when it failed, by matching regular expressions with the build log. It doesn't have any power over which builds fail or not. Since TestNG tells you that tests failed, I would start there, with the testng-plugin. Br Tomas
            Hide
            ppothnis pavan pothnis added a comment -

            As per Tomas Suggestion, I looked into Publish TestNG plugin configuration (See attached Screenshot).

            The threshold is set to mark build unstable / failure when there is 100 % of test cases fails or gets skipped.

            However, even when we have 11/11 Test cases failed. build status is still shown as Success.

            Thanks,
            Pavan

            Show
            ppothnis pavan pothnis added a comment - As per Tomas Suggestion, I looked into Publish TestNG plugin configuration (See attached Screenshot). The threshold is set to mark build unstable / failure when there is 100 % of test cases fails or gets skipped. However, even when we have 11/11 Test cases failed. build status is still shown as Success. Thanks, Pavan
            Hide
            ppothnis pavan pothnis added a comment -

            Trying to solve this issue with JENKINS-35254.

            Thanks Tom.

            Show
            ppothnis pavan pothnis added a comment - Trying to solve this issue with JENKINS-35254 . Thanks Tom.

              People

              • Assignee:
                t_westling Tomas Westling
                Reporter:
                ppothnis pavan pothnis
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: