Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20985

TestNG plugin should has a possibility to use thresholds for failed tests

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: testng-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      Current version of TestNG plugin does not have a possibility to use threshold values for failed and skipped tests. If some build has at least one failed test, then this build will be marked as unstable.
      Maybe it would be better to have some mechanism to mark build as successfull instead of unstable with using a threshold value for failed tests.

        Attachments

          Issue Links

            Activity

            Hide
            cwcam Cam Spencer added a comment -

            Hey nalin_makar, I don't think the default threshold value should be 100%. We updated the plugin without noticing this feature, and this was causing tests that failed to show as stable.

            Show
            cwcam Cam Spencer added a comment - Hey nalin_makar , I don't think the default threshold value should be 100%. We updated the plugin without noticing this feature, and this was causing tests that failed to show as stable.
            Hide
            nullin Nalin Makar added a comment -

            Cam Spencer, can you please file a separate bug? And, it'll be great if you can give specific information about the properties you are referring to. Thanks!

            Show
            nullin Nalin Makar added a comment - Cam Spencer , can you please file a separate bug? And, it'll be great if you can give specific information about the properties you are referring to. Thanks!
            Hide
            chriseverling Chris Everling added a comment -

            Cam Spencer, nalin_makar, I don't think that is a bug. The default behavior before the update was to mark the build as UNSTABLE whenever there were test failures. The default failure threshold to FAIL the build is 100%, but the default failure threshold to mark the build as UNSTABLE is 0. The default behavior should not have changed with the update.

            Show
            chriseverling Chris Everling added a comment - Cam Spencer , nalin_makar , I don't think that is a bug. The default behavior before the update was to mark the build as UNSTABLE whenever there were test failures. The default failure threshold to FAIL the build is 100%, but the default failure threshold to mark the build as UNSTABLE is 0. The default behavior should not have changed with the update.
            Hide
            jatheisenii21 James Theisen added a comment -

            The default is fixed, but now I am stuck in a situation where all my jobs are set to 100 for failed tests to be considered unstable. Ungrading to 1.13 fixes the default for new jobs. But how about the rest of us that upgraded to 1.12 and ALL our jobs are stuck with 100% for this value? I guess I am stuck with having the server admin help me change all the job xmls directly (since I dont have access). Or maybe you want to come up with a config slicer or something for this. Any other suggestions on how I can correct this now since your initial release was very poorly configured and tested?

            Show
            jatheisenii21 James Theisen added a comment - The default is fixed, but now I am stuck in a situation where all my jobs are set to 100 for failed tests to be considered unstable. Ungrading to 1.13 fixes the default for new jobs. But how about the rest of us that upgraded to 1.12 and ALL our jobs are stuck with 100% for this value? I guess I am stuck with having the server admin help me change all the job xmls directly (since I dont have access). Or maybe you want to come up with a config slicer or something for this. Any other suggestions on how I can correct this now since your initial release was very poorly configured and tested?
            Hide
            nullin Nalin Makar added a comment -

            original issue reported here is fixed. Editing config xml is the way to go (sorry for the trouble).

            Show
            nullin Nalin Makar added a comment - original issue reported here is fixed. Editing config xml is the way to go (sorry for the trouble).

              People

              • Assignee:
                gavingray Gavin Gray
                Reporter:
                ndriuchatyi Nikolay Driuchatyi
              • Votes:
                3 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: