Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34822

TestNG plugin has incorrect default threshold values

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: testng-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.2
      TestNG Results Plugin 1.12
    • Similar Issues:

      Description

      After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

          * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
          * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
          * @return resolved object
           */
         protected Object readResolve() {
            if (unstableSkips == null) {
               unstableSkips = unstableOnSkippedTests ? 0 : 100;
            }
            if (failedFails == null) {
               failedFails = 100;
            }
            if (failedSkips == null) {
               failedSkips = 100;
            }
            if (thresholdMode == null) {
               thresholdMode = 2;
            }
            return this;
         }
      

      This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

        Attachments

          Issue Links

            Activity

            cwcam Cam Spencer created issue -
            cwcam Cam Spencer made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-20985 [ JENKINS-20985 ]
            cwcam Cam Spencer made changes -
            Description After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not set this up yet.


            {code:java}
                * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
                * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
                * @return resolved object
                 */
               protected Object readResolve() {
                  if (unstableSkips == null) {
                     unstableSkips = unstableOnSkippedTests ? 0 : 100;
                  }
                  if (failedFails == null) {
                     failedFails = 100;
                  }
                  if (failedSkips == null) {
                     failedSkips = 100;
                  }
                  if (thresholdMode == null) {
                     thresholdMode = 2;
                  }
                  return this;
               }
            {code}

            This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1% to be more inline to what previous versions would do.
            After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.


            {code:java}
                * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
                * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
                * @return resolved object
                 */
               protected Object readResolve() {
                  if (unstableSkips == null) {
                     unstableSkips = unstableOnSkippedTests ? 0 : 100;
                  }
                  if (failedFails == null) {
                     failedFails = 100;
                  }
                  if (failedSkips == null) {
                     failedSkips = 100;
                  }
                  if (thresholdMode == null) {
                     thresholdMode = 2;
                  }
                  return this;
               }
            {code}

            This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1% to be more inline to what previous versions would do.
            cwcam Cam Spencer made changes -
            Attachment TestNG Plugin.jpg [ 32714 ]
            nullin Nalin Makar made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            cwcam Cam Spencer made changes -
            Description After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.


            {code:java}
                * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
                * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
                * @return resolved object
                 */
               protected Object readResolve() {
                  if (unstableSkips == null) {
                     unstableSkips = unstableOnSkippedTests ? 0 : 100;
                  }
                  if (failedFails == null) {
                     failedFails = 100;
                  }
                  if (failedSkips == null) {
                     failedSkips = 100;
                  }
                  if (thresholdMode == null) {
                     thresholdMode = 2;
                  }
                  return this;
               }
            {code}

            This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1% to be more inline to what previous versions would do.
            After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.


            {code:java}
                * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
                * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
                * @return resolved object
                 */
               protected Object readResolve() {
                  if (unstableSkips == null) {
                     unstableSkips = unstableOnSkippedTests ? 0 : 100;
                  }
                  if (failedFails == null) {
                     failedFails = 100;
                  }
                  if (failedSkips == null) {
                     failedSkips = 100;
                  }
                  if (thresholdMode == null) {
                     thresholdMode = 2;
                  }
                  return this;
               }
            {code}

            This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.
            cwcam Cam Spencer made changes -
            Description After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.


            {code:java}
                * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
                * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
                * @return resolved object
                 */
               protected Object readResolve() {
                  if (unstableSkips == null) {
                     unstableSkips = unstableOnSkippedTests ? 0 : 100;
                  }
                  if (failedFails == null) {
                     failedFails = 100;
                  }
                  if (failedSkips == null) {
                     failedSkips = 100;
                  }
                  if (thresholdMode == null) {
                     thresholdMode = 2;
                  }
                  return this;
               }
            {code}

            This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 1, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.
            After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.


            {code:java}
                * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
                * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
                * @return resolved object
                 */
               protected Object readResolve() {
                  if (unstableSkips == null) {
                     unstableSkips = unstableOnSkippedTests ? 0 : 100;
                  }
                  if (failedFails == null) {
                     failedFails = 100;
                  }
                  if (failedSkips == null) {
                     failedSkips = 100;
                  }
                  if (thresholdMode == null) {
                     thresholdMode = 2;
                  }
                  return this;
               }
            {code}

            This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.
            nullin Nalin Makar made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            cwcam Cam Spencer made changes -
            Link This issue is related to JENKINS-34711 [ JENKINS-34711 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 171013 ] JNJira + In-Review [ 199007 ]
            feyat vivian zhang made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            benapgar Ben Apgar made changes -
            Link This issue relates to JENKINS-42819 [ JENKINS-42819 ]

              People

              • Assignee:
                nullin Nalin Makar
                Reporter:
                cwcam Cam Spencer
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: