Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34822

TestNG plugin has incorrect default threshold values

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: testng-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.2
      TestNG Results Plugin 1.12
    • Similar Issues:

      Description

      After updating the TestNG plugin to 1.12, the new feature from JENKINS-20985 caused our tests to pass regardless of failures/skips in our TestNG results. By default, the threshold is set to 100% if users have not configured the plugin yet.

          * Helps resolve XML configs for versions before 1.11 when these new config options were introduced.
          * See https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility
          * @return resolved object
           */
         protected Object readResolve() {
            if (unstableSkips == null) {
               unstableSkips = unstableOnSkippedTests ? 0 : 100;
            }
            if (failedFails == null) {
               failedFails = 100;
            }
            if (failedSkips == null) {
               failedSkips = 100;
            }
            if (thresholdMode == null) {
               thresholdMode = 2;
            }
            return this;
         }
      

      This means that by default, a test will require MORE THAN 100% skips/failures in order to mark the build as unstable/failed from TestNG results. It would make more sense if the default was set to 0, and the thresholdMode set to 1. This way, a single failure/skip will cause the build to be unstable/fail like in previous versions.

        Attachments

          Issue Links

            Activity

            Hide
            feyat vivian zhang added a comment - - edited

            Nalin Makar I have testng plugin 1.14 installed but found that my build result still shown as successful even though all tests have been skipped.  The threshold mode was set to % and threshold to mark build as unstable for skipped tests were set as 0.

            I found the piece of code that does the checking and found the issue could be caused by the wrong default value.  They are set to 100 which I think should be 0. 

            --------------------------
            testng-plugin-plugin/src/main/java/hudson/plugins/testng/Publisher.java

               //number of skips that will trigger "Unstable"
               private Integer unstableSkips = 100;

               //number of skips that will trigger "Failed"
               private Integer failedSkips = 100;

             

            } else if (results.getSkipCount() > unstableSkips) {
               logger.println(String.format("%d tests were skipped, which exceeded threshold of %d. Marking     build as UNSTABLE",
               results.getSkipCount(), unstableSkips));
               build.setResult(Result.UNSTABLE);

             

            As a result, the unstable skips on Jenkins job has to be > 100% to mark the build as unstable.  But the number will not pass the validate method that makes sure the number is between 1 and 100.

            Can you take a look?  Thanks.

            Show
            feyat vivian zhang added a comment - - edited Nalin Makar I have testng plugin 1.14 installed but found that my build result still shown as successful even though all tests have been skipped.  The threshold mode was set to % and threshold to mark build as unstable for skipped tests were set as 0. I found the piece of code that does the checking and found the issue could be caused by the wrong default value.  They are set to 100 which I think should be 0.  -------------------------- testng-plugin-plugin/src/main/java/hudson/plugins/testng/Publisher.java    //number of skips that will trigger "Unstable"    private Integer unstableSkips = 100;    //number of skips that will trigger "Failed"    private Integer failedSkips = 100;   } else if (results.getSkipCount() > unstableSkips) {    logger.println(String.format("%d tests were skipped, which exceeded threshold of %d. Marking     build as UNSTABLE",    results.getSkipCount(), unstableSkips));    build.setResult(Result.UNSTABLE);   As a result, the unstable skips on Jenkins job has to be > 100% to mark the build as unstable.  But the number will not pass the validate method that makes sure the number is between 1 and 100. Can you take a look?  Thanks.
            Hide
            cwcam Cam Spencer added a comment -

            Hey Chris Everling,

            You're correct that the original behavior wouldn't FAIL a build for TestNG skips/fails, and that the default threshold for FAIL should be 100%. However if you look at the screenshot I posted, the default threshold for UNSTABLE was set to 100% which resulted in some notoriously bad tests to suddenly pass. I just tested the fix that nalin_makar implemented in 1.113, and the default UNSTABLE threshold for TestNG fails was correctly set to 0%. The default TestNG skip for UNSTABLE was still set to 100% which I think is correct with previous behavior (need confirmation).

            Show
            cwcam Cam Spencer added a comment - Hey Chris Everling , You're correct that the original behavior wouldn't FAIL a build for TestNG skips/fails, and that the default threshold for FAIL should be 100%. However if you look at the screenshot I posted, the default threshold for UNSTABLE was set to 100% which resulted in some notoriously bad tests to suddenly pass. I just tested the fix that nalin_makar implemented in 1.113, and the default UNSTABLE threshold for TestNG fails was correctly set to 0%. The default TestNG skip for UNSTABLE was still set to 100% which I think is correct with previous behavior (need confirmation).
            Hide
            nullin Nalin Makar added a comment -

            I think I resolved it. Fixed in v1.13.

            Show
            nullin Nalin Makar added a comment - I think I resolved it. Fixed in v1.13.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: nullin
            Path:
            README
            src/main/java/hudson/plugins/testng/Publisher.java
            http://jenkins-ci.org/commit/testng-plugin-plugin/ed80688950e8b0c2bfbf6399c9956bcb15e9fcc0
            Log:
            Fixed: JENKINS-34822 - TestNG plugin has incorrect default threshold values

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: nullin Path: README src/main/java/hudson/plugins/testng/Publisher.java http://jenkins-ci.org/commit/testng-plugin-plugin/ed80688950e8b0c2bfbf6399c9956bcb15e9fcc0 Log: Fixed: JENKINS-34822 - TestNG plugin has incorrect default threshold values

              People

              • Assignee:
                nullin Nalin Makar
                Reporter:
                cwcam Cam Spencer
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: