Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22018

Support for ratcheting of minimum thresholds in Jacoco coverage

    Details

    • Similar Issues:

      Description

      The addition of an option to "ratchet" or update the minimum thresholds for build failure would be very useful. This request is inspired by and heavily related to JENKINS-8326 for the Cobertura coverage plugin.

      Ideally we'd be able to set the publisher to automatically update the minimum thresholds to higher values when a project's observed value is higher than a previously set minimum. This enables a project to help enforce that code coverage cannot decrease over time as a code quality standard.

        Attachments

          Issue Links

            Activity

            Hide
            seadub Chris Williams added a comment -

            Because I've had to work around this myself in the meantime, I use the following groovy postbuild script to achieve this:

            // Grab the API for Jacoco
            def b = manager.build;
            
            def action = b.getActions().find { it.getUrlName() == "jacoco" }
            if (action == null) {
            	// no Jacoco coverage, so nothing to do!
            	manager.listener.logger.println "Unable to get Jacoco Build Action on build, so bailing out"
            	return;
            }
            
            def publishers = b.getProject().publishersList;
            def jacocoPublisher = publishers.find { it.getDescriptor().getId() == "hudson.plugins.jacoco.JacocoPublisher" }
            
            // Collect the current "observed" values for coverage
            def line = action.getLineCoverage();
            def clazz = action.getClassCoverage();
            def method = action.getMethodCoverage();
            def instruction = action.getInstructionCoverage();
            def branch = action.getBranchCoverage();
            //def complexity = action.getComplexityScore();
            
            // Now let's set the new threshold minimums to last value!
            // Only set these values if the observed percent is _higher_ than the existing minimum threshold!
            def thresholds = action.getThresholds();
            
            // Classes
            int percent = clazz.getPercentage();
            manager.listener.logger.println "Observed class coverage: ${percent}%"
            if (percent > thresholds.getMinClass()) {
            	manager.listener.logger.println "Increasing minimum threshold to observed value for class coverage"
            	thresholds.setMinClass(percent);
            }
            // methods
            percent = method.getPercentage();
            manager.listener.logger.println "Observed methods coverage: ${percent}%"
            if (percent > thresholds.getMinMethod()) {
            	manager.listener.logger.println "Increasing minimum threshold to observed value for methods coverage"
            	thresholds.setMinMethod(percent);
            }
            // Branches
            percent = branch.getPercentage();
            manager.listener.logger.println "Observed branches coverage: ${percent}%"
            if (percent > thresholds.getMinBranch()) {
            	manager.listener.logger.println "Increasing minimum threshold to observed value for branches coverage"
            	thresholds.setMinBranch(percent);
            }
            // Lines
            percent = line.getPercentage();
            manager.listener.logger.println "Observed lines coverage: ${percent}%"
            if (percent > thresholds.getMinLine()) {
            	manager.listener.logger.println "Increasing minimum threshold to observed value for lines coverage"
            	thresholds.setMinLine(percent);
            }
            // Instructions
            percent = instruction.getPercentage();
            manager.listener.logger.println "Observed instruction coverage: ${percent}%"
            if (percent > thresholds.getMinInstruction()) {
            	manager.listener.logger.println "Increasing minimum threshold to observed value for instruction coverage"
            	thresholds.setMinInstruction(percent);
            }
            // Complexity
            //percent = complexity.getPercentage();
            //if (percent > thresholds.getMinComplexity()) {
            //	thresholds.setMinComplexity(percent);
            //}
            
            // We need to replace the publisher with a new instance that has updated minimums
            // HACK to work around bug in plugin where we don't have access to plugin classes.
            def c = Class.forName("hudson.plugins.jacoco.JacocoPublisher", true, manager.hudson.getPluginManager().uberClassLoader)
            def constructor = c.getConstructor( [ String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, boolean ] as Class[] )
            // make a new version of the coverage publisher with updated minimums
            def newPub = constructor.newInstance(jacocoPublisher.getExecPattern(), jacocoPublisher.getClassPattern(), jacocoPublisher.getSourcePattern(),
             jacocoPublisher.getInclusionPattern(), jacocoPublisher.getExclusionPattern(), jacocoPublisher.getMaximumInstructionCoverage(),
             jacocoPublisher.getMaximumBranchCoverage(), jacocoPublisher.getMaximumComplexityCoverage(), jacocoPublisher.getMaximumLineCoverage(),
             jacocoPublisher.getMaximumMethodCoverage(), jacocoPublisher.getMaximumClassCoverage(), Integer.toString(thresholds.getMinInstruction()), Integer.toString(thresholds.getMinBranch()),
             Integer.toString(thresholds.getMinComplexity()), Integer.toString(thresholds.getMinLine()), Integer.toString(thresholds.getMinMethod()), Integer.toString(thresholds.getMinClass()),
             jacocoPublisher.isChangeBuildStatus())
            // Now remove the old version and replace with the new publisher with updated minimums
            publishers.replace(jacocoPublisher, newPub); // remove old and replace with new
            b.getProject().save(); // persist the change
            
            Show
            seadub Chris Williams added a comment - Because I've had to work around this myself in the meantime, I use the following groovy postbuild script to achieve this: // Grab the API for Jacoco def b = manager.build; def action = b.getActions().find { it.getUrlName() == "jacoco" } if (action == null ) { // no Jacoco coverage, so nothing to do ! manager.listener.logger.println "Unable to get Jacoco Build Action on build, so bailing out" return ; } def publishers = b.getProject().publishersList; def jacocoPublisher = publishers.find { it.getDescriptor().getId() == "hudson.plugins.jacoco.JacocoPublisher" } // Collect the current "observed" values for coverage def line = action.getLineCoverage(); def clazz = action.getClassCoverage(); def method = action.getMethodCoverage(); def instruction = action.getInstructionCoverage(); def branch = action.getBranchCoverage(); // def complexity = action.getComplexityScore(); // Now let's set the new threshold minimums to last value! // Only set these values if the observed percent is _higher_ than the existing minimum threshold! def thresholds = action.getThresholds(); // Classes int percent = clazz.getPercentage(); manager.listener.logger.println "Observed class coverage: ${percent}%" if (percent > thresholds.getMinClass()) { manager.listener.logger.println "Increasing minimum threshold to observed value for class coverage" thresholds.setMinClass(percent); } // methods percent = method.getPercentage(); manager.listener.logger.println "Observed methods coverage: ${percent}%" if (percent > thresholds.getMinMethod()) { manager.listener.logger.println "Increasing minimum threshold to observed value for methods coverage" thresholds.setMinMethod(percent); } // Branches percent = branch.getPercentage(); manager.listener.logger.println "Observed branches coverage: ${percent}%" if (percent > thresholds.getMinBranch()) { manager.listener.logger.println "Increasing minimum threshold to observed value for branches coverage" thresholds.setMinBranch(percent); } // Lines percent = line.getPercentage(); manager.listener.logger.println "Observed lines coverage: ${percent}%" if (percent > thresholds.getMinLine()) { manager.listener.logger.println "Increasing minimum threshold to observed value for lines coverage" thresholds.setMinLine(percent); } // Instructions percent = instruction.getPercentage(); manager.listener.logger.println "Observed instruction coverage: ${percent}%" if (percent > thresholds.getMinInstruction()) { manager.listener.logger.println "Increasing minimum threshold to observed value for instruction coverage" thresholds.setMinInstruction(percent); } // Complexity //percent = complexity.getPercentage(); // if (percent > thresholds.getMinComplexity()) { // thresholds.setMinComplexity(percent); //} // We need to replace the publisher with a new instance that has updated minimums // HACK to work around bug in plugin where we don't have access to plugin classes. def c = Class .forName( "hudson.plugins.jacoco.JacocoPublisher" , true , manager.hudson.getPluginManager().uberClassLoader) def constructor = c.getConstructor( [ String , String , String , String , String , String , String , String , String , String , String , String , String , String , String , String , String , boolean ] as Class [] ) // make a new version of the coverage publisher with updated minimums def newPub = constructor.newInstance(jacocoPublisher.getExecPattern(), jacocoPublisher.getClassPattern(), jacocoPublisher.getSourcePattern(), jacocoPublisher.getInclusionPattern(), jacocoPublisher.getExclusionPattern(), jacocoPublisher.getMaximumInstructionCoverage(), jacocoPublisher.getMaximumBranchCoverage(), jacocoPublisher.getMaximumComplexityCoverage(), jacocoPublisher.getMaximumLineCoverage(), jacocoPublisher.getMaximumMethodCoverage(), jacocoPublisher.getMaximumClassCoverage(), Integer .toString(thresholds.getMinInstruction()), Integer .toString(thresholds.getMinBranch()), Integer .toString(thresholds.getMinComplexity()), Integer .toString(thresholds.getMinLine()), Integer .toString(thresholds.getMinMethod()), Integer .toString(thresholds.getMinClass()), jacocoPublisher.isChangeBuildStatus()) // Now remove the old version and replace with the new publisher with updated minimums publishers.replace(jacocoPublisher, newPub); // remove old and replace with new b.getProject().save(); // persist the change
            Hide
            fluffy88 Seán Dunne added a comment -

            I'd like to add my voice to this issue.
            IMO being able to enforce that unit test coverage does not regress is a key feature of a coverage plugin.

            Would love to see it implemented!

            Show
            fluffy88 Seán Dunne added a comment - I'd like to add my voice to this issue. IMO being able to enforce that unit test coverage does not regress is a key feature of a coverage plugin. Would love to see it implemented!

              People

              • Assignee:
                ognjenb Ognjen Bubalo
                Reporter:
                seadub Chris Williams
              • Votes:
                5 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: