Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58184

Jacoco delta threshold failures if coverage *increases* over threshold

    Details

    • Similar Issues:

      Description

      Jacoco delta threshold got failure if some coverage metrics decreases less than delta threshold but some increases over delta threshold.

      Problematic code is 

      JacocoPublisher.java

      public Result checkBuildOverBuildResult(Run<?,?> run, PrintStream logger){
      
              JacocoDeltaCoverageResultSummary deltaCoverageResultSummary = JacocoDeltaCoverageResultSummary.build(run);
              logger.println("[JaCoCo plugin] Delta coverage: class: " + deltaCoverageResultSummary.getClassCoverage()
                      + ", method: " + deltaCoverageResultSummary.getMethodCoverage()
                      + ", line: " + deltaCoverageResultSummary.getLineCoverage()
                      + ", branch: " + deltaCoverageResultSummary.getBranchCoverage()
                      + ", instruction: " + deltaCoverageResultSummary.getInstructionCoverage()
                      + ", complexity: " + deltaCoverageResultSummary.getComplexityCoverage());
      
              if(Math.abs(deltaCoverageResultSummary.getInstructionCoverage()) <= deltaHealthReport.getDeltaInstruction() &&
                      Math.abs(deltaCoverageResultSummary.getBranchCoverage()) <= deltaHealthReport.getDeltaBranch() &&
                      Math.abs(deltaCoverageResultSummary.getComplexityCoverage()) <= deltaHealthReport.getDeltaComplexity() &&
                      Math.abs(deltaCoverageResultSummary.getLineCoverage()) <= deltaHealthReport.getDeltaLine() &&
                      Math.abs(deltaCoverageResultSummary.getMethodCoverage()) <= deltaHealthReport.getDeltaMethod() &&
                      Math.abs(deltaCoverageResultSummary.getClassCoverage()) <= deltaHealthReport.getDeltaClass())
                  return Result.SUCCESS;
              else if(deltaCoverageResultSummary.isCoverageBetterThanPrevious())
                  return Result.SUCCESS;
              else
                  return Result.FAILURE;
          }
      

       The above code uses abs. This means that check not make any difference of increase or decrease. If not all metrics are increased then isCoverageBetterThanPrevious gives false. so Despite all decreases are below delta threshold  build fails if some of increase more than delta. This is not logical because coverage increase should not be measured against delta at all.

       

        Attachments

          Activity

          Hide
          rainoko Raino Kolk added a comment -

          Why not check before abs that coverage delta is negative? It should be ok to ignore positive delta from that check.

          Show
          rainoko Raino Kolk added a comment - Why not check before abs that coverage delta is negative? It should be ok to ignore positive delta from that check.
          Hide
          centic centic added a comment -

          this was contributed, so I am not sure what the original intent was on these checks, however I also would not expect a failure when coverage actually increases. Can you provide a PR which simplifies/improves this handling?

          Show
          centic centic added a comment - this was contributed, so I am not sure what the original intent was on these checks, however I also would not expect a failure when coverage actually increases. Can you provide a PR which simplifies/improves this handling?

            People

            • Assignee:
              ognjenb Ognjen Bubalo
              Reporter:
              rainoko Raino Kolk
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: