Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24076

Incorrect builds used to generate deltas (No deltas for builds fixed after failure where errors were introduced)

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      As an example, consider the following:

      Build 1 - Stable (within CPPCheck threshold)
      Build 2 - Unstable (exceeds CPPCheck threshold)
      Build 3 - Unstable (no change)
      Build 4 - Unstable (no change)

      If you view the CPPCheck results for build 4, it shows the resolved issues since the dawn of time, which is arguably incorrect (see attached image for an example of solved errors showing when the delta is 0), but more crucially, generates its delta for new errors against the previous build, regardless of the build status.

      This creates the problem that, once build 2 drops off the history, you lose the deltas that help you identify the new errors that were introduced, because the deltas don't exist for build 3 or build 4 (they are zero).

      There are one of two solutions I can think of:

      1. Allow the previous build status for delta generation to be specified in the configuration: lastStableBuild, lastSuccessfulBuild

      2. Hard code it to use the lastStableBuild only.

        Attachments

          Activity

          Hide
          iwonbigbro Craig Phillips added a comment -

          I have noticed another "bug" recently. If you set the maximum new error threshold to 1 and you have a build where one new error is added, but at the same time, one is resolve, the delta is zero but the build is flagged as stable. This doesn't seem right.

          Show
          iwonbigbro Craig Phillips added a comment - I have noticed another "bug" recently. If you set the maximum new error threshold to 1 and you have a build where one new error is added, but at the same time, one is resolve, the delta is zero but the build is flagged as stable. This doesn't seem right.
          Hide
          iwonbigbro Craig Phillips added a comment -

          My previous comment with regards to the bug. This is actually a side effect of the fact the plugin currently compares against the last build instead of the last stable build. If you get a stable build followed by a build that resolves one error and introduces one error, then this build is marked as unstable. However, if there is no change in the following build, the build status for the last build becomes stable again, because the number of new errors is zero, even though there has been one new error introduced since the last time the build was stable. This really does indicate that the comparison of new errors between each build is not right and that my original statement still stands, that the builds should always compare the current results against the last stable build. The last stable build is after all, the benchmark for where the cppcheck results need to be. If new errors have been introduced since then, then the build can't possibly be considered stable - ever.

          Show
          iwonbigbro Craig Phillips added a comment - My previous comment with regards to the bug. This is actually a side effect of the fact the plugin currently compares against the last build instead of the last stable build. If you get a stable build followed by a build that resolves one error and introduces one error, then this build is marked as unstable. However, if there is no change in the following build, the build status for the last build becomes stable again, because the number of new errors is zero, even though there has been one new error introduced since the last time the build was stable. This really does indicate that the comparison of new errors between each build is not right and that my original statement still stands, that the builds should always compare the current results against the last stable build. The last stable build is after all, the benchmark for where the cppcheck results need to be. If new errors have been introduced since then, then the build can't possibly be considered stable - ever.
          Hide
          iwonbigbro Craig Phillips added a comment -

          This fix maintains backwards compatibility with the current strategy (which also remains the default). But it adds the option to change the behaviour such that it compares the current report against the last "stable" report.

          https://github.com/iwonbigbro/cppcheck-plugin/commit/d639413bf2b06c47ffa14cf3e2fe8c70073faa76

          Show
          iwonbigbro Craig Phillips added a comment - This fix maintains backwards compatibility with the current strategy (which also remains the default). But it adds the option to change the behaviour such that it compares the current report against the last "stable" report. https://github.com/iwonbigbro/cppcheck-plugin/commit/d639413bf2b06c47ffa14cf3e2fe8c70073faa76
          Hide
          mixalturek Michal Turek added a comment -

          Please see comments in your pull request, they should be solved before merge.

          https://github.com/jenkinsci/cppcheck-plugin/pull/24

          Show
          mixalturek Michal Turek added a comment - Please see comments in your pull request, they should be solved before merge. https://github.com/jenkinsci/cppcheck-plugin/pull/24
          Hide
          douglas_lee Douglas Lee added a comment -

          Is there any update on this issue? I'd really love to see this feature added!

          Show
          douglas_lee Douglas Lee added a comment - Is there any update on this issue? I'd really love to see this feature added!

            People

            • Assignee:
              iwonbigbro Craig Phillips
              Reporter:
              iwonbigbro Craig Phillips
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: