Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44937

Performance Plugin - Builds failed inappropriately?

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: performance-plugin
    • Labels:
      None
    • Environment:
      jenkins 2.65, performance plugin 3.1
    • Similar Issues:

      Description

      Since at least version 3.1, relativeFailedThreshold and relativeUnstableThreshold appear to be failing builds or making them unstable if the number of failed builds exceeds or varies by the value set, rather than if the response time varied by those amounts.

      07:54:24 Performance: Percentage of errors greater or equal than 0% sets the build as unstable
      07:54:24 Performance: Percentage of errors greater or equal than 0% sets the build as failure
      07:54:24 Performance: File abcJunit.xml reported 0.0% of errors [SUCCESS]. Build status is: UNSTABLE
      07:54:24 Performance: File xyzJunit.xml reported 4.0% of errors [FAILURE]. Build status is: FAILURE

      Our servers running V2.0 don't behave this way, obviously, and the same builds of the same code don't fail there.

      14:02:30 Performance: Percentage of errors greater or equal than 0% sets the build as unstable
      14:02:30 Performance: Percentage of errors greater or equal than 0% sets the build as failure
      

      I've never understood what this output means, either, and the value is always 0%, which seems improbable but at least it doesn't fail builds.

      Pipeline settings:

      	performanceReport compareBuildPrevious: true,
      			configType: 'ART',
      			modeOfThreshold: false,
      			modePerformancePerTestCase: true,
      			parsers: [[$class: 'JUnitParser', glob: '**/*Junit.xml, **/TEST-*.xml']],
      			relativeFailedThresholdNegative: 0.02,
      			relativeFailedThresholdPositive: 0.02,
      			relativeUnstableThresholdNegative: 0.01,
      			relativeUnstableThresholdPositive: 0.01
      

      We really need some proper, clear explanations and examples of how this plugin is supposed to work as the docs are still not clear to me, today.

        Attachments

          Activity

          Hide
          artem_fedorov Artem Fedorov added a comment -

          To use relative threshold in compare you need to set:
          modeOfThreshold: true
          and you can see the next output: 

          Performance: Recording JMeterCsv reports 'result.csv'
          Performance: Parsing JMeter report file '/var/lib/jenkins/jobs/COMPARE_GUI/builds/3/performance-reports/JMeterCSV/result.csv'.
          Performance: Percentage of relative difference outside -4.0 to +6.0 % sets the build as failure
          Performance: Percentage of relative difference outside -3.0 to +5.0 % sets the build as unstable
          Performance: Recording JMeterCsv reports 'result.csv'
          
          Comparison build no. - 1 and 3 using Average response time
          
          
          PrevBuildURI	CurrentBuildURI		PrevBuildURIAvg		CurrentBuildURIAvg	RelativeDiff	RelativeDiffPercentage 
          HTTP Request	HTTP Request		207			147			-60.0		-28.0
          
          The label "HTTP Request" caused the build to fail
          Build step 'Publish Performance test result report' changed build result to FAILURE
          Finished: FAILURE
          

          But in 3.1 version we have little problem with  relativeUnstableThresholdNegative, that fixed in this issue: https://issues.jenkins-ci.org/browse/JENKINS-44754 

          Show
          artem_fedorov Artem Fedorov added a comment - To use relative threshold in compare you need to set: modeOfThreshold: true and you can see the next output:  Performance: Recording JMeterCsv reports 'result.csv' Performance: Parsing JMeter report file '/ var /lib/jenkins/jobs/COMPARE_GUI/builds/3/performance-reports/JMeterCSV/result.csv' . Performance: Percentage of relative difference outside -4.0 to +6.0 % sets the build as failure Performance: Percentage of relative difference outside -3.0 to +5.0 % sets the build as unstable Performance: Recording JMeterCsv reports 'result.csv' Comparison build no. - 1 and 3 using Average response time PrevBuildURI CurrentBuildURI PrevBuildURIAvg CurrentBuildURIAvg RelativeDiff RelativeDiffPercentage HTTP Request HTTP Request 207 147 -60.0 -28.0 The label "HTTP Request" caused the build to fail Build step 'Publish Performance test result report' changed build result to FAILURE Finished: FAILURE But in 3.1 version we have little problem with  relativeUnstableThresholdNegative, that fixed in this issue: https://issues.jenkins-ci.org/browse/JENKINS-44754  
          Hide
          ghs1 g hs1 added a comment -

          To use relative threshold in compare you need to set:
          modeOfThreshold: true

          Thanks, that's useful. Please document fully the snippet builder UI so we can all understand it and get the best from it. The current state is very poor, some property names are dubious and the plugin shouldn't have been released undocumented. The expert mode should be hidden when the the standard mode is selected and vice versa  Don't feel too bad, though, in general the docs for every plugin I've seen are terrible.

          Show
          ghs1 g hs1 added a comment - To use relative threshold in compare you need to set: modeOfThreshold: true Thanks, that's useful. Please document fully the snippet builder UI so we can all understand it and get the best from it. The current state is very poor, some property names are dubious and the plugin shouldn't have been released undocumented. The expert mode should be hidden when the the standard mode is selected and vice versa  Don't feel too bad, though, in general the docs for every plugin I've seen are terrible.
          Hide
          ghs1 g hs1 added a comment -

          How is this fixed?? The discussion above has nothing to do with the problem.

          Show
          ghs1 g hs1 added a comment - How is this fixed?? The discussion above has nothing to do with the problem.
          Hide
          ghs1 g hs1 added a comment -

          Whatever the problem was it seems to have been fixed in 3.1 or 3.2.

          Show
          ghs1 g hs1 added a comment - Whatever the problem was it seems to have been fixed in 3.1 or 3.2.
          Hide
          artem_fedorov Artem Fedorov added a comment -

          Can you specify what is the problem now?

          Show
          artem_fedorov Artem Fedorov added a comment - Can you specify what is the problem now?

            People

            • Assignee:
              artem_fedorov Artem Fedorov
              Reporter:
              ghs1 g hs1
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: