Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46046

Publisher with Standard Mode / Relative Threshold does not work for more than one sourceDataFiles

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: performance-plugin
    • Labels:
      None
    • Environment:
      Jenkins v2.70
      Performance Plugin v3.3-SNAPSHOT
      Windows Server 2012
      Oracle Java 1.8.0_66
    • Similar Issues:

      Description

      How to reproduce:

      In either a Freestyle or Pipeline job,

      1. Specify 2 sourceDataFiles, say JMeter csv files: a.csv;b.csv
      2. Run Publisher (this will create serialized versions of the parsed files).
      3. Run once more for a relative comparison.
      4. Run once again but with files in different order: b.csv;a.csv

      This will produce output similar to the following.

      One build will have a relative comparison for just one file's samples, whereas the other build will have no comparison (as it tries to compare a.csv with b.csv), as below.

      [Pipeline] step
       Performance: Recording JMeterCsv reports 'b.csv'
       Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\30\performance-reports\JMeterCSV\b.csv'.
       Performance: Recording JMeterCsv reports 'a.csv'
       Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\30\performance-reports\JMeterCSV\a.csv'.
       Performance: Percentage of relative difference outside -10.0 to +10.0 % sets the build as failure
       Performance: Percentage of relative difference outside -5.0 to +5.0 % sets the build as unstable
       Performance: Recording JMeterCsv reports 'b.csv'
       Performance: Recording JMeterCsv reports 'a.csv'
      Comparison build no. - 29 and 30 using 90 Percentile response time
      PrevBuildURI CurrentBuildURI PrevBuildURI90% CurrentBuildURI90% RelativeDiff RelativeDiffPercentage 
       a a 900 900 0.0 0.0
       [Pipeline] }
       [Pipeline] // node
       [Pipeline] End of Pipeline
       Finished: SUCCESS
      

       

      [Pipeline] step
      Performance: Recording JMeterCsv reports 'a.csv'
      Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\29\performance-reports\JMeterCSV\a.csv'.
      Performance: Recording JMeterCsv reports 'b.csv'
      Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\29\performance-reports\JMeterCSV\b.csv'.
      Performance: Percentage of relative difference outside -10.0 to +10.0 % sets the build as failure
      Performance: Percentage of relative difference outside -5.0 to +5.0 % sets the build as unstable
      Performance: Recording JMeterCsv reports 'a.csv'
      Performance: Recording JMeterCsv reports 'b.csv'
      
      Comparison build no. - 28 and 29 using 90 Percentile response time
      
      
      PrevBuildURI	CurrentBuildURI		PrevBuildURI90%		CurrentBuildURI90%	RelativeDiff	RelativeDiffPercentage 
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: SUCCESS

       

      Sample Jenkinsfile:

      node {
          ['a','b'].each { 
              sh "echo timeStamp,elapsed,label,responseCode,success > ${it}.csv"
              10.times { i -> sh "echo ${1500000000000+i*1000},${i*100},$it,200,true >> ${it}.csv" }
          }
      
          perfReport([
              sourceDataFiles: 'a.csv;b.csv',
              modeEvaluation: false,
              modeOfThreshold: true, 
              failBuildIfNoResultFile: true, 
              relativeUnstableThresholdNegative: 5,
              relativeUnstableThresholdPositive: 5,
              relativeFailedThresholdNegative: 10,
              relativeFailedThresholdPositive: 10,
              compareBuildPrevious: true,
              configType: 'PRT'
          ])
      }

       

        Attachments

          Activity

          Hide
          tilln Till Neunast added a comment -

          Cause seems to be that PerformancePublisher.getBuildUriReports() returns just the samples from one file (the first it finds, as the order depends on different methods' return lists):

              private List<UriReport> getBuildUriReports(Run<?, ?> build, FilePath workspace, TaskListener listener,
                                                         List<PerformanceReportParser> parsers, boolean locatePerformanceReports)
                      throws IOException, InterruptedException {
          
                  if (locatePerformanceReports) {
                      Collection<PerformanceReport> performanceReports = locatePerformanceReports(build, workspace, listener, parsers);
                      if (performanceReports == null) {
                          return null;
                      }
                      for (PerformanceReport r : performanceReports) {
                          // URI list is the list of labels in the current JMeter results
                          // file
                          return r.getUriListOrdered();
                      }
                  } else {
                      for (PerformanceReportParser parser : parsers) {
                          // add the report to the build object.
                          List<File> localReports = getExistingReports(build, listener.getLogger(), parser.getDescriptor().getDisplayName());
                          Collection<PerformanceReport> parsedReports = parser.parse(build, localReports, listener);
          
                          for (PerformanceReport r : parsedReports) {
                              // uri list is the list of labels in the previous jmeter results
                              // file
                              return r.getUriListOrdered();
                          }
          
                      }
                  }
                  return Collections.emptyList();
              }
          Show
          tilln Till Neunast added a comment - Cause seems to be that PerformancePublisher.getBuildUriReports() returns just the samples from one file (the first it finds, as the order depends on different methods' return lists):     private List<UriReport> getBuildUriReports(Run<?, ?> build, FilePath workspace, TaskListener listener,                                                List<PerformanceReportParser> parsers, boolean locatePerformanceReports)             throws IOException, InterruptedException {         if (locatePerformanceReports) {             Collection<PerformanceReport> performanceReports = locatePerformanceReports(build, workspace, listener, parsers);             if (performanceReports == null ) {                 return null ;             }             for (PerformanceReport r : performanceReports) {                 // URI list is the list of labels in the current JMeter results                 // file                 return r.getUriListOrdered();             }         } else {             for (PerformanceReportParser parser : parsers) {                 // add the report to the build object.                 List<File> localReports = getExistingReports(build, listener.getLogger(), parser.getDescriptor().getDisplayName());                 Collection<PerformanceReport> parsedReports = parser.parse(build, localReports, listener);                 for (PerformanceReport r : parsedReports) {                     // uri list is the list of labels in the previous jmeter results                     // file                     return r.getUriListOrdered();                 }             }         }         return Collections.emptyList();     }
          Hide
          tilln Till Neunast added a comment -

          Return just one report also masks another issue:

          If 2 or more sourceDataFiles for the same parser are specified, then the call to getExistingReports in the loop for (PerformanceReportParser parser : parsers) will find the same results reports multiple times.
          With above example, there will be 2 JMeterCSV parsers (a.csv, b.csv), i.e. 2 loop iterations, and each will find 2 existing reports (a.csv, b.csv), resulting in doubling up the URI comparisons.

          Show
          tilln Till Neunast added a comment - Return just one report also masks another issue: If 2 or more sourceDataFiles for the same parser are specified, then the call to getExistingReports in the loop for (PerformanceReportParser parser : parsers) will find the same results reports multiple times. With above example, there will be 2 JMeterCSV parsers (a.csv, b.csv), i.e. 2 loop iterations, and each will find 2 existing reports (a.csv, b.csv), resulting in doubling up the URI comparisons.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Till Neunast
          Path:
          src/main/java/hudson/plugins/performance/PerformancePublisher.java
          http://jenkins-ci.org/commit/performance-plugin/5546f9d63b7fd68447ac8e6bab5a254b3461bcff
          Log:
          Proper handling of existing reports for relative comparisons (JENKINS-46046) (#127)

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Till Neunast Path: src/main/java/hudson/plugins/performance/PerformancePublisher.java http://jenkins-ci.org/commit/performance-plugin/5546f9d63b7fd68447ac8e6bab5a254b3461bcff Log: Proper handling of existing reports for relative comparisons ( JENKINS-46046 ) (#127)
          Hide
          artem_fedorov Artem Fedorov added a comment -

          Fix will be available in version 3.3

          Show
          artem_fedorov Artem Fedorov added a comment - Fix will be available in version 3.3

            People

            • Assignee:
              undera Andrey Pokhilko
              Reporter:
              tilln Till Neunast
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: