-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Jenkins v2.70
Performance Plugin v3.3-SNAPSHOT
Windows Server 2012
Oracle Java 1.8.0_66
How to reproduce:
In either a Freestyle or Pipeline job,
- Specify 2 sourceDataFiles, say JMeter csv files: a.csv;b.csv
- Run Publisher (this will create serialized versions of the parsed files).
- Run once more for a relative comparison.
- Run once again but with files in different order: b.csv;a.csv
This will produce output similar to the following.
One build will have a relative comparison for just one file's samples, whereas the other build will have no comparison (as it tries to compare a.csv with b.csv), as below.
[Pipeline] step Performance: Recording JMeterCsv reports 'b.csv' Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\30\performance-reports\JMeterCSV\b.csv'. Performance: Recording JMeterCsv reports 'a.csv' Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\30\performance-reports\JMeterCSV\a.csv'. Performance: Percentage of relative difference outside -10.0 to +10.0 % sets the build as failure Performance: Percentage of relative difference outside -5.0 to +5.0 % sets the build as unstable Performance: Recording JMeterCsv reports 'b.csv' Performance: Recording JMeterCsv reports 'a.csv' Comparison build no. - 29 and 30 using 90 Percentile response time PrevBuildURI CurrentBuildURI PrevBuildURI90% CurrentBuildURI90% RelativeDiff RelativeDiffPercentage a a 900 900 0.0 0.0 [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
[Pipeline] step Performance: Recording JMeterCsv reports 'a.csv' Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\29\performance-reports\JMeterCSV\a.csv'. Performance: Recording JMeterCsv reports 'b.csv' Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\29\performance-reports\JMeterCSV\b.csv'. Performance: Percentage of relative difference outside -10.0 to +10.0 % sets the build as failure Performance: Percentage of relative difference outside -5.0 to +5.0 % sets the build as unstable Performance: Recording JMeterCsv reports 'a.csv' Performance: Recording JMeterCsv reports 'b.csv' Comparison build no. - 28 and 29 using 90 Percentile response time PrevBuildURI CurrentBuildURI PrevBuildURI90% CurrentBuildURI90% RelativeDiff RelativeDiffPercentage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
Sample Jenkinsfile:
node {
['a','b'].each {
sh "echo timeStamp,elapsed,label,responseCode,success > ${it}.csv"
10.times { i -> sh "echo ${1500000000000+i*1000},${i*100},$it,200,true >> ${it}.csv" }
}
perfReport([
sourceDataFiles: 'a.csv;b.csv',
modeEvaluation: false,
modeOfThreshold: true,
failBuildIfNoResultFile: true,
relativeUnstableThresholdNegative: 5,
relativeUnstableThresholdPositive: 5,
relativeFailedThresholdNegative: 10,
relativeFailedThresholdPositive: 10,
compareBuildPrevious: true,
configType: 'PRT'
])
}
