-
Improvement
-
Resolution: Incomplete
-
Major
-
None
The current criteria for the regression/improvement triggers is based on failure stats only (e.g., regression triggers if number of failures has increased since last build). However, would like to extend this set of criteria to optionally include successes and total tests run. For example:
regression = # of failures has increased OR # of successes has decreased OR # of total tests run has decreased
improvement = # of failures has decreased OR # of successes has increased OR # of total tests run has increased
The background behind this request is a unit test "timeout" (mstest) which was not reported as a failure. Looking at the normal Jenkins test result stats, there was no indication that anything was wrong: there were no failures and the total number of tests was the same. If the test result stats reported number of successes, then I might have noticed this number had dropped, but it's a moot point since this is not displayed. The regression trigger would not have kicked in since the number of failures did not increase. For this specific scenario, the "# of successes has decreased" regression trigger applies. The "total tests run" trigger would come in handy in the situation where developers inadvertently (or intentionally) enable/disable tests. Thanks, Steve