Trigger Still-Unstable is too heavy. We are mostly on unstable state and monitor change in Test results trend. Some test take time to fix. Some are forgotten to be erased/ingored (after you find that you are not able to fix them due some wierd error in environment). Meanwhile you commit and commit and the test results graph go up and down and you are still on unstable state.
So you constantly fire Still-Unstable trigger on email-ext. Unstable is synonymum for "Unknown" or "Unusable", because you still need to check manually Hudson dashboard for test results. Emails, Jabber are worhless.
Yes, it is for some part problem of company policy and developers experience, but anywhere you need time for fixing the test and meanwhile new changes are coming from SCM...
- new triggers for email configuration, based on Test result change, not on build status change.
a) Still-Unstable is good enough for someone. So must be preserved.
b) New triggers are substates of Still-Unstable state, so must be available only on this trigger selected and triggered only when "master" Still-Unstable is triggered. For UI - when I select Still-Unstable trigger, new selectbox with these substates & default "no substate"
c) New triggers are based on diff, so not only Test Results, but in general anything that can produce "Hudson Result table with diff" must be taken to consideration, from core or another plugins (Task scanner, PMD, CheckStyle, Emma etc). For UI - checkboxs
- 3 new substates:
- Still-Unstable-Increased ( "Test failed", "Checkstyle violation", "TODO tasks" whatever) - when there is +X diff on Result table
- Still-Unstable-Decreased - when there is -X diff on Result table
- Still-Unstable-Changed - when there is both -X diff and +X on Result table
– note_1: Still-Unstable-Unchanged substate (0 diff) is in fact the same as master Still-Unstable state. Can be added as a forth substate to complete the list of substates for selectbox and avoid ambiguity.
– note_2 The "Still-Unstable-Changed" is needed, it is not good for usability to have only +X and -X state and fire them both. One change can be "fix this, but screw another" and of cource one build can be triggerd by many SCM change, so this substate is valid.
d) As in general any diff can trigger email and any change can produce diff in some results, for usability is better to have two tresholds:
– change treshold:
if (+X) or (-Y) <= change treshold than trigger is not fired.
for example, in mostly any commit you change the Checkstyle results. You want to be emailed only if build violate more than for example 50 Checkstyle rules. For Test result the usual treshold from my perspective is 1; every new test failed is wrong.
– total treshold: If you are not interested in movements in some results, but wanted to be informed when it is "too bad" (mostly it is not change from STABLE to UNSTABLE build but a long slipping regression in something, so on UNSTABLE build).
=> with both substates and tresholds very strong Verify + Notify functionm for example when Email-ext plugin combined with Violations plugin.
=> call for this to be a new plugin / core function and not only improvement of email-ext plugin , because this kind of plugable configuration is I think usefull for not only anything that produce diff (any kind of verifier), but also anything, that notify somebody somehow (email, jabber plugin, etc.).
So not only to better select WHEN and email it, but select WHEN do it & HOW do it. Content for each notifier must be separated, it of course difference between email and jabber.