Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16081

xUnit plugin fails to recognize regression if total number of tests doesn't go up.

    XMLWordPrintable

    Details

    • Similar Issues:

      Attachments

        Activity

        Hide
        vladichko_jira Vlad Aginsky added a comment -

        Example:
        If there are 5 tests that failed, and on next run another set of 5 tests fails, it will not be able to detect a regression case here.

        Show
        vladichko_jira Vlad Aginsky added a comment - Example: If there are 5 tests that failed, and on next run another set of 5 tests fails, it will not be able to detect a regression case here.
        Hide
        vladichko_jira Vlad Aginsky added a comment -

        here is simplest set of files to reproduce the problem.
        Run 38 has all pass.
        run 40 has one test failed, was reported as "unstable"
        run 41 has another test failed, was reported as "unstable" by jenkins, while i was expecting to see "broken".

        Show
        vladichko_jira Vlad Aginsky added a comment - here is simplest set of files to reproduce the problem. Run 38 has all pass. run 40 has one test failed, was reported as "unstable" run 41 has another test failed, was reported as "unstable" by jenkins, while i was expecting to see "broken".
        Hide
        gbois Gregory Boissinot added a comment -

        xUnit Jenkins plugins provides a threshold section in order to change build status according the number of errors tests, the number of new errors tests, etc.
        Maybe it is a wrong configuration of threshold values.
        Could you attach your job configuration file (config.xml)?

        Show
        gbois Gregory Boissinot added a comment - xUnit Jenkins plugins provides a threshold section in order to change build status according the number of errors tests, the number of new errors tests, etc. Maybe it is a wrong configuration of threshold values. Could you attach your job configuration file (config.xml)?
        Hide
        vladichko Vlad Aginsky added a comment -

        Gregory, see attached file.

        Show
        vladichko Vlad Aginsky added a comment - Gregory, see attached file.
        Hide
        kerese Andras Kerese added a comment -

        I experience the same bug. Actually the job state doesn't change to failed or unstable, but regressions will be shown on the summary page of the build. Let me know if need more information.

        Show
        kerese Andras Kerese added a comment - I experience the same bug. Actually the job state doesn't change to failed or unstable, but regressions will be shown on the summary page of the build. Let me know if need more information.
        Hide
        x29a x29a added a comment -

        I am still experiencing this on xUnit 1.78 with Jenkins 1.544 (Windows).

        In the "Testresult" overview, testcases are marked with "Age: 1day". When i click that testcase, i see 12 testruns with "regression" and the last one with "failed". So the actual data for the testcase seems to be correct but the parsing for the "testresult" overview is somewhat off.

        Please tell me which other information might be useful for you.

        Show
        x29a x29a added a comment - I am still experiencing this on xUnit 1.78 with Jenkins 1.544 (Windows). In the "Testresult" overview, testcases are marked with "Age: 1day". When i click that testcase, i see 12 testruns with "regression" and the last one with "failed". So the actual data for the testcase seems to be correct but the parsing for the "testresult" overview is somewhat off. Please tell me which other information might be useful for you.
        Hide
        nfalco Nikolas Falco added a comment -

        The configuration is quite strange, you want that if there are new test failed the build is FAILED instead if there are no new failure but some test continue to fails than the build is marked as UNSTABLE?

        Show
        nfalco Nikolas Falco added a comment - The configuration is quite strange, you want that if there are new test failed the build is FAILED instead if there are no new failure but some test continue to fails than the build is marked as UNSTABLE?
        Hide
        nfalco Nikolas Falco added a comment -

        By design total wins on new (failures in your case) and failure threshold have precedences over unstable threshold.
        For your case I will expect that a simple 1,2,3,4 new failures mark the build as unstable untill you exceed the threshold of total failures.
        Setup it with

        • 0 new failure -> UNSTABLE
        • 2 totale failures -> FAILED

        So that in your case:
        Run 38 has all pass.
        run 40 has one test failed -> "unstable"
        run 42 has another two test failed -> "failed"

        Show
        nfalco Nikolas Falco added a comment - By design total wins on new (failures in your case) and failure threshold have precedences over unstable threshold. For your case I will expect that a simple 1,2,3,4 new failures mark the build as unstable untill you exceed the threshold of total failures. Setup it with 0 new failure -> UNSTABLE 2 totale failures -> FAILED So that in your case: Run 38 has all pass. run 40 has one test failed -> "unstable" run 42 has another two test failed -> "failed"

          People

          • Assignee:
            nfalco Nikolas Falco
            Reporter:
            vladichko_jira Vlad Aginsky
          • Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: