Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38530

incorrect verdict when processing reports in parallel

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The outcome of the report happens to be incorrect in 2 scenarios.

      Classic jobs

      Scenario:

      1. A project is created with FailedThreshold/unstableThreshold set to 1 because there's 1 test expected to fail in the next run
      2. Job N starts executing, execution time will take minutes
      3. FailedThreshold/unstableThreshold gets adjusted to 0 because the test is expected to pass in the next run
      4. Job N terminates: the execution is then reported as UNSTABLE instead of the expected SUCCESS by the xUnit publisher, which wrongly uses the adjusted threshold (0) instead of the one configured at startup (1)

      Pipeline job

      Scenario:

      1. A parallel section is defined with 2 test steps (A and B), each one with FailedThreshold/unstableThreshold set to 0
      2. test step A will generate 1 test failure
      3. test step B will generate 0 test failure
      4. test steps starts in parallel
      5. test step A terminates with 1 test failure: the execution is correctly reported as UNSTABLE
      6. test step B terminates with 0 test failure: the execution is wrongly reported as UNSTABLE
        When test step B terminates before test step A, that's the other way around: test step B is correctly reported as SUCCESS, then test step A is wrongly reported as SUCCESS.

        Attachments

          Issue Links

            Activity

            Hide
            agomez_io Alberto Gomez-Estrada added a comment - - edited

            Are we sure that this is a bug with the code and not inadequate documentation?

            I encountered a similar issue, and upon testing, I noticed the following behavior:

            If the threshold for UNSTABLE is set to N, then the test will have a SUCCESS value up until N - 1 FAILS.

            If the threshold for FAILED is set to N, then the test will have a SUCCESS or UNSTABLE value up until N - 1 FAILS.

            I haven't looked at the source code, but this feels like more of an issue with the documentation being unclear on how this feature works.

            Note about parallel builds

            I also have two tests running in parallel. I noticed that the xml that xUnit was parsing for each of the test results had the same "title" value, despite the filenames being different. This caused xUnit to overwrite the first file it parsed, and only report the results of the second file.

            Show
            agomez_io Alberto Gomez-Estrada added a comment - - edited Are we sure that this is a bug with the code and not inadequate documentation? I encountered a similar issue, and upon testing, I noticed the following behavior: If the threshold for UNSTABLE is set to N, then the test will have a SUCCESS value up until N - 1 FAILS. If the threshold for FAILED is set to N, then the test will have a SUCCESS or UNSTABLE value up until N - 1 FAILS. I haven't looked at the source code, but this feels like more of an issue with the documentation being unclear on how this feature works. Note about parallel builds I also have two tests running in parallel. I noticed that the xml that xUnit was parsing for each of the test results had the same "title" value, despite the filenames being different. This caused xUnit to overwrite the first file it parsed, and only report the results of the second file.

              People

              • Assignee:
                nfalco Nikolas Falco
                Reporter:
                rdesgroppes RĂ©gis Desgroppes
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: