Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21986

RunTestSet Function Within AlmTestSetsRunner.cs Incorrectly Setting Test Execution Duration

    Details

    • Similar Issues:

      Description

      Refer to AlmTestSetsRunner.cs file located here: https://github.com/jenkinsci/hp-application-automation-tools-plugin/blob/master/HpToolsLauncher/Runners/AlmTestSetsRunner.cs

      The issue is in the RunTestSet function.

      1. Before running the test set, a stopwatch timer variable testSw is initialized to null. Line 529.
      2. A While loop then runs through each test instance in the test set. The While loop is on line 548.
      a. Before each test instance is executed, the stopwatch is started, and that testSw variable is set to “Stopwatch.StartNew()”. This occurs on line 595.
      3. Unfortunately, as each test instance is completed, the current value of the stopwatch is not saved/stored.
      a. The stopwatch is reset, the next test instance starts, and the previous stopwatch value is lost.
      4. When the last test instance in the test set is completed, the code exits the While loop.
      a. At that point, the testSw variable contains the duration of the last test instance.
      b. For our test run, the last test instance took 4 minutes 20 seconds.
      5. Next, on line 655, the code loops through all test instances to consolidate information for each test instance.
      a. The Runtime for each test instance is set to testSw.Elapsed on line 672.
      b. Remember, that the testSw value is currently set to the duration of the last test instance … 4 minutes 20 seconds for our example.
      c. Also, remember that the stopwatch is still running. A stop command was never executed.
      d. So, the duration for the first test instance is actually getting set to the duration of the last test instance.
      e. Each subsequent test instance duration is getting set to the duration of the last test instance plus some small additional time because the stopwatch is still running and updating testSw.
      f. That explains why our durations start at 4 minutes 20 seconds, and gradually increase to 4 minutes 48 seconds. It took 28 seconds to loop through the test instances in this FOR loop.
      6. The incorrect Total Runtime in the upper right corner of the All Tests page is just the sum of these incorrect test instance durations.
      a. Fix the test instance problem, and the Total Runtime problem will automatically be fixed.

      This code needs an update to fix this test duration issue.

      Thanks,
      Keith

        Attachments

          Activity

          Hide
          allan_burdajewicz Allan BURDAJEWICZ added a comment -

          Any update on this issue?

          I believe I saw the same problem: in my scenario, the job builds successfully and takes about 3m 30s but the test result show a duration of 16 hrs which is incredibly wrong.

          Show
          allan_burdajewicz Allan BURDAJEWICZ added a comment - Any update on this issue? I believe I saw the same problem: in my scenario, the job builds successfully and takes about 3m 30s but the test result show a duration of 16 hrs which is incredibly wrong.
          Hide
          roy_lu Roy Lu added a comment -

          xiwen zhao 

          Heres' some advise on HPToolsLaucher which I don't understand. Please check and reply if you can.

          Show
          roy_lu Roy Lu added a comment - xiwen zhao   Heres' some advise on HPToolsLaucher which I don't understand. Please check and reply if you can.

            People

            • Assignee:
              aslaakso Anda Sorina Laakso
              Reporter:
              jacobsk Keith Jacobs
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: