Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49528

Peformance graphs for individual tests are not created when a single test has failed

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: perfpublisher-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.89.3
      Performance Publisher plugin 8.05
      JRE 1.8.0_144-b01
      Windows 7
    • Similar Issues:

      Description

      I have tried a lot of things, but whenever a single test fails, the performance publisher plugin does not create graphs for individual tests. Only the overview (sum) of all tests is created. If no test has failed, the graphs are created as expected.

      Since I cannot think of any reason why someone would want this, I must assume that this is a bug.

      My Jenkins job runs the tests with MSTest. Each test measures the execution time for various scenarios and each test creates a new xml file that is parsed by the performance publisher plugin. If a test reports an error (e.g. the execution time exceeds a certain limit), the job is marked as unstable. This is done automatically by the MSTest plugin.

      The plugin works fine with multiple xml files, as long as all tests have passed.

        Attachments

          Activity

          Hide
          eschava Eugene Schava added a comment -

          Sorry, I need more details

          Is your issue related to multiple XML files only? I usually use only one XML report file and cannot confirm that some chart differs if there is no or only one failed test

          Show
          eschava Eugene Schava added a comment - Sorry, I need more details Is your issue related to multiple XML files only? I usually use only one XML report file and cannot confirm that some chart differs if there is no or only one failed test
          Hide
          tln Tobias Lingemann added a comment -

          I don't believe it is related to the number of XML files. I took a look at the source code and while I'm not sure, I believe the problem is in TestDetails.java.

          The graph for execution time, compile time and performance is only created when the result of the abstract build is success.

           

          if (!abstractBuild.isBuilding() && abstractBuild.getResult().isBetterOrEqualTo(Result.SUCCESS))
          

          Curiously the graph for custom metrics is created if the build is unstable or better. I have updated my tests to use a custom metric in the meantime. I will check tomorrow if it worked.

           

          Show
          tln Tobias Lingemann added a comment - I don't believe it is related to the number of XML files. I took a look at the source code and while I'm not sure, I believe the problem is in TestDetails.java . The graph for execution time, compile time and performance is only created when the result of the abstract build is success.   if (!abstractBuild.isBuilding() && abstractBuild.getResult().isBetterOrEqualTo(Result.SUCCESS)) Curiously the graph for custom metrics is created if the build is unstable or better. I have updated my tests to use a custom metric in the meantime. I will check tomorrow if it worked.  
          Hide
          tln Tobias Lingemann added a comment -

          As I suspected. The custom metric graph was created successfully.

          Show
          tln Tobias Lingemann added a comment - As I suspected. The custom metric graph was created successfully.
          Hide
          eschava Eugene Schava added a comment - - edited

          Thanks, I see now.

          So now

          execution time chart is shown if the result is success

          compile time chart is shown if the result is success

          performance chart is shown is the result is success

          metrics chart is shown if the result is unstable or success

           

          It looks logical to change these conditions to:

          execution time chart is shown if the result is unstable or success

          compile time chart is shown if the result is failure, unstable or success (better than not built)

          performance chart is shown is the result is unstable or success

          metrics chart is shown if the result is unstable or success (do not change)

           

          Agree?

           

           

           

          Show
          eschava Eugene Schava added a comment - - edited Thanks, I see now. So now execution time chart is shown if the result is success compile time chart is shown if the result is success performance chart is shown is the result is success metrics chart is shown if the result is unstable or success   It looks logical to change these conditions to: execution time chart is shown if the result is unstable or success compile time chart is shown if the result is  failure, unstable or success (better than not built) performance chart is shown is the result is unstable or success metrics chart is shown if the result is unstable or success (do not change)   Agree?      
          Hide
          tln Tobias Lingemann added a comment -

          Agreed

          Show
          tln Tobias Lingemann added a comment - Agreed
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Eugene Schava
          Path:
          src/main/java/hudson/plugins/PerfPublisher/TestDetails.java
          http://jenkins-ci.org/commit/perfpublisher-plugin/f49132726ea71fef2d0b8df91a33ba42c929ad16
          Log:
          JENKINS-49528 Peformance graphs for individual tests are not created when a single test has failed

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Eugene Schava Path: src/main/java/hudson/plugins/PerfPublisher/TestDetails.java http://jenkins-ci.org/commit/perfpublisher-plugin/f49132726ea71fef2d0b8df91a33ba42c929ad16 Log: JENKINS-49528 Peformance graphs for individual tests are not created when a single test has failed
          Hide
          eschava Eugene Schava added a comment -

          I've committed and pushed this change to repository

          But I think it's too minor change to create new release

          Are you OK to build and update your Jenkins installation manually with the version from github repository?

          Show
          eschava Eugene Schava added a comment - I've committed and pushed this change to repository But I think it's too minor change to create new release Are you OK to build and update your Jenkins installation manually with the version from github repository?
          Hide
          tln Tobias Lingemann added a comment -

          For now I simply create the graph via the custom metric. I will wait until a new release is published.

          Show
          tln Tobias Lingemann added a comment - For now I simply create the graph via the custom metric. I will wait until a new release is published.

            People

            • Assignee:
              eschava Eugene Schava
              Reporter:
              tln Tobias Lingemann
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: