Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37739

Plugin not displaying Robot Test Summary correctly if robot framework task running in parallel in pipeline

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Trivial
    • Resolution: Unresolved
    • Component/s: robot-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      I am building running my robot framework test case parallel in Jenkins pipeline. However the Robot Test Summary generated post build is not correct, it displaying the count/summary of last finished in parallelism, however it should show the count of individual job. It was overwriting the report also earlier but that I saved as different name & it worked fine, but still the Robot Test Summary is problem(please not my both suite below have different number of test cases).

      suite1:{
      try

      { mvn -s /settings.xml install robotframework:run -Dexcludes=inprogress -Dsuites=test1 -DLogFileName=suite1_log.html -DoutputFileName=suite1_output.xml -DreportFileName=suite1_report.html' currentBuild.result = 'SUCCESS' }

      catch(any)

      { currentBuild.result = 'FAILURE' throw any }

      finally

      { step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite1_log.html', outputFileName: 'suite1_output.xml', reportFileName: 'suite1_report.html']) }

      },
      suite2:{
      try

      { mvn -s /settings.xml robotframework:run -Dexcludes=inprogress -Dsuites=test2 -DLogFileName=suite2_log.html -DoutputFileName=suite2_output.xml -DreportFileName=suite2_report.html' currentBuild.result = 'SUCCESS' }

      catch(any)

      { currentBuild.result = 'FAILURE' //sendMail() throw any }

      finally

      { step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports/', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite2_log.html', outputFileName: 'suite2_output.xml', reportFileName: 'suite2_report.html']) }

        Attachments

          Activity

          Hide
          iwelch Ian Welch added a comment -

          Forewarning: I'm not a Java Developer so I may say stupid things
          I'm working on a pull request to resolve this issue. Thus far I have been able to add an optional parameter that defines a path in the Job directory to store the results. I'm also able to properly display the results of each test result file (using the result directory in the summary window. The next, and final, piece of the change is to properly link out to the Report visualization under "Browse Results".
          I've pored over the code for the last few days and I think I have identified the this data comes from `RobotResult.java` but I can't for the life of me figure out how the data is populated into this class. Can anyone point me in the proper direction to figure out how this data is populated?
          My best guess is that the data is stored in

          private transient List<RobotResultStatistics> overallStats;
          

          and populated from

          public void setStatsByCategory(List<RobotResultStatistics> statsByCategory) {
          		this.overallStats = statsByCategory;
          	}
          

          Show
          iwelch Ian Welch added a comment - Forewarning: I'm not a Java Developer so I may say stupid things I'm working on a pull request to resolve this issue. Thus far I have been able to add an optional parameter that defines a path in the Job directory to store the results. I'm also able to properly display the results of each test result file (using the result directory in the summary window. The next, and final, piece of the change is to properly link out to the Report visualization under "Browse Results". I've pored over the code for the last few days and I think I have identified the this data comes from `RobotResult.java` but I can't for the life of me figure out how the data is populated into this class. Can anyone point me in the proper direction to figure out how this data is populated? My best guess is that the data is stored in private transient List<RobotResultStatistics> overallStats; and populated from public void setStatsByCategory(List<RobotResultStatistics> statsByCategory) { this .overallStats = statsByCategory; }
          Hide
          pradeeppt Pradeep PT added a comment -

          Is this issue fixed ?

          Show
          pradeeppt Pradeep PT added a comment - Is this issue fixed ?
          Hide
          dirkrichter Dirk Richter added a comment -

          I do not see any open or merged pull requests. Thus i think issue is still present.

          In my opinion having multiple test summaries is ugly. Did you considered merging them up to only 1 output.xml e.g. using rebot (see here)

          When your tests run locally on the same host, try using pabot which has less overhead and merges results, too.

          I decreased priority, because there are 2 possible workarounds.

          Show
          dirkrichter Dirk Richter added a comment - I do not see any open or merged pull requests. Thus i think issue is still present. In my opinion having multiple test summaries is ugly. Did you considered merging them up to only 1 output.xml e.g. using rebot (see here ) When your tests run locally on the same host, try using pabot which has less overhead and merges results, too. I decreased priority, because there are 2 possible workarounds.
          Hide
          singhshyam shyam singh added a comment -

          Dirk Richter, Yes we are doing the same way now, merging the reports  & constructing a merged output.xml & same feed to plugin, but with the number of parallel plan & Test cases growing, merging of report is an overhead, it's increasing the pipeline time. Also Sometime it's great if we can plot the data on module/domain basis separately instead of merging all. Here main concern is the time while merging of report. we have around 10 parallel plan running & merging itself take ~5 min. Good to have this feature so we can restrict merging to module or domain level only.

          Show
          singhshyam shyam singh added a comment - Dirk Richter , Yes we are doing the same way now, merging the reports  & constructing a merged output.xml & same feed to plugin, but with the number of parallel plan & Test cases growing, merging of report is an overhead, it's increasing the pipeline time. Also Sometime it's great if we can plot the data on module/domain basis separately instead of merging all. Here main concern is the time while merging of report. we have around 10 parallel plan running & merging itself take ~5 min. Good to have this feature so we can restrict merging to module or domain level only.
          Hide
          megafon Jan Duris added a comment -

          Hi guys. Any progress with this issue? I used it same way (merged results by rebot), but I would prefer to display separate Robot Test Summaries for separate tests which runs in parallel.

          Show
          megafon Jan Duris added a comment - Hi guys. Any progress with this issue? I used it same way (merged results by rebot), but I would prefer to display separate Robot Test Summaries for separate tests which runs in parallel.
          Hide
          jpiironen jpiironen added a comment -

          Hi,

          We've resumed development of the plugin and currently doing a release for more urgent bugs. This seems like a useful improvement for the next iteration.

          Show
          jpiironen jpiironen added a comment - Hi, We've resumed development of the plugin and currently doing a release for more urgent bugs. This seems like a useful improvement for the next iteration.
          Hide
          beeofdoom Matt Calderaz added a comment -

          Just encountered this issue today in overhauling CI for our RF tests; it's not critical, but a fix would be greatly appreciated.

          Show
          beeofdoom Matt Calderaz added a comment - Just encountered this issue today in overhauling CI for our RF tests; it's not critical, but a fix would be greatly appreciated.
          Hide
          graines Glen Raines added a comment -

          Our team here would also prefer to display separate Robot Test Summaries, so a fix would be really appreciated.

          Show
          graines Glen Raines added a comment - Our team here would also prefer to display separate Robot Test Summaries, so a fix would be really appreciated.
          Hide
          mpenven Mériadec Penven added a comment -

          I'm facing the same issue on my project. I confirm that a fix would be really appreciated 

          Thanks in advance

          Show
          mpenven Mériadec Penven added a comment - I'm facing the same issue on my project. I confirm that a fix would be really appreciated  Thanks in advance
          Hide
          singhshyam shyam singh added a comment -

          We have achieved the same in-house by modifying the same plugin, instead of merging the report, we are showing the merge count and all the report in tabular form,  user can access individual report. Indeed this is very fast, whereas we used to spend lots of time in merging the report . Please refer the attached screen shot.

          Show
          singhshyam shyam singh added a comment - We have achieved the same in-house by modifying the same plugin, instead of merging the report, we are showing the merge count and all the report in tabular form,  user can access individual report. Indeed this is very fast, whereas we used to spend lots of time in merging the report . Please refer the attached screen shot.
          Hide
          sipani909 Abhijit Sipani added a comment -

          shyam singh Can you please provide details of how you were able to generate the table?

          Show
          sipani909 Abhijit Sipani added a comment - shyam singh Can you please provide details of how you were able to generate the table?
          Hide
          astaruch Andrej Staruch added a comment -

          Hi, something similar was addressed in the warning-ng plugin https://issues.jenkins-ci.org/browse/JENKINS-54027?attachmentViewMode=list.
          Maybe if the id and name parameters would be added, it could solve this issue.

          Show
          astaruch Andrej Staruch added a comment - Hi, something similar was addressed in the warning-ng plugin https://issues.jenkins-ci.org/browse/JENKINS-54027?attachmentViewMode=list . Maybe if the id and name parameters would be added, it could solve this issue.

            People

            • Assignee:
              hifi Juho Saarinen
              Reporter:
              singhshyam shyam singh
            • Votes:
              9 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated: