Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41560

Robot Framework plugin on Windows not honoring passed/unstable percentages

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: robot-plugin
    • Labels:
      None
    • Environment:
      Windows 10 Pro slave using jnlp start
    • Similar Issues:

      Description

      We have Jenkins master running on Ubuntu 14.04 with Windows 10 Pro and Mac OSX slaves. All seem to be working normally except for the Windows 10 slave. If ANY testcases fail (even just one) then the build/run is marked a failure. We have a pass percentage of 95% and an unstable percentage of 90% set for the build and this is working for everything EXCEPT Windows 10. The Windows 10 machine is using the WebStart jnlp file to connect to the master machine, the same as the Mac OSX box is using.

        Attachments

          Activity

          Hide
          aleksisimell Aleksi Simell added a comment - - edited

          Just to verify, you are running robot with --nostatusrc option?

          Show
          aleksisimell Aleksi Simell added a comment - - edited Just to verify, you are running robot with --nostatusrc option?
          Hide
          jerry57 Jerry Schneider added a comment -

          At the time I submitted this issue, no I was not using the --nostatusrc flag.  Right now I am using it ONLY on my Windows Slaves, the Linux and Mac boxes are NOT using this flag.  I am a little puzzled as to why it is required on Windows but not Mac or Linux?

           

          Show
          jerry57 Jerry Schneider added a comment - At the time I submitted this issue, no I was not using the --nostatusrc flag.  Right now I am using it ONLY on my Windows Slaves, the Linux and Mac boxes are NOT using this flag.  I am a little puzzled as to why it is required on Windows but not Mac or Linux?  
          Hide
          jpiironen jpiironen added a comment -

          Hiya, Robot Framework returns the number of failed critical test cases as the return code after each run. Non-zero return codes will fail the 'execute shell' or 'batch' build step on any OS, which renders using the pass percentage as a threshold for failed/unstable build meaningless in many use cases. If the build result is already marked failed when the plugin kicks in, it won't change it. Forcing the exit code to be always zero with --nostatusrc will always pass the shell/batch build step and leave the decision of pass/fail/unstable to the plugin (or other steps run afterwards).

          Is your robot run the same on the other OS or are you perhaps using some other shell commands or flags to make the non-zero return code not count?

          Show
          jpiironen jpiironen added a comment - Hiya, Robot Framework returns the number of failed critical test cases as the  return code  after each run. Non-zero return codes will fail the 'execute shell' or 'batch' build step on any OS, which renders using the pass percentage as a threshold for failed/unstable build meaningless in many use cases. If the build result is already marked failed when the plugin kicks in, it won't change it. Forcing the exit code to be always zero with --nostatusrc will always pass the shell/batch build step and leave the decision of pass/fail/unstable to the plugin (or other steps run afterwards). Is your robot run the same on the other OS or are you perhaps using some other shell commands or flags to make the non-zero return code not count?
          Hide
          aleksisimell Aleksi Simell added a comment -

          Jerry Schneider any new information?

          Show
          aleksisimell Aleksi Simell added a comment - Jerry Schneider any new information?
          Hide
          jerry57 Jerry Schneider added a comment -

          For now, I have added "–nostatusrc" to all my windows jobs, but it is not on the Linux or Max OS X jobs.  The Linux and Mac OS X jobs are running normally and honoring the percentages entered for the Robotframework plugin.  Windows is now honoring it as long as I have the "–nostatusrc" flag.  Why the difference between the operating systems?

           

          Show
          jerry57 Jerry Schneider added a comment - For now, I have added "–nostatusrc" to all my windows jobs, but it is not on the Linux or Max OS X jobs.  The Linux and Mac OS X jobs are running normally and honoring the percentages entered for the Robotframework plugin.  Windows is now honoring it as long as I have the "–nostatusrc" flag.  Why the difference between the operating systems?  

            People

            • Assignee:
              jpiironen jpiironen
              Reporter:
              jerry57 Jerry Schneider
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: