Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35911

CTest returning non-zero error code should not stop the build

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      If a test fails then CTest returns a non-zero return code (8 in my configuration) and jenkins build is stopped.
      This makes the build fail even if higher unstable and fail thresholds are selected.
      An option to select the return code to be ignored will work the best.

        Attachments

          Activity

          Hide
          15knots Martin Weber added a comment -

          Would it be sufficient to add a boolean option that allows to ignore the exit status of ctest it it does not indicate success?

          Show
          15knots Martin Weber added a comment - Would it be sufficient to add a boolean option that allows to ignore the exit status of ctest it it does not indicate success?
          Hide
          dcarp Dragos Carp added a comment - - edited

          Yes, adding an option to ignore the exit status would do the job in my case.
          On the other side, it can be that CTest returns a different error code if there is another kind of error (wrong parameters, etc.). That kind of errors shouldn't be ignored. Unfortunately the CTest documentation does not specifies the error codes.

          Show
          dcarp Dragos Carp added a comment - - edited Yes, adding an option to ignore the exit status would do the job in my case. On the other side, it can be that CTest returns a different error code if there is another kind of error (wrong parameters, etc.). That kind of errors shouldn't be ignored. Unfortunately the CTest documentation does not specifies the error codes.
          Hide
          dcarp Dragos Carp added a comment -

          I have reported the issue to CTest: https://gitlab.kitware.com/cmake/cmake/issues/16151

          Show
          dcarp Dragos Carp added a comment - I have reported the issue to CTest: https://gitlab.kitware.com/cmake/cmake/issues/16151
          Hide
          15knots Martin Weber added a comment -

          Please try https://jenkins.ci.cloudbees.com/job/plugins/job/cmakebuilder-plugin/org.jenkins-ci.plugins$cmakebuilder/102/artifact/org.jenkins-ci.plugins/cmakebuilder/2.4.3-SNAPSHOT/cmakebuilder-2.4.3-SNAPSHOT.hpi and provide some feedback.
          My unittests are unreliable here, since ctest v. 2.8.11.2 seems to always exit with success so they never fail for this issue .
          And please have a look at the online help and send suggestions, as I am not a native speaker.

          Show
          15knots Martin Weber added a comment - Please try https://jenkins.ci.cloudbees.com/job/plugins/job/cmakebuilder-plugin/org.jenkins-ci.plugins$cmakebuilder/102/artifact/org.jenkins-ci.plugins/cmakebuilder/2.4.3-SNAPSHOT/cmakebuilder-2.4.3-SNAPSHOT.hpi and provide some feedback. My unittests are unreliable here, since ctest v. 2.8.11.2 seems to always exit with success so they never fail for this issue . And please have a look at the online help and send suggestions, as I am not a native speaker.
          Hide
          dcarp Dragos Carp added a comment - - edited

          Unfortunately it didn't work out as expected. Actually even that the Plugin Manager shows that "CMake plugin 2.4.3-SNAPSHOT (private-e95ede2f-jenkins)" is installed, I couldn't find any option to disable the checking of the exit code. Currently I'm using Jenkins 2.5 installed on Windows 2008 Server.
          The relevant section in the "Console Output":

          The following tests FAILED:
          	 72 - testFoo (Failed)
          	 74 - testBar (Failed)
          	 75 - testBaz (Failed)
          	116 - testQux (SEGFAULT)
          
          Errors while running CTest
          Build step 'CMake/CPack/CTest execution' changed build result to FAILURE
          
          Show
          dcarp Dragos Carp added a comment - - edited Unfortunately it didn't work out as expected. Actually even that the Plugin Manager shows that "CMake plugin 2.4.3-SNAPSHOT (private-e95ede2f-jenkins)" is installed, I couldn't find any option to disable the checking of the exit code. Currently I'm using Jenkins 2.5 installed on Windows 2008 Server. The relevant section in the "Console Output": The following tests FAILED: 72 - testFoo (Failed) 74 - testBar (Failed) 75 - testBaz (Failed) 116 - testQux (SEGFAULT) Errors while running CTest Build step 'CMake/CPack/CTest execution' changed build result to FAILURE
          Hide
          15knots Martin Weber added a comment -

          You will find a new text box labeled Ignore exit codes in the CMake/CPack/CTest build step now, which should satisfy your needs.

          Show
          15knots Martin Weber added a comment - You will find a new text box labeled Ignore exit codes in the CMake/CPack/CTest build step now, which should satisfy your needs.
          Hide
          dcarp Dragos Carp added a comment -

          Today I got the chance to test this feature in version 2.4.3 and it works as expected. Thank you!

          Show
          dcarp Dragos Carp added a comment - Today I got the chance to test this feature in version 2.4.3 and it works as expected. Thank you!

            People

            • Assignee:
              15knots Martin Weber
              Reporter:
              dcarp Dragos Carp
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: