Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12815

Add an option to select Build status when no Test is found

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: junit-plugin
    • Labels:
      None
    • Environment:
      any
    • Similar Issues:

      Description

      Currently, when no test report is found by JUnit report plugin (ie fileset is empty), build fails.
      In some cases, this behaviour is too constraining and implies stopping a cascade of builds for example, and this is not always the mist convenient behavior for a build stack.

      This plugin should provide an option to select build status when fileset is empty:

      • FAILED (Red): current default behaviour
      • UNSTABLE (Orange)
      • STABLE (Blue/Green)

      This is an issue we have to build JBoss Tools, which we had to workaround by adding build steps, or adding dummy files to the fileset to avoid locking all builds in case of test failing.

        Attachments

          Issue Links

            Activity

            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            What I'm really trying to avoid in Jenkins is to have two dozen checkboxes and text fields to cover every imaginable use case. Because Jenkins is a GUI tool primarily, every such added switch has a real cost. In my humble opinion, this feature falls on the wrong side of this trade-off.

            I suppose what we can do is to define an extension point that abstracts away the mapping from the test result into the result code. The config UI can be smart enough not to show this option when no implementation is provided (and we will not ship any in the junit plugin), and in this mode it can retain the current behavior.

            You can then develop an implementation of this extension point and achieve the semantics you prefer.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - What I'm really trying to avoid in Jenkins is to have two dozen checkboxes and text fields to cover every imaginable use case. Because Jenkins is a GUI tool primarily, every such added switch has a real cost. In my humble opinion, this feature falls on the wrong side of this trade-off. I suppose what we can do is to define an extension point that abstracts away the mapping from the test result into the result code. The config UI can be smart enough not to show this option when no implementation is provided (and we will not ship any in the junit plugin), and in this mode it can retain the current behavior. You can then develop an implementation of this extension point and achieve the semantics you prefer.
            Hide
            nickboldt Nick Boldt added a comment -

            It would also be nice to say that missing test coverage files (or empty ones) should result in a warning (UNSTABLE) rather than an error (FAILED), or in some cases be ignored entirely (STABLE).

            +1 for being able to tweak the output colour/status of test & test coverage failures to define them as less critical.

            Show
            nickboldt Nick Boldt added a comment - It would also be nice to say that missing test coverage files (or empty ones) should result in a warning (UNSTABLE) rather than an error (FAILED), or in some cases be ignored entirely (STABLE). +1 for being able to tweak the output colour/status of test & test coverage failures to define them as less critical.
            Hide
            mcklaus Klaus Azesberger added a comment - - edited

            if you run an incremental maven build (as part of a free style project) and try to publish surefire-reports there is a real possibility that there were no modules within the project list to build, that had led to testexecution and thus surefire-reports to publish. so that's a real pain in the a... when you are forced to write dummy reports in order to have a green build after a green build that didnt need to execute tests.

            Show
            mcklaus Klaus Azesberger added a comment - - edited if you run an incremental maven build (as part of a free style project) and try to publish surefire-reports there is a real possibility that there were no modules within the project list to build, that had led to testexecution and thus surefire-reports to publish. so that's a real pain in the a... when you are forced to write dummy reports in order to have a green build after a green build that didnt need to execute tests.
            Hide
            dozd Zdenek Dolezal added a comment -

            I want to have one job template config for all my projects and this is also an unpleasant issue for me.

            Show
            dozd Zdenek Dolezal added a comment - I want to have one job template config for all my projects and this is also an unpleasant issue for me.
            Hide
            dthomas Dirk Thomas added a comment -

            +1 since I generate jobs from a template from thousands of projects I simply don't know if a project will generate test results files or not.

            Using the `xunit` plugin instead (which has the requested configuration option) does not work for me either since it is much stricter when validating the xml files.
            As a temporary fix I conditionally generate a dummy result file. But that "pollutes" all test statistics.

            I do understand the fine line of flexibility vs. bloated but this seems to break a use case several users are having.

            Show
            dthomas Dirk Thomas added a comment - +1 since I generate jobs from a template from thousands of projects I simply don't know if a project will generate test results files or not. Using the `xunit` plugin instead (which has the requested configuration option) does not work for me either since it is much stricter when validating the xml files. As a temporary fix I conditionally generate a dummy result file. But that "pollutes" all test statistics. I do understand the fine line of flexibility vs. bloated but this seems to break a use case several users are having.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - https://github.com/jenkinsci/junit-plugin/pull/28 has been integrated
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The fix has been released in 1.10

            Show
            oleg_nenashev Oleg Nenashev added a comment - The fix has been released in 1.10

              People

              • Assignee:
                Unassigned
                Reporter:
                mickael_istria Mickael Istria
              • Votes:
                10 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: