Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37663

Build is not marked as UNSTABLE even though failures are found

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I'm using JUnitResultsArchive to record test results, and I'm seeing a weird bug where there are failures reported but the build is not marked unstable.

      This is via Jenkins pipeline, and immediately after the step, I'm echoing the build result:

      step([$class: 'JUnitResultArchiver', testResults: 'build/xunit.xml'])
      echo ">>> BUILD RESULT ${currentBuild.result}"
      

      I can confirm that a test failed in that step, and the result printed is ">>> BUILD RESULT null"

      This doesn't always happen - in some other runs, the result is UNSTABLE as expected.

        Attachments

          Issue Links

            Activity

            Hide
            tolar robert tolar haining added a comment -

            I'm seeing the same thing.

            Show
            tolar robert tolar haining added a comment - I'm seeing the same thing.
            Hide
            joshma Joshua Ma added a comment -

            This is still affecting us - I have two builds (pipeline runs) back to back, where the second build does not change anything other than static images (not test related). Both builds have 1 test failing, but the first is SUCCESS while the second is UNSTABLE.

            Show
            joshma Joshua Ma added a comment - This is still affecting us - I have two builds (pipeline runs) back to back, where the second build does not change anything other than static images (not test related). Both builds have 1 test failing, but the first is SUCCESS while the second is UNSTABLE.
            Hide
            girardc79 Chris Girard added a comment -

            We are also seeing this in our builds. We are using the multi-branch pipeline (i.e. Jenkinsfile) with the "junit" step. We have also seen back-to-back builds where the actual failing tests did not change, but one build was marked unstable while the other was passing.

            Peculiar symptom: On the build status page for such builds, note the "Test Result (no failures)" followed immediately by the "Show All Failed Tests >>>" link:

            Clicking the Show All link expanded to the 39 failing tests we had for that build.

            Show
            girardc79 Chris Girard added a comment - We are also seeing this in our builds. We are using the multi-branch pipeline (i.e. Jenkinsfile) with the "junit" step. We have also seen back-to-back builds where the actual failing tests did not change, but one build was marked unstable while the other was passing. Peculiar symptom: On the build status page for such builds, note the "Test Result (no failures)" followed immediately by the "Show All Failed Tests >>>" link: Clicking the Show All link expanded to the 39 failing tests we had for that build.
            Hide
            therealwaldo Will Freeman added a comment - - edited

            I'm also seeing this; Jenkins 2.90 Junit 1.22.1  using 'junit' in declarative pipeline.

            Show
            therealwaldo Will Freeman added a comment - - edited I'm also seeing this; Jenkins 2.90 Junit 1.22.1  using 'junit' in declarative pipeline.
            Hide
            dantran dan tran added a comment - - edited

            I started to see this issue after upgrading to 1.22.x at my declarative pipeline. Ended up to revert to 1.21

            our pipeline maven surefire test stages with test failure set to ignore, and use JUnit plugin at post always block to detect build failure state. 

            Andrew Bayer , I think this is reproducible regression

            post {

                always {

                    //detect test failure
                    junit allowEmptyResults: true, testResults: '**/junitreports/.xml
                }

                success{

                    //this always gets called even with test failure in always block
                    sendEmail(config, 'SUCCESS', 'ecdm-mail')}

                }

            ....

            }

             

             

            Show
            dantran dan tran added a comment - - edited I started to see this issue after upgrading to 1.22.x at my declarative pipeline. Ended up to revert to 1.21 our pipeline maven surefire test stages with test failure set to ignore, and use JUnit plugin at post always block to detect build failure state.  Andrew Bayer , I think this is reproducible regression post {     always {         //detect test failure         junit allowEmptyResults: true, testResults: '** /junitreports/ .xml     }     success{         //this always gets called even with test failure in always block         sendEmail(config, 'SUCCESS', 'ecdm-mail')}     } .... }    
            Hide
            abayer Andrew Bayer added a comment -

            dan tran Thanks for bringing this to my attention - will dig into it tomorrow.

            Show
            abayer Andrew Bayer added a comment - dan tran Thanks for bringing this to my attention - will dig into it tomorrow.
            Hide
            abayer Andrew Bayer added a comment -

            So this is due to this line here - we're calling getContext().setResult(Result.UNSTABLE), which is setting the result on the CpsFlowExecution but not on the run directly. The run still ends up getting set to UNSTABLE at completion, but not until then. That is, in theory, the right behavior overall - once we eventually get better per-step/stage status granularity, we want to avoid setting the Run result directly. But Declarative's post doesn't currently look at the CpsFlowExecution's result, just the Run result.

            What I'm thinking right now is updating Declarative to look at both the CpsFlowExecution result (which doesn't get set if you just set the Run result directly) and the Run result (which doesn't get set if you just set the CpsFlowExecution result directly) - that'd let us leave junit doing the forward-looking Right Thing while fixing this regression.

            Show
            abayer Andrew Bayer added a comment - So this is due to this line here - we're calling getContext().setResult(Result.UNSTABLE) , which is setting the result on the CpsFlowExecution but not on the run directly. The run still ends up getting set to UNSTABLE at completion, but not until then. That is, in theory, the right behavior overall - once we eventually get better per-step/stage status granularity, we want to avoid setting the Run result directly. But Declarative's post doesn't currently look at the CpsFlowExecution 's result, just the Run result. What I'm thinking right now is updating Declarative to look at both the CpsFlowExecution result (which doesn't get set if you just set the Run result directly) and the Run result (which doesn't get set if you just set the CpsFlowExecution result directly) - that'd let us leave junit doing the forward-looking Right Thing while fixing this regression.
            Hide
            abayer Andrew Bayer added a comment -

            Ok, PRs up for Declarative (actually fixing the post behavior) and junit (verifying the Declarative fix).

            Show
            abayer Andrew Bayer added a comment - Ok, PRs up for Declarative (actually fixing the post behavior) and junit (verifying the Declarative fix).
            Hide
            abayer Andrew Bayer added a comment -

            Releasing shortly in Declarative 1.2.5

            Show
            abayer Andrew Bayer added a comment - Releasing shortly in Declarative 1.2.5
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pipeline-model-definition/pom.xml
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Aborted.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Changed.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Failure.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/NotBuilt.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Success.groovy
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Unstable.groovy
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BuildConditionResponderTest.java
            pipeline-model-definition/src/test/resources/contextResultOverridesRunResult.groovy
            pipeline-model-definition/src/test/resources/junitResult.xml
            pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/model/BuildCondition.java
            pom.xml
            http://jenkins-ci.org/commit/pipeline-model-definition-plugin/d533dd551dadca3d278f3c877aead9016d514b8c
            Log:
            [FIXED JENKINS-37663] Check execution result in post as well

            The new junit step doesn't set the Run to UNSTABLE directly, it sets
            the context (and therefore the execution) result to UNSTABLE. We
            should be checking the execution result as well as the Run result.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/pom.xml pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Aborted.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Changed.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Failure.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/NotBuilt.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Success.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/conditions/Unstable.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BuildConditionResponderTest.java pipeline-model-definition/src/test/resources/contextResultOverridesRunResult.groovy pipeline-model-definition/src/test/resources/junitResult.xml pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/model/BuildCondition.java pom.xml http://jenkins-ci.org/commit/pipeline-model-definition-plugin/d533dd551dadca3d278f3c877aead9016d514b8c Log: [FIXED JENKINS-37663] Check execution result in post as well The new junit step doesn't set the Run to UNSTABLE directly, it sets the context (and therefore the execution) result to UNSTABLE. We should be checking the execution result as well as the Run result.
            Hide
            therealwaldo Will Freeman added a comment - - edited

            Some time back, declarative pipelines were marking a stage as unstable, and I was able to have something like:

            pipeline {
              stage {
                steps {
                  sh 'runuittests'
                }
                post {
                  always {
                    junit 'junit.xml'
                  }
                }
              }
              stage {
                when {
                  expression {
                    currentBuild.resultIsBetterOrEqualTo('SUCCESS')
                  }
                }
                steps {
                  // Only runs when tests are stable
                }
              }
            }

            However, this stopped working at some point in the recent past.  I'm not sure what the cause was, but was hoping this ticket would resolve the issue.

            I have applied the latest update, but this is still not working.

            Andrew Bayer can you suggest how one might prevent, in a declarative manner or with an expression, a build stage from executing if the build is marked unstable in a previous stage?  Or is there still a bug present?

            Show
            therealwaldo Will Freeman added a comment - - edited Some time back, declarative pipelines were marking a stage as unstable, and I was able to have something like: pipeline { stage { steps { sh 'runuittests' } post { always { junit 'junit.xml' } } } stage { when { expression { currentBuild.resultIsBetterOrEqualTo( 'SUCCESS' ) } } steps { // Only runs when tests are stable } } } However, this stopped working at some point in the recent past.  I'm not sure what the cause was, but was hoping this ticket would resolve the issue. I have applied the latest update, but this is still not working. Andrew Bayer can you suggest how one might prevent, in a declarative manner or with an expression, a build stage from executing if the build is marked unstable in a previous stage?  Or is there still a bug present?
            Hide
            abayer Andrew Bayer added a comment - - edited

            Will Freeman Aaaaah. I thought of that and then forgot it because argh. Could you open another ticket against the workflow-cps-plugin component for currentBuild.result not picking up the junit status? Thanks.

            Show
            abayer Andrew Bayer added a comment - - edited Will Freeman Aaaaah. I thought of that and then forgot it because argh. Could you open another ticket against the workflow-cps-plugin component for currentBuild.result not picking up the junit status? Thanks.
            Hide
            therealwaldo Will Freeman added a comment -
            Show
            therealwaldo Will Freeman added a comment - Andrew Bayer filed as JENKINS-48178

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                joshma Joshua Ma
              • Votes:
                7 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: