Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46354

Retry in Declarative options skips retried stages due to earlier failure

    Details

    • Sprint:
      Declarative - 1.2, Pipeline - April 2018
    • Similar Issues:

      Description

      When running the following Pipeline...

      // Keeping the run count
      int runCount = 0
      
      // Main pipeline
      pipeline {
      
          agent any
      
          options {
              retry(3)
          }
      
          stages {
      
              stage('Init') {
                  steps {
                      echo "Init stage"
      
                      script {
                          // Increment the run count
                          runCount++
                      }
      
                      echo "runCount is ${runCount}"
                  }
              }
      
              stage("Foo") {
      
                  steps {
      
                      echo "Stage Foo executing"
      
                      script {
                          if (runCount < 2) {
                              error "Failing - retry me!"
                          } else {
                              echo "Stage Foo will not fail..."
                          }
                      }
                  }
              }
          }
      }
      

      ...the result is that the second, third, etc retries of the Init and Foo stages get skipped "due to earlier failure(s)". That's our standard logic for ensuring that we execute every stage even if there was an earlier failure, to make sure the stage graph is consistent. So we need to do something to ensure that we actually run retried stages regardless.

        Attachments

          Issue Links

            Activity

            Hide
            mkj Michal Matyjek added a comment -

            I have this pipeline:

            pipeline {
                agent any
            
                options {
                    timestamps()
                    buildDiscarder(logRotator(daysToKeepStr:'7'))
                    disableConcurrentBuilds()
                    retry(3)
                }
            
                stages {
                    stage('reset demo') {
                        steps {
                            sh 'exit 0'
                        }
                    }
                    stage('Run tests') {
                        steps {
                            sh 'if (( RANDOM % 2 )); then exit 0; else exit 1; fi;'
                        }
                    }
                }
            
                post {
                    failure {
                        slackSend color: "danger", channel: "slack", message: "FAILED"
                    }
                    success {
                        slackSend color: "good", channel: "slack", message: "PASS"
                        cleanWs()
                    }
                    unstable {
                        slackSend color: "warning", channel: "slack", message: "UNSTABLE"
                    }
                    unsuccessful {
                        cleanWs()
                    }
                }
            }
            

            and seeing this 2 times (hence my comment above that retry doesn't work):
            ERROR: script returned exit code 1
            Retrying
            [Pipeline] {
            [Pipeline] stage
            [Pipeline]

            { (reset demo) Stage "reset demo" skipped due to earlier failure(s) [Pipeline] }

            [Pipeline] // stage
            [Pipeline] stage
            [Pipeline] { (Run tests)
            Stage "Run tests" skipped due to earlier failure(s)

            My understanding of retry is that both stages should be rerun....

            Also a side note - I am getting post-failure step ran on each retry... which is unexpected - I would expect it to run once only when entire pipeline (including retries) is done....

            Show
            mkj Michal Matyjek added a comment - I have this pipeline: pipeline { agent any options { timestamps() buildDiscarder(logRotator(daysToKeepStr: '7' )) disableConcurrentBuilds() retry(3) } stages { stage( 'reset demo' ) { steps { sh 'exit 0' } } stage( 'Run tests' ) { steps { sh ' if (( RANDOM % 2 )); then exit 0; else exit 1; fi;' } } } post { failure { slackSend color: "danger" , channel: "slack" , message: "FAILED" } success { slackSend color: "good" , channel: "slack" , message: "PASS" cleanWs() } unstable { slackSend color: "warning" , channel: "slack" , message: "UNSTABLE" } unsuccessful { cleanWs() } } } and seeing this 2 times (hence my comment above that retry doesn't work): ERROR: script returned exit code 1 Retrying [Pipeline] { [Pipeline] stage [Pipeline] { (reset demo) Stage "reset demo" skipped due to earlier failure(s) [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Run tests) Stage "Run tests" skipped due to earlier failure(s) My understanding of retry is that both stages should be rerun.... Also a side note - I am getting post-failure step ran on each retry... which is unexpected - I would expect it to run once only when entire pipeline (including retries) is done....
            Hide
            atharvai Atharva Inamdar added a comment -

            Andrew, thanks for working on this. Is there a workaround available until this issue is fixed? Currently, we have to manually restart the build which isn't great.

            Show
            atharvai Atharva Inamdar added a comment - Andrew, thanks for working on this. Is there a workaround available until this issue is fixed? Currently, we have to manually restart the build which isn't great.
            Hide
            abayer Andrew Bayer added a comment -
            Show
            abayer Andrew Bayer added a comment - Ok, real PR up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/324 - sorry for the delay!
            Hide
            abayer Andrew Bayer added a comment -

            Releasing as Declarative 1.3.8.

            Show
            abayer Andrew Bayer added a comment - Releasing as Declarative 1.3.8.
            Hide
            bitwiseman Liam Newman added a comment -

            Bulk closing resolved issues.

            Show
            bitwiseman Liam Newman added a comment - Bulk closing resolved issues.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                abayer Andrew Bayer
              • Votes:
                6 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: