Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39194

Honor SCM checkout retry count from CpsScmFlowDefinition

    Details

    • Similar Issues:
    • Released As:
      Pipeline Groovy 2.44

      Description

      Hi Team,

      We have set the value of SCM checkout retry count to 5. This is working well for free style jobs. The same is not working while loading pipeline using "Pipeline script from scm" option.

      This could be a bug. Can you please check into this.

      Thanks!

        Attachments

        1. FreeStyle_Configuration.PNG
          FreeStyle_Configuration.PNG
          48 kB
        2. FreeStyle_Output.PNG
          FreeStyle_Output.PNG
          159 kB
        3. Pipeline_Configuration.PNG
          Pipeline_Configuration.PNG
          38 kB
        4. Pipeline_Output.PNG
          Pipeline_Output.PNG
          117 kB
        5. SCMRetryCount.PNG
          SCMRetryCount.PNG
          13 kB

          Activity

          anudeeplalam Anudeep Lalam created issue -
          anudeeplalam Anudeep Lalam made changes -
          Field Original Value New Value
          Description Hi Team,

          We have set the value of {{SCM checkout retry count}} to 5. This is working well for free style jobs. The same is not working while loading pipeline using "Pipeline script from scm" option.

          This could be a bug. Can you please check into this.

          Thanks!
          Hi Team,

          We have set the value of {color:red}SCM checkout retry count{color} to 5. This is working well for free style jobs. The same is not working while loading pipeline using "Pipeline script from scm" option.

          This could be a bug. Can you please check into this.

          Thanks!
          markewaite Mark Waite made changes -
          Assignee Mark Waite [ markewaite ]
          Hide
          markewaite Mark Waite added a comment -

          Anudeep Lalam can you provide more details, including step by step instructions and specific examples so that others can duplicate the problem you are reporting?

          For example, please provide the pipeline script you're using which is not honoring the retry count, a description of how you detect the failure when the retry count is not honored, a detailed description of how you configured the working freestyle job (or a copy of the job configuration file), and the types of failures which cause the job to retry.

          Show
          markewaite Mark Waite added a comment - Anudeep Lalam can you provide more details, including step by step instructions and specific examples so that others can duplicate the problem you are reporting? For example, please provide the pipeline script you're using which is not honoring the retry count, a description of how you detect the failure when the retry count is not honored, a detailed description of how you configured the working freestyle job (or a copy of the job configuration file), and the types of failures which cause the job to retry.
          anudeeplalam Anudeep Lalam made changes -
          Attachment FreeStyle_Configuration.PNG [ 34470 ]
          Attachment FreeStyle_Output.PNG [ 34471 ]
          Attachment SCMRetryCount.PNG [ 34472 ]
          anudeeplalam Anudeep Lalam made changes -
          Attachment Pipeline_Configuration.PNG [ 34473 ]
          Attachment Pipeline_Output.PNG [ 34474 ]
          Hide
          anudeeplalam Anudeep Lalam added a comment -

          Mark Waite - Thanks for your comment. Here is the scenario.

          In Manage Jenkins -> Configure System, I have put the SCM checkout retry count to 5.

          Now to test the retry, I have created a free style job to clone a Git repository https://github.com/anudeeplalam/JenkinsTest1.git , which does not exist.

          When I build this job, As you can see in the below picture, The retry is happening.(The retry happened 5 times. To keep short I have captured single retry.)

          Now, When I try to use this in a Pipeline Job for loading Jenkinsfile using "Pipeline script from scm", it's not working

          As you can see in the below picture, The pipeline job exited immediately after the first attempt. Ideally it has to retry "5 Times" just like the Free style Job.

          Show
          anudeeplalam Anudeep Lalam added a comment - Mark Waite - Thanks for your comment. Here is the scenario. In Manage Jenkins -> Configure System, I have put the SCM checkout retry count to 5. Now to test the retry, I have created a free style job to clone a Git repository https://github.com/anudeeplalam/JenkinsTest1.git , which does not exist. When I build this job, As you can see in the below picture, The retry is happening.(The retry happened 5 times. To keep short I have captured single retry.) Now, When I try to use this in a Pipeline Job for loading Jenkinsfile using "Pipeline script from scm", it's not working As you can see in the below picture, The pipeline job exited immediately after the first attempt. Ideally it has to retry "5 Times" just like the Free style Job.
          Hide
          anudeeplalam Anudeep Lalam added a comment -

          Mark Waite - Sorry for a very big message. But I thought it would be better to show all the steps I followed so that it will be easy for you to understand what went wrong.

          So, now coming to the point. Am I missing something here. Ideally my expectation is SCM checkout retry count should work for both Freestyle and Pipeline Jobs.

          This is really a blocker for us. We would really appreciate if someone look into this.

          Thanks!

          Show
          anudeeplalam Anudeep Lalam added a comment - Mark Waite - Sorry for a very big message. But I thought it would be better to show all the steps I followed so that it will be easy for you to understand what went wrong. So, now coming to the point. Am I missing something here. Ideally my expectation is SCM checkout retry count should work for both Freestyle and Pipeline Jobs. This is really a blocker for us. We would really appreciate if someone look into this. Thanks!
          Hide
          markewaite Mark Waite added a comment -

          Thanks for the description. That's exactly what is needed.

          I don't expect to investigate it further for at least several months due to other, more common use cases which need investigation and fixes. I wanted enough of a description in the bug report that others might be able to investigate if they encounter the same problem.

          I don't know if the global SCM retry setting that you are using should be honored in pipeline jobs or not. I'm not sure which layer of the Jenkins code implements that retry mechanism. I suspect it is in some component outside the git plugin, since it mentions "SCM checkout retry count" rather than "Git retry count".

          Show
          markewaite Mark Waite added a comment - Thanks for the description. That's exactly what is needed. I don't expect to investigate it further for at least several months due to other, more common use cases which need investigation and fixes. I wanted enough of a description in the bug report that others might be able to investigate if they encounter the same problem. I don't know if the global SCM retry setting that you are using should be honored in pipeline jobs or not. I'm not sure which layer of the Jenkins code implements that retry mechanism. I suspect it is in some component outside the git plugin, since it mentions "SCM checkout retry count" rather than "Git retry count".
          markewaite Mark Waite made changes -
          Component/s git-client-plugin [ 17423 ]
          esuero Ed Suero made changes -
          Rank Ranked higher
          Hide
          jglick Jesse Glick added a comment -

          CpsScmFlowDefinition is merely a convenience. You can always write a CpsFlowDefinition (inline script) that does an explicit checkout wrapped in retry.

          Show
          jglick Jesse Glick added a comment - CpsScmFlowDefinition is merely a convenience. You can always write a CpsFlowDefinition (inline script) that does an explicit checkout wrapped in retry .
          jglick Jesse Glick made changes -
          Summary SCM checkout retry count is not working while loading pipeline using "Pipeline script from scm" option Honor SCM checkout retry count from CpsScmFlowDefinition
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Component/s workflow-cps-plugin [ 21713 ]
          Component/s pipeline [ 21692 ]
          Component/s scm-api-plugin [ 18054 ]
          Priority Critical [ 2 ] Major [ 3 ]
          jglick Jesse Glick made changes -
          Component/s workflow-scm-step-plugin [ 21717 ]
          Component/s workflow-cps-plugin [ 21713 ]
          Hide
          mmitche Matthew Mitchell added a comment -

          Jesse Glick I'll take a look at this.  It's a blocker for us right now.

          Show
          mmitche Matthew Mitchell added a comment - Jesse Glick  I'll take a look at this.  It's a blocker for us right now.
          Hide
          mmitche Matthew Mitchell added a comment -

          My take is that the retry count should be at least honored for the pipeline scm checkout, though not for checkout steps within the pipeline script.

          Show
          mmitche Matthew Mitchell added a comment - My take is that the retry count should be at least honored for the pipeline scm checkout, though not for checkout steps within the pipeline script.
          Hide
          jglick Jesse Glick added a comment -

          Yeah I would say CpsScmFlowDefinition could pay attention to GlobalSCMRetryCountConfiguration. Within a build, you can just use retry as needed.

          Show
          jglick Jesse Glick added a comment - Yeah I would say CpsScmFlowDefinition could pay attention to GlobalSCMRetryCountConfiguration . Within a build, you can just use retry as needed.
          Hide
          mmitche Matthew Mitchell added a comment -

          Jesse Glick This should be straightforward, though I was also contemplating adding the retry count override option (like with freestyle jobs) into the CpsScmFlowDefinition UI.  Thoughts?  It's probably not necessary but it could be useful to someone.

          Show
          mmitche Matthew Mitchell added a comment - Jesse Glick This should be straightforward, though I was also contemplating adding the retry count override option (like with freestyle jobs) into the CpsScmFlowDefinition UI.  Thoughts?  It's probably not necessary but it could be useful to someone.
          Hide
          jglick Jesse Glick added a comment -

          As long as it goes into an f:advanced block.

          Show
          jglick Jesse Glick added a comment - As long as it goes into an f:advanced block.
          mmitche Matthew Mitchell made changes -
          Assignee Matthew Mitchell [ mmitche ]
          Show
          mmitche Matthew Mitchell added a comment - Implemented in  https://github.com/jenkinsci/workflow-cps-plugin/pull/147
          jglick Jesse Glick made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          jglick Jesse Glick made changes -
          Status In Progress [ 3 ] In Review [ 10005 ]
          jglick Jesse Glick made changes -
          Component/s workflow-cps-plugin [ 21713 ]
          Component/s workflow-scm-step-plugin [ 21717 ]
          Hide
          asegretto Anthony Segretto added a comment -

          The retry feature on Pipelines would be extremely helpful for us.  Our SCM is often unreliable where a checkout will time out:

          stderr: fatal: unable to access 'https://tfs-...': Failed to connect to tfs-... port 443: Timed out

           

          We implement retries around the git fetches completed within the pipeline itself, and typically a single retry will do the trick.  Short of using pull/147 I don't see any way for us to work around the issue of failed git fetches within the (Declarative: Checkout SCM) stage.  We are not interested in installing an unofficial fix.

           

          The fix mentioned on this issue would resolve our current most frequent reason of build failures.  Is there an ETA for resolution?

          Show
          asegretto Anthony Segretto added a comment - The retry feature on Pipelines would be extremely helpful for us.  Our SCM is often unreliable where a checkout will time out: stderr: fatal: unable to access 'https://tfs-...': Failed to connect to tfs-... port 443: Timed out   We implement retries around the git fetches completed within the pipeline itself, and typically a single retry will do the trick.  Short of using pull/147 I don't see any way for us to work around the issue of failed git fetches within the (Declarative: Checkout SCM) stage.  We are not interested in installing an unofficial fix.   The fix mentioned on this issue would resolve our current most frequent reason of build failures.  Is there an ETA for resolution?
          Hide
          zmh998 Minghua Zhao added a comment -

          Is there any update for this issue? I saw the pull request was approved but not merged.

          Show
          zmh998 Minghua Zhao added a comment - Is there any update for this issue? I saw the pull request was approved but not merged.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Sam Van Oort
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/d91d14d0323d0944dc5a89fef83c83283c7d13d9
          Log:
          Merge pull request #147 from mmitche/add-retry

          JENKINS-39194 Implement SCM retry count

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/e9b183243542...d91d14d0323d

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/d91d14d0323d0944dc5a89fef83c83283c7d13d9 Log: Merge pull request #147 from mmitche/add-retry JENKINS-39194 Implement SCM retry count Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/e9b183243542...d91d14d0323d
          dnusbaum Devin Nusbaum made changes -
          Status In Review [ 10005 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Released As Pipeline Groovy 2.44

            People

            • Assignee:
              mmitche Matthew Mitchell
              Reporter:
              anudeeplalam Anudeep Lalam
            • Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: