Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49757

Git plugin calls fetch twice for a single job

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      When performing a git clone operation,

      git-client and git-plugin both triggers a git fetch.

       

      Below are the two lines of code running in the same git clone command both doing fetches.

       

      The second link is part of the GITSCM step, it triggers the git-client-plugin's CloneCommand (first link) to perform the clone operation. And afterwards, the GITSCM step performs a fetch once the CloneCommand Completes.

      However, within the CloneCommand (first link) itself, a fetch command was already done.

       

      https://github.com/jenkinsci/git-client-plugin/blob/4241e818dbe27fb78093094f29ebcd2203166610/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L609

       

      https://github.com/jenkinsci/git-plugin/blob/cb58c74efafafd08bc03bba2787997c6a0272df0/src/main/java/hudson/plugins/git/GitSCM.java#L1153

       

      I am aware there are needs to support for git clone to fetch all refs first -JENKINS-31393-. The suggestion provided in that ticket was to use "honor git refspec", so shouldn't "honor git refspec" skip the first fetch here?

       

      Thanks,

      Sam

        Attachments

          Activity

          Hide
          dogin2006 Kaiyuan Zhao added a comment -

          Mark Waite

          First of all, my job is using Gerrit Trigger.

          When my Jenkinsfile is like

           

          def scm = [
                  $class                           : 'GitSCM',
                  branches                         : [[name: '*/master']],
                  doGenerateSubmoduleConfigurations: false,
                  extensions                       : [
                          [$class: 'WipeWorkspace'],
                          [$class: 'CloneOption', depth: 2, honorRefspec: true, noTags: true, reference: '', shallow: true, timeout: 10],
                          [$class: 'BuildChooserSetting', buildChooser: [$class: 'GerritTriggerBuildChooser']]
                  ],
                  submoduleCfg                     : [],
                  userRemoteConfigs                : [
                          [
                                  credentialsId: 'hide',
                                  refspec      : '$GERRIT_REFSPEC',
                                  url          : 'hide'
                          ]
                  ]
          ]
          
          pipeline {
              agent {
                  node {
                      label 'gerrit'
                  }
              }
              options {
                  skipDefaultCheckout true
              }
              stages {
                  stage('CheckOut') {
                      steps {
                          checkout(scm)
                      }
                  }
              }
          }
          

          The console log

           

           

          Wiping out workspace first.
          Cloning the remote Git repository
          Using shallow clone
          shallow clone depth 2
          Avoid fetching tags
          Honoring refspec on initial clone
          Cloning repository hide
          > git init hide # timeout=10
          Fetching upstream changes from hide
          > git --version # timeout=10
          using GIT_SSH to set credentials
          > git fetch --no-tags --progress hide $GERRIT_REFSPEC --depth=2 # timeout=10
          > git config remote.origin.url hide # timeout=10
          > git config --add remote.origin.fetch $GERRIT_REFSPEC # timeout=10
          > git config remote.origin.url hide # timeout=10
          Fetching upstream changes from hide
          using GIT_SSH to set credentials
          > git fetch --no-tags --progress hide refs/changes/64/4264/7 --depth=2 # timeout=10
          > git rev-parse 4b5226960edd4551bbafc058db4afc0e69bd103f^{commit} # timeout=10
          Checking out Revision 4b5226960edd4551bbafc058db4afc0e69bd103f (refs/changes/64/4264/7)
          > git config core.sparsecheckout # timeout=10
          > git checkout -f 4b5226960edd4551bbafc058db4afc0e69bd103f
          Commit message: "hide"
          > git rev-parse FETCH_HEAD^{commit} # timeout=10
          > git rev-list --no-walk a1323feb6fa57ce13c87a5fd6886b14aa3f75dab # timeout=10
          

          Fetch twice and the first fetch cannot recognize $GERRIT_REFSPEC

          Then I changed my Jenkinsfile like

           

          def scm = [
                  $class                           : 'GitSCM',
                  branches                         : [[name: '*/master']],
                  doGenerateSubmoduleConfigurations: false,
                  extensions                       : [
                          [$class: 'CloneOption', depth: 2, honorRefspec: true, noTags: true, reference: '', shallow: true, timeout: 10],
                          [$class: 'BuildChooserSetting', buildChooser: [$class: 'GerritTriggerBuildChooser']]
                  ],
                  submoduleCfg                     : [],
                  userRemoteConfigs                : [
                          [
                                  credentialsId: 'hide',
                                  refspec      : '$GERRIT_REFSPEC',
                                  url          : 'hide'
                          ]
                  ]
          ]
          
          pipeline {
              agent {
                  node {
                      label 'gerrit'
                  }
              }
              options {
                  skipDefaultCheckout true
              }
              stages {
                  stage('CheckOut') {
                      steps {
                          sh('git init')
                          checkout(scm)
                      }
                  }
              }
              post {
                  cleanup {
                      cleanWs()
                  }
              }
          }
          

          The console log

           

           

          [gerrit_test_2] Running shell script
          + git init
          Initialized empty Git repository in hide
          [Pipeline] checkout
           > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repository
           > git config remote.origin.url hide # timeout=10
          Fetching upstream changes from hide
           > git --version # timeout=10
          using GIT_SSH to set credentials 
           > git fetch --no-tags --progress hide refs/changes/64/4264/7 --depth=2 # timeout=10
           > git rev-parse 4b5226960edd4551bbafc058db4afc0e69bd103f^{commit} # timeout=10
          Checking out Revision 4b5226960edd4551bbafc058db4afc0e69bd103f (refs/changes/64/4264/7)
           > git config core.sparsecheckout # timeout=10
           > git checkout -f 4b5226960edd4551bbafc058db4afc0e69bd103f
          Commit message: "hide"
           > git rev-parse FETCH_HEAD^{commit} # timeout=10
           > git rev-list --no-walk a1323feb6fa57ce13c87a5fd6886b14aa3f75dab # timeout=10
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] stage
          [Pipeline] { (Declarative: Post Actions)
          [Pipeline] cleanWs
          [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          

          Fetch only once

           

           

          Jenkins Version: 2.120

          Gerrit Trigger: 2.27.5

          Git client plugin: 2.7.1

          Git plugin: 3.8.0

           

          Show
          dogin2006 Kaiyuan Zhao added a comment - Mark Waite First of all, my job is using Gerrit Trigger. When my Jenkinsfile is like   def scm = [ $class : 'GitSCM' , branches : [[name: '*/master' ]], doGenerateSubmoduleConfigurations: false , extensions : [ [$class: 'WipeWorkspace' ], [$class: 'CloneOption' , depth: 2, honorRefspec: true , noTags: true , reference: '', shallow: true , timeout: 10], [$class: 'BuildChooserSetting' , buildChooser: [$class: 'GerritTriggerBuildChooser' ]] ], submoduleCfg : [], userRemoteConfigs : [ [ credentialsId: 'hide' , refspec : '$GERRIT_REFSPEC' , url : 'hide' ] ] ] pipeline { agent { node { label 'gerrit' } } options { skipDefaultCheckout true } stages { stage( 'CheckOut' ) { steps { checkout(scm) } } } } The console log     Wiping out workspace first. Cloning the remote Git repository Using shallow clone shallow clone depth 2 Avoid fetching tags Honoring refspec on initial clone Cloning repository hide > git init hide # timeout=10 Fetching upstream changes from hide > git --version # timeout=10 using GIT_SSH to set credentials > git fetch --no-tags --progress hide $GERRIT_REFSPEC --depth=2 # timeout=10 > git config remote.origin.url hide # timeout=10 > git config --add remote.origin.fetch $GERRIT_REFSPEC # timeout=10 > git config remote.origin.url hide # timeout=10 Fetching upstream changes from hide using GIT_SSH to set credentials > git fetch --no-tags --progress hide refs/changes/64/4264/7 --depth=2 # timeout=10 > git rev-parse 4b5226960edd4551bbafc058db4afc0e69bd103f^{commit} # timeout=10 Checking out Revision 4b5226960edd4551bbafc058db4afc0e69bd103f (refs/changes/64/4264/7) > git config core.sparsecheckout # timeout=10 > git checkout -f 4b5226960edd4551bbafc058db4afc0e69bd103f Commit message: "hide" > git rev-parse FETCH_HEAD^{commit} # timeout=10 > git rev-list --no-walk a1323feb6fa57ce13c87a5fd6886b14aa3f75dab # timeout=10 Fetch twice and the first fetch cannot recognize $GERRIT_REFSPEC Then I changed my Jenkinsfile like   def scm = [ $class : 'GitSCM' , branches : [[name: '*/master' ]], doGenerateSubmoduleConfigurations: false , extensions : [ [$class: 'CloneOption' , depth: 2, honorRefspec: true , noTags: true , reference: '', shallow: true , timeout: 10], [$class: 'BuildChooserSetting' , buildChooser: [$class: 'GerritTriggerBuildChooser' ]] ], submoduleCfg : [], userRemoteConfigs : [ [ credentialsId: 'hide' , refspec : '$GERRIT_REFSPEC' , url : 'hide' ] ] ] pipeline { agent { node { label 'gerrit' } } options { skipDefaultCheckout true } stages { stage( 'CheckOut' ) { steps { sh( 'git init' ) checkout(scm) } } } post { cleanup { cleanWs() } } } The console log     [gerrit_test_2] Running shell script + git init Initialized empty Git repository in hide [Pipeline] checkout > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url hide # timeout=10 Fetching upstream changes from hide > git --version # timeout=10 using GIT_SSH to set credentials > git fetch --no-tags --progress hide refs/changes/64/4264/7 --depth=2 # timeout=10 > git rev-parse 4b5226960edd4551bbafc058db4afc0e69bd103f^{commit} # timeout=10 Checking out Revision 4b5226960edd4551bbafc058db4afc0e69bd103f (refs/changes/64/4264/7) > git config core.sparsecheckout # timeout=10 > git checkout -f 4b5226960edd4551bbafc058db4afc0e69bd103f Commit message: "hide" > git rev-parse FETCH_HEAD^{commit} # timeout=10 > git rev-list --no-walk a1323feb6fa57ce13c87a5fd6886b14aa3f75dab # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] cleanWs [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS Fetch only once     Jenkins Version: 2.120 Gerrit Trigger: 2.27.5 Git client plugin: 2.7.1 Git plugin: 3.8.0  
          Hide
          markewaite Mark Waite added a comment -

          Kaiyuan Zhao since you're using Pipeline, you can expand the value of GERRIT_REFSPEC without relying on the git plugin internal environment variable expansion. Use:

          refspec      : "${GERRIT_REFSPEC}",
          

          instead of

          refspec      : '$GERRIT_REFSPEC',
          

          The bug report is specific to Freestyle jobs because the variable expansion for strings is not available to Freestyle jobs.

          Show
          markewaite Mark Waite added a comment - Kaiyuan Zhao since you're using Pipeline, you can expand the value of GERRIT_REFSPEC without relying on the git plugin internal environment variable expansion. Use: refspec : "${GERRIT_REFSPEC}" , instead of refspec : '$GERRIT_REFSPEC' , The bug report is specific to Freestyle jobs because the variable expansion for strings is not available to Freestyle jobs.
          Hide
          dogin2006 Kaiyuan Zhao added a comment -

          Mark Waite

          Using "${GERRIT_REFSPEC}" fix the first problem.

           

          10:38:45 Cloning the remote Git repository
          10:38:45 Using shallow clone
          10:38:45 shallow clone depth 2
          10:38:45 Avoid fetching tags
          10:38:45 Honoring refspec on initial clone
          10:38:45 Cloning repository hide
          10:38:45 > git init hide # timeout=10
          10:38:45 Fetching upstream changes from hide
          10:38:45 > git --version # timeout=10
          10:38:45 using GIT_SSH to set credentials 
          10:38:45 > git fetch --no-tags --progress hide refs/changes/50/4750/1 --depth=2 # timeout=10
          10:39:34 > git config remote.origin.url hide # timeout=10
          10:39:34 > git config --add remote.origin.fetch refs/changes/50/4750/1 # timeout=10
          10:39:34 > git config remote.origin.url hide # timeout=10
          10:39:34 Fetching upstream changes from hide
          10:39:34 using GIT_SSH to set credentials 
          10:39:34 > git fetch --no-tags --progress hide refs/changes/50/4750/1 --depth=2 # timeout=10
          10:40:34 > git rev-parse 44fd5ca794ef1d4ccbfb437d94ef804626dea726^{commit} # timeout=10
          10:40:34 Checking out Revision 44fd5ca794ef1d4ccbfb437d94ef804626dea726 (refs/changes/50/4750/1)
          10:40:34 > git config core.sparsecheckout # timeout=10
          10:40:34 > git checkout -f 44fd5ca794ef1d4ccbfb437d94ef804626dea726 # timeout=10
          10:40:38 Commit message: "hide"
          10:40:38 > git rev-parse FETCH_HEAD^{commit} # timeout=10
          10:40:38 > git rev-list --no-walk eb696053ba8f4c94c388db0334b886879e6aaa42 # timeout=10
          

          But it's also triggering fetch twice and the second fetch is not cost free.

          In our case, both fetch cost about one minute.

          Show
          dogin2006 Kaiyuan Zhao added a comment - Mark Waite Using "${GERRIT_REFSPEC}" fix the first problem.   10:38:45 Cloning the remote Git repository 10:38:45 Using shallow clone 10:38:45 shallow clone depth 2 10:38:45 Avoid fetching tags 10:38:45 Honoring refspec on initial clone 10:38:45 Cloning repository hide 10:38:45 > git init hide # timeout=10 10:38:45 Fetching upstream changes from hide 10:38:45 > git --version # timeout=10 10:38:45 using GIT_SSH to set credentials 10:38:45 > git fetch --no-tags --progress hide refs/changes/50/4750/1 --depth=2 # timeout=10 10:39:34 > git config remote.origin.url hide # timeout=10 10:39:34 > git config --add remote.origin.fetch refs/changes/50/4750/1 # timeout=10 10:39:34 > git config remote.origin.url hide # timeout=10 10:39:34 Fetching upstream changes from hide 10:39:34 using GIT_SSH to set credentials 10:39:34 > git fetch --no-tags --progress hide refs/changes/50/4750/1 --depth=2 # timeout=10 10:40:34 > git rev-parse 44fd5ca794ef1d4ccbfb437d94ef804626dea726^{commit} # timeout=10 10:40:34 Checking out Revision 44fd5ca794ef1d4ccbfb437d94ef804626dea726 (refs/changes/50/4750/1) 10:40:34 > git config core.sparsecheckout # timeout=10 10:40:34 > git checkout -f 44fd5ca794ef1d4ccbfb437d94ef804626dea726 # timeout=10 10:40:38 Commit message: "hide" 10:40:38 > git rev-parse FETCH_HEAD^{commit} # timeout=10 10:40:38 > git rev-list --no-walk eb696053ba8f4c94c388db0334b886879e6aaa42 # timeout=10 But it's also triggering fetch twice and the second fetch is not cost free. In our case, both fetch cost about one minute.
          Hide
          markewaite Mark Waite added a comment -

          Kaiyuan Zhao you're welcome to propose a pull request to the plugin to improve its behavior. Alternately, you could try replacing the "depth=2" with a reference repository to see if that would reduce the fetch time.

          Show
          markewaite Mark Waite added a comment - Kaiyuan Zhao you're welcome to propose a pull request to the plugin to improve its behavior. Alternately, you could try replacing the "depth=2" with a reference repository to see if that would reduce the fetch time.
          Hide
          oliverp Oliver Pereira added a comment -

          Using a reference repository doesn't fix the issue.

          Using the command line git clone including reference repository, it takes approx 11 seconds and the same configuration via the checkout scm option takes 1 minute 3x seconds.

           

          Show
          oliverp Oliver Pereira added a comment - Using a reference repository doesn't fix the issue. Using the command line git clone including reference repository, it takes approx 11 seconds and the same configuration via the checkout scm option takes 1 minute 3x seconds.  

            People

            • Assignee:
              Unassigned
              Reporter:
              slcy sam leung
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: