Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54612

Reference repo not used for Git clones on Windows SSH agents

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
    • Environment:
      Jenkins 2.138.2
      Git Plugin 3.9.1
      Windows 2012r2
      Cygwin Git 2.17.0
    • Similar Issues:

      Description

      We use a reference repo on all our agents to speed up our clones. This seems to work on all agents we have except for our Windows nodes. I first noticed that the fetch times were slower on the Windows nodes. I then confirmed by checking the size of the .git folder on each machine after the clone. All non-windows nodes were 11MB whereas the .git folder on Windows was 1.9GB.

      Our Windows agents are connected over ssh (Cygwin). The Remote Root is on C:\ (windows style path) and the reference repository is also on C:\. We use the Checkout/SCM pipeline step in order to take advantage of extra options not available using the Git step.

      This is a simplified snippet of what we use I pulled from the Syntax generator.

      checkout([
           $class: 'GitSCM',
           branches: [[name: '$BRANCH']],
           doGenerateSubmoduleConfigurations: false,
           extensions: [[$class: 'CloneOption',
               depth: 0,
               noTags: false,
               reference: '$REFERENCE_REPO',
               shallow: false]],
           gitTool: 'Default',
           submoduleCfg: [],
           userRemoteConfigs: [[url: '$REPO']]])
      

      The console output tries to convince me it's using the reference repo but it appears it isn't (3:20 to fetch).

      00:41:39.279 Cloning the remote Git repository
      00:41:39.355 Cloning repository https://github.com/***/***.git
      00:41:39.368  > git init C:\Users\jenkins\workspace\Build # timeout=10
      00:41:39.574 Using reference repository: C:\repo_cache
      00:41:39.574 Fetching upstream changes from https://github.com/***/***.git
      00:41:39.574  > git --version # timeout=10
      00:41:39.606  > git fetch --tags --progress https://github.com/***/***.git +refs/heads/*:refs/remotes/origin/* # timeout=30
      00:45:00.369  > git config remote.origin.url https://github.com/***/***.git # timeout=10
      00:45:00.431  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
      00:45:00.603  > git config remote.origin.url https://github.com/***/***.git # timeout=10
      00:45:00.775 Fetching upstream changes from https://github.com/***/***.git
      00:45:00.776  > git fetch --tags --progress https://github.com/***/***.git +refs/heads/*:refs/remotes/origin/* # timeout=30
      00:45:02.416  > git rev-parse "origin/master^{commit}" # timeout=10
      00:45:02.448 Checking out Revision *** (origin/master)
      00:45:02.588  > git config core.sparsecheckout # timeout=10
      00:45:02.619  > git checkout -f *** # timeout=30
      00:46:45.479 Commit message: "Merge pull request #63 from ***/***"
      

      I've tried various things like using Windows Git to populate the reference repo, using a Jenkins git step to populate the repo, using cygwin style paths for both the ref repo location and/or the remote root dir for the agent. Not sure what else I can try but I assume its going to be something related to windows vs cygwin vs path styles vs line-endings.

       

      Edit: I can use an sh step to clone using the ref repo successfully.

        Attachments

          Activity

          broussar Adam Brousseau created issue -
          markewaite Mark Waite made changes -
          Field Original Value New Value
          Assignee Mark Waite [ markewaite ]
          broussar Adam Brousseau made changes -
          Description We use a reference repo on all our agents to speed up our clones. This seems to work on all agents we have except for our Windows nodes. I first noticed that the fetch times were slower on the Windows nodes. I then confirmed by checking the size of the .git folder on each machine after the clone. All non-windows nodes were 11MB whereas the .git folder on Windows was 1.9GB.

          Our Windows agents are connected over ssh (Cygwin). The Remote Root is on C:\ (windows style path) and the reference repository is also on C:\. We use the Checkout/SCM pipeline step in order to take advantage of extra options not available using the Git step.

          This is a simplified snippet of what we use I pulled from the Syntax generator.
          {code:none}
          checkout([
               $class: 'GitSCM',
               branches: [[name: '$BRANCH']],
               doGenerateSubmoduleConfigurations: false,
               extensions: [[$class: 'CloneOption',
                   depth: 0,
                   noTags: false,
                   reference: '$REFERENCE_REPO',
                   shallow: false]],
               gitTool: 'Default',
               submoduleCfg: [],
               userRemoteConfigs: [[url: '$REPO']]])
          {code}
          The console output tries to convince me it's using the reference repo but it appears it isn't (3:20 to fetch).
          {code:none}
          00:41:39.279 Cloning the remote Git repository
          00:41:39.355 Cloning repository https://github.com/***/***.git
          00:41:39.368 > git init C:\Users\jenkins\workspace\Build # timeout=10
          00:41:39.574 Using reference repository: C:\repo_cache
          00:41:39.574 Fetching upstream changes from https://github.com/***/***.git
          00:41:39.574 > git --version # timeout=10
          00:41:39.606 > git fetch --tags --progress https://github.com/***/***.git +refs/heads/*:refs/remotes/origin/* # timeout=30
          00:45:00.369 > git config remote.origin.url https://github.com/***/***.git # timeout=10
          00:45:00.431 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
          00:45:00.603 > git config remote.origin.url https://github.com/***/***.git # timeout=10
          00:45:00.775 Fetching upstream changes from https://github.com/***/***.git
          00:45:00.776 > git fetch --tags --progress https://github.com/***/***.git +refs/heads/*:refs/remotes/origin/* # timeout=30
          00:45:02.416 > git rev-parse "origin/master^{commit}" # timeout=10
          00:45:02.448 Checking out Revision *** (origin/master)
          00:45:02.588 > git config core.sparsecheckout # timeout=10
          00:45:02.619 > git checkout -f *** # timeout=30
          00:46:45.479 Commit message: "Merge pull request #63 from ***/***"
          {code}
          I've tried various things like using Windows Git to populate the reference repo, using a Jenkins git step to populate the repo, using cygwin style paths for both the ref repo location and/or the remote root dir for the agent. Not sure what else I can try but I assume its going to be something related to windows vs cygwin vs path styles vs line-endings.
          We use a reference repo on all our agents to speed up our clones. This seems to work on all agents we have except for our Windows nodes. I first noticed that the fetch times were slower on the Windows nodes. I then confirmed by checking the size of the .git folder on each machine after the clone. All non-windows nodes were 11MB whereas the .git folder on Windows was 1.9GB.

          Our Windows agents are connected over ssh (Cygwin). The Remote Root is on C:\ (windows style path) and the reference repository is also on C:\. We use the Checkout/SCM pipeline step in order to take advantage of extra options not available using the Git step.

          This is a simplified snippet of what we use I pulled from the Syntax generator.
          {code:none}
          checkout([
               $class: 'GitSCM',
               branches: [[name: '$BRANCH']],
               doGenerateSubmoduleConfigurations: false,
               extensions: [[$class: 'CloneOption',
                   depth: 0,
                   noTags: false,
                   reference: '$REFERENCE_REPO',
                   shallow: false]],
               gitTool: 'Default',
               submoduleCfg: [],
               userRemoteConfigs: [[url: '$REPO']]])
          {code}
          The console output tries to convince me it's using the reference repo but it appears it isn't (3:20 to fetch).
          {code:none}
          00:41:39.279 Cloning the remote Git repository
          00:41:39.355 Cloning repository https://github.com/***/***.git
          00:41:39.368 > git init C:\Users\jenkins\workspace\Build # timeout=10
          00:41:39.574 Using reference repository: C:\repo_cache
          00:41:39.574 Fetching upstream changes from https://github.com/***/***.git
          00:41:39.574 > git --version # timeout=10
          00:41:39.606 > git fetch --tags --progress https://github.com/***/***.git +refs/heads/*:refs/remotes/origin/* # timeout=30
          00:45:00.369 > git config remote.origin.url https://github.com/***/***.git # timeout=10
          00:45:00.431 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
          00:45:00.603 > git config remote.origin.url https://github.com/***/***.git # timeout=10
          00:45:00.775 Fetching upstream changes from https://github.com/***/***.git
          00:45:00.776 > git fetch --tags --progress https://github.com/***/***.git +refs/heads/*:refs/remotes/origin/* # timeout=30
          00:45:02.416 > git rev-parse "origin/master^{commit}" # timeout=10
          00:45:02.448 Checking out Revision *** (origin/master)
          00:45:02.588 > git config core.sparsecheckout # timeout=10
          00:45:02.619 > git checkout -f *** # timeout=30
          00:46:45.479 Commit message: "Merge pull request #63 from ***/***"
          {code}
          I've tried various things like using Windows Git to populate the reference repo, using a Jenkins git step to populate the repo, using cygwin style paths for both the ref repo location and/or the remote root dir for the agent. Not sure what else I can try but I assume its going to be something related to windows vs cygwin vs path styles vs line-endings.

           

          Edit: I can use an sh step to clone using the ref repo successfully.
          dheeraj0060 dheeraj reddy made changes -
          Priority Minor [ 4 ] Critical [ 2 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              broussar Adam Brousseau
            • Votes:
              5 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: