Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32258

Excessive Git --Rev-Parse invocation when using Prune

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      History

      This was originally discussed in the comments of JENKINS-26587 as it was first identified as a regression when testing that issue. As Git Plugin 2.4.1 has how been release w/o resolving this regression, it has escaped and I am opening a separate ticket.

      Description of Problem

      Previously in (and correctly?) when using the "Prune stale remote-tracking branches" behavior, in every build, after the git checkout -f but before the git reset --hard there was a git rev-parse --verify HEAD command. Now in place of the single rev-parse command there is a git rev-parse /remotes/origin/... command for every remote branch that exists.

      Steps to reproduce.

      1. Install a baseline Jenkins Master, which will include the Git Plugin v 2.4.1. (We used the 1.625.3 LTS version.)
      2. Setup credentials so that it can authenticate to a remote repo. (We use Atlasssian Stas.h.
      3. Create a freestyle job to pull from a remote git repo that has multiple (preferably many) branches.
      • branch specifier origin/master
      • Additional Behaviors: Prune stale remote-tracking branches.

      Run the job and observe the output. It will be something similar to the following:

      Building on master in workspace /var/lib/jenkins/workspace/git-test
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url ssh://git@stash:7999/TEST/test.git # timeout=10
      Pruning obsolete local branches
      Fetching upstream changes from ssh://git@stash:7999/TEST/test.git
       > git --version # timeout=10
      using GIT_SSH to set credentials Credential ID: jenkins-stash-ssh
       > git fetch --tags --progress ssh://git@stash:7999/TEST/test.git +refs/heads/*:refs/remotes/origin/* --prune
       > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
       > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
      Checking out Revision 38b70706154d37132bb67b82af7317e2ed85f9aa (refs/remotes/origin/master)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 38b70706154d37132bb67b82af7317e2ed85f9aa
      First time build. Skipping changelog.
       > git branch -a # timeout=10
       > git rev-parse remotes/origin/CMOBL-4832-create-separate-onarrival-api^{commit} # timeout=10
       > git rev-parse remotes/origin/CROSSTAB-5545^{commit} # timeout=10
       > git rev-parse remotes/origin/CROSSTAB-5802^{commit} # timeout=10
       > git rev-parse remotes/origin/CROSSTAB-5803^{commit} # timeout=10
      
      ... several thousand ref-parses later ...
      
       > git rev-parse remotes/origin/webformtomvc-part1^{commit} # timeout=10
       > git rev-parse remotes/origin/wip-csn-6200^{commit} # timeout=10
       > git rev-parse remotes/origin/wip-prod-20347^{commit} # timeout=10
      Finished: SUCCESS

      With 17K+ remote branches in our repository, this added fifteen minutes to builds that previously took three to eight minutes total. Obviously we could stand to clean up some old branches but this behavior is not acceptable.

      Workarounds

      1. Remove the prune stale branches behavior.
      2. Downgrade to Git Plugin 2.4.0

        Attachments

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
          http://jenkins-ci.org/commit/git-client-plugin/3a8d219569652fbe0531f6ffefa441818acaca06
          Log:
          Test for JENKINS-32258

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java http://jenkins-ci.org/commit/git-client-plugin/3a8d219569652fbe0531f6ffefa441818acaca06 Log: Test for JENKINS-32258
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
          http://jenkins-ci.org/commit/git-client-plugin/1a450a5ee993209cc9f27f2a4bfb31bc08a33257
          Log:
          [Fix JENKINS-32258] remove excess rev-parse calls

          The BuildData cleanup change to remove dead branches from BuildData
          needs the list of current branches. Generating the list of current
          branches called rev-parse for each branch. With small branch counts,
          that isn't a huge problem. Large branch counts make it painfully slow.

          One of the problem repositories has 17000 branches.

          Since the BuildData cleanup step is introducing extra work, this may
          still be slower than earlier versions, since larger sized BuildDData
          was accepted prior to git plugin 2.4.1 and git client 1.19.1.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java http://jenkins-ci.org/commit/git-client-plugin/1a450a5ee993209cc9f27f2a4bfb31bc08a33257 Log: [Fix JENKINS-32258] remove excess rev-parse calls The BuildData cleanup change to remove dead branches from BuildData needs the list of current branches. Generating the list of current branches called rev-parse for each branch. With small branch counts, that isn't a huge problem. Large branch counts make it painfully slow. One of the problem repositories has 17000 branches. Since the BuildData cleanup step is introducing extra work, this may still be slower than earlier versions, since larger sized BuildDData was accepted prior to git plugin 2.4.1 and git client 1.19.1.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
          src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
          http://jenkins-ci.org/commit/git-client-plugin/89ab085182b7a4da6ae575fa4e154a3f3bf0add0
          Log:
          Merge pull request #196 from MarkEWaite/master-JENKINS-32258-many-rev-parse-calls

          Fix JENKINS-32258 - many rev-parse calls pruning remote tracking branches

          Compare: https://github.com/jenkinsci/git-client-plugin/compare/3139a94ab5b8...89ab085182b7

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java http://jenkins-ci.org/commit/git-client-plugin/89ab085182b7a4da6ae575fa4e154a3f3bf0add0 Log: Merge pull request #196 from MarkEWaite/master- JENKINS-32258 -many-rev-parse-calls Fix JENKINS-32258 - many rev-parse calls pruning remote tracking branches Compare: https://github.com/jenkinsci/git-client-plugin/compare/3139a94ab5b8...89ab085182b7
          Hide
          markewaite Mark Waite added a comment -

          Fixed in git client plugin 1.19.2. Release to artifactory failed due to credentials error.

          Show
          markewaite Mark Waite added a comment - Fixed in git client plugin 1.19.2. Release to artifactory failed due to credentials error.
          Hide
          kbaltrinic Kenneth Baltrinic added a comment - - edited

          Confirmed 1.19.2 fixes the issue for us. Thank You!

          Show
          kbaltrinic Kenneth Baltrinic added a comment - - edited Confirmed 1.19.2 fixes the issue for us. Thank You!

            People

            • Assignee:
              markewaite Mark Waite
              Reporter:
              kbaltrinic Kenneth Baltrinic
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: