Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45447

Git checkout with many tags is much slower with 3.3.0 release if pre-build merge is used

    Details

    • Similar Issues:

      Description

      We have 62 thousands of tags in our git repo.
      After upgrade of git plugin from 3.1.0 to 3.3.0 checkout of the repo with PreBuildMerge option takes about an hour.

      14:54:44  > git rev-parse HEAD^{commit} # timeout=10
      14:54:45 Seen branch in repository origin/10098_update_header_link_and_my_account_menu
      ...
      ...
      14:54:45 Seen 1,592 remote branches
      14:54:45  > git tag -l # timeout=10
      14:54:46  > git rev-parse refs/tags/test01wxunit/20160617_094816^{commit} # timeout=10
      ...
      ...
      15:49:57  > git rev-parse refs/tags/b2bAATpipeline/tag/2015-03-06_12-08-07^{commit} # timeout=10
      15:49:57 Checking out Revision ab591971099f42107cbd7915120ee1381a206047 (origin/hybris551aws)
      

      It looks like the root cause of the issue is commit 31fedce9c41c9006c886835e03c9fe825d49aba6.

        Attachments

          Issue Links

            Activity

            Hide
            yehorov Mykhaylo Yehorov added a comment -

            I have tested pre-release plugins.
            Number of git rev-parse calls decreased from 71665 to 5.

            Show
            yehorov Mykhaylo Yehorov added a comment - I have tested pre-release plugins. Number of git rev-parse calls decreased from 71665 to 5.
            Hide
            splatteredbits Aaron Jensen added a comment -

            I have tested the pre-release plugins. It fixes the issue for us. (Dozens of builds polling every minute.)

            Show
            splatteredbits Aaron Jensen added a comment - I have tested the pre-release plugins. It fixes the issue for us. (Dozens of builds polling every minute.)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            pom.xml
            src/main/java/hudson/plugins/git/Branch.java
            src/main/java/hudson/plugins/git/GitObject.java
            src/main/java/hudson/plugins/git/Tag.java
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
            src/test/java/hudson/plugins/git/BranchTest.java
            src/test/java/hudson/plugins/git/GitObjectTest.java
            src/test/java/hudson/plugins/git/TagTest.java
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            http://jenkins-ci.org/commit/git-client-plugin/4dfc9368a7ce65bbb91294c22a0cd439fa7f77ea
            Log:
            Merge pull request #288 from MarkEWaite/add-getTags

            JENKINS-45447 Add GitClient.getTags()

            Compare: https://github.com/jenkinsci/git-client-plugin/compare/5526a61878c1...4dfc9368a7ce

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: pom.xml src/main/java/hudson/plugins/git/Branch.java src/main/java/hudson/plugins/git/GitObject.java src/main/java/hudson/plugins/git/Tag.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java src/test/java/hudson/plugins/git/BranchTest.java src/test/java/hudson/plugins/git/GitObjectTest.java src/test/java/hudson/plugins/git/TagTest.java src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java http://jenkins-ci.org/commit/git-client-plugin/4dfc9368a7ce65bbb91294c22a0cd439fa7f77ea Log: Merge pull request #288 from MarkEWaite/add-getTags JENKINS-45447 Add GitClient.getTags() Compare: https://github.com/jenkinsci/git-client-plugin/compare/5526a61878c1...4dfc9368a7ce
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/hudson/plugins/git/util/GitUtils.java
            http://jenkins-ci.org/commit/git-plugin/18bc3dc8ab0fa590725f09cd31443e0aee298bbf
            Log:
            JENKINS-45447 Improve poll and checkout performance for repos with many tags

            JENKINS-45447 reports that checkout of a repository with many tags
            from a freestyle job using a wildcard in the branch name is
            dramatically slower than earlier releases. Many other cases which use
            the advanced branch selection mechanism show the same performance
            problem.

            31fedce9c added tag checks to the evaluation loop for branch
            names. Unfortunately, tag evaluation needs both the tag name and the
            SHA1 of the commit identified by the tag. The original; implementation
            called revParse() to compute that SHA1 for each tag. With many tags in
            the repository (bug report example was 60,000 tags), the checkout time
            increased dramatically.

            I found that even the number of tags in the git plugin repository
            could add as much as 5 seconds for the computation of SHA1 hashes.
            Those computed SHA1 hashes were then immediately discarded because
            they did not satisfy the branch name selection criteria.

            Calling GitClient.getTags() returns more information than
            GitClient.getTagNames(), so it is possible that this change is still
            not fast enough.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/util/GitUtils.java http://jenkins-ci.org/commit/git-plugin/18bc3dc8ab0fa590725f09cd31443e0aee298bbf Log: JENKINS-45447 Improve poll and checkout performance for repos with many tags JENKINS-45447 reports that checkout of a repository with many tags from a freestyle job using a wildcard in the branch name is dramatically slower than earlier releases. Many other cases which use the advanced branch selection mechanism show the same performance problem. 31fedce9c added tag checks to the evaluation loop for branch names. Unfortunately, tag evaluation needs both the tag name and the SHA1 of the commit identified by the tag. The original; implementation called revParse() to compute that SHA1 for each tag. With many tags in the repository (bug report example was 60,000 tags), the checkout time increased dramatically. I found that even the number of tags in the git plugin repository could add as much as 5 seconds for the computation of SHA1 hashes. Those computed SHA1 hashes were then immediately discarded because they did not satisfy the branch name selection criteria. Calling GitClient.getTags() returns more information than GitClient.getTagNames(), so it is possible that this change is still not fast enough.
            Hide
            markewaite Mark Waite added a comment - - edited

            Fixed in git client plugin 2.7.0 and git plugin 3.7.0, released 21 Dec 2017.

            Show
            markewaite Mark Waite added a comment - - edited Fixed in git client plugin 2.7.0 and git plugin 3.7.0, released 21 Dec 2017.

              People

              • Assignee:
                markewaite Mark Waite
                Reporter:
                yehorov Mykhaylo Yehorov
              • Votes:
                6 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: