Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45447

Git checkout with many tags is much slower with 3.3.0 release if pre-build merge is used

    Details

    • Similar Issues:

      Description

      We have 62 thousands of tags in our git repo.
      After upgrade of git plugin from 3.1.0 to 3.3.0 checkout of the repo with PreBuildMerge option takes about an hour.

      14:54:44  > git rev-parse HEAD^{commit} # timeout=10
      14:54:45 Seen branch in repository origin/10098_update_header_link_and_my_account_menu
      ...
      ...
      14:54:45 Seen 1,592 remote branches
      14:54:45  > git tag -l # timeout=10
      14:54:46  > git rev-parse refs/tags/test01wxunit/20160617_094816^{commit} # timeout=10
      ...
      ...
      15:49:57  > git rev-parse refs/tags/b2bAATpipeline/tag/2015-03-06_12-08-07^{commit} # timeout=10
      15:49:57 Checking out Revision ab591971099f42107cbd7915120ee1381a206047 (origin/hybris551aws)
      

      It looks like the root cause of the issue is commit 31fedce9c41c9006c886835e03c9fe825d49aba6.

        Attachments

          Issue Links

            Activity

            yehorov Mykhaylo Yehorov created issue -
            markewaite Mark Waite made changes -
            Field Original Value New Value
            Summary Git checkout lasts too much time Git checkout with many tags is much slower with 3.3.0 release
            markewaite Mark Waite made changes -
            Summary Git checkout with many tags is much slower with 3.3.0 release Git checkout with many tags is much slower with 3.3.0 release if pre-build merge is used
            yehorov Mykhaylo Yehorov made changes -
            Description We have 62 thousands of tags in out git repo.
            After upgrade of git plugin from 3.1.0 to 3.3.0 checkout of the repo with PreBuildMerge option takes about an hour.

            {noformat}
            14:54:44 > git rev-parse HEAD^{commit} # timeout=10
            14:54:45 Seen branch in repository origin/10098_update_header_link_and_my_account_menu
            ...
            ...
            14:54:45 Seen 1,592 remote branches
            14:54:45 > git tag -l # timeout=10
            14:54:46 > git rev-parse refs/tags/test01wxunit/20160617_094816^{commit} # timeout=10
            ...
            ...
            15:49:57 > git rev-parse refs/tags/b2bAATpipeline/tag/2015-03-06_12-08-07^{commit} # timeout=10
            15:49:57 Checking out Revision ab591971099f42107cbd7915120ee1381a206047 (origin/hybris551aws)
            {noformat}


            It looks like the root cause of the issue is commit 31fedce9c41c9006c886835e03c9fe825d49aba6.
            We have 62 thousands of tags in our git repo.
            After upgrade of git plugin from 3.1.0 to 3.3.0 checkout of the repo with PreBuildMerge option takes about an hour.

            {noformat}
            14:54:44 > git rev-parse HEAD^{commit} # timeout=10
            14:54:45 Seen branch in repository origin/10098_update_header_link_and_my_account_menu
            ...
            ...
            14:54:45 Seen 1,592 remote branches
            14:54:45 > git tag -l # timeout=10
            14:54:46 > git rev-parse refs/tags/test01wxunit/20160617_094816^{commit} # timeout=10
            ...
            ...
            15:49:57 > git rev-parse refs/tags/b2bAATpipeline/tag/2015-03-06_12-08-07^{commit} # timeout=10
            15:49:57 Checking out Revision ab591971099f42107cbd7915120ee1381a206047 (origin/hybris551aws)
            {noformat}


            It looks like the root cause of the issue is commit 31fedce9c41c9006c886835e03c9fe825d49aba6.
            markewaite Mark Waite made changes -
            Link This issue is duplicated by JENKINS-47169 [ JENKINS-47169 ]
            Hide
            jakob_tsd Jakob Unterwurzacher added a comment - - edited

            The rev-parse thing also breaks building the linux kernel. The reason is interesting: the v2.6.11  tag ( https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tag/?h=v2.6.11 ) does not have a backing commit:

             > git rev-parse refs/tags/v2.6.11^{commit} # timeout=10
            FATAL: Command "git rev-parse refs/tags/v2.6.11^{commit}" returned status code 128:
            stdout: refs/tags/v2.6.11^{commit}
            
            stderr: error: refs/tags/v2.6.11^{commit}: expected commit type, but the object dereferences to tree type
            error: refs/tags/v2.6.11^{commit}: expected commit type, but the object dereferences to tree type
            fatal: ambiguous argument 'refs/tags/v2.6.11^{commit}': unknown revision or path not in the working tree.
            Use '--' to separate paths from revisions, like this:
            'git <command> [<revision>...] -- [<file>...]'
            

            As for the workaround: Downgrading the git plugin fixes the issue. The newest version that I found was working is 3.1.0, which can be downloaded from http://updates.jenkins-ci.org/download/plugins/git/ .

            Show
            jakob_tsd Jakob Unterwurzacher added a comment - - edited The rev-parse thing also breaks building the linux kernel. The reason is interesting: the v2.6.11  tag ( https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tag/?h=v2.6.11  ) does not have a backing commit: > git rev-parse refs/tags/v2.6.11^{commit} # timeout=10 FATAL: Command "git rev-parse refs/tags/v2.6.11^{commit}" returned status code 128: stdout: refs/tags/v2.6.11^{commit} stderr: error: refs/tags/v2.6.11^{commit}: expected commit type, but the object dereferences to tree type error: refs/tags/v2.6.11^{commit}: expected commit type, but the object dereferences to tree type fatal: ambiguous argument 'refs/tags/v2.6.11^{commit}' : unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this : 'git <command> [<revision>...] -- [<file>...]' As for the workaround: Downgrading the git plugin fixes the issue. The newest version that I found was working is 3.1.0, which can be downloaded from http://updates.jenkins-ci.org/download/plugins/git/  .
            Hide
            markewaite Mark Waite added a comment -

            Jakob Unterwurzacher please file a separate bug report for that issue. It has a very different cause than this bug and is an interesting case that I believe is also noted in the JGit tests. A tag that points to a tree rather than a commit is an interesting tagging case.

            Show
            markewaite Mark Waite added a comment - Jakob Unterwurzacher please file a separate bug report for that issue. It has a very different cause than this bug and is an interesting case that I believe is also noted in the JGit tests. A tag that points to a tree rather than a commit is an interesting tagging case.
            Hide
            jakob_tsd Jakob Unterwurzacher added a comment -
            Show
            jakob_tsd Jakob Unterwurzacher added a comment - Yes, that's true. Done:  https://issues.jenkins-ci.org/browse/JENKINS-47350
            Hide
            markewaite Mark Waite added a comment -

            I have a duplication case for the problem in the JENKINS-32577-regex-branch-to-build-does-not-ignore-user-exclusion job in my regression test kit. That doesn't help anyone else work on it (private repo), but it shows that I can duplicate it.

            If others want to develop a fix, I can provide the job definition.

            Show
            markewaite Mark Waite added a comment - I have a duplication case for the problem in the JENKINS-32577-regex-branch-to-build-does-not-ignore-user-exclusion job in my regression test kit. That doesn't help anyone else work on it (private repo), but it shows that I can duplicate it. If others want to develop a fix, I can provide the job definition.
            markewaite Mark Waite made changes -
            Comment [ I have a duplication case for the problem in the [JENKINS-32577-regex-branch-to-build-does-not-ignore-user-exclusion|https://github.com/MarkEWaite/docker-private-lfs/blob/lts-with-plugins-add-credentials/ref/jobs/Bugs-Individual/jobs/JENKINS-32577-regex-branch-to-build-does-not-ignore-user-exclusion/config.xml] job in my regression test kit. That doesn't help anyone else work on it (private repo), but it shows that I can duplicate it.

            If others want to develop a fix, I can provide the job definition.
            ]
            markewaite Mark Waite made changes -
            Comment [ I have a duplication case for the problem in the [JENKINS-32577-regex-branch-to-build-does-not-ignore-user-exclusion|https://github.com/MarkEWaite/docker-private-lfs/blob/lts-with-plugins-add-credentials/ref/jobs/Bugs-Individual/jobs/JENKINS-32577-regex-branch-to-build-does-not-ignore-user-exclusion/config.xml] job in my regression test kit. That doesn't help anyone else work on it (private repo), but it shows that I can duplicate it.

            If others want to develop a fix, I can provide the job definition. ]
            Hide
            splatteredbits Aaron Jensen added a comment -

            Can someone please fix this ASAP? We are locked at 3.0.5. Several of the plugins we use depend on later versions of the Git plugin and are now disabled:

            • Static Analysis Utilities
            • Static Analysis Collector Plug-in
            • OWASP Dependency-Check Plugin
            • Checkstyle Plug-in 
            Show
            splatteredbits Aaron Jensen added a comment - Can someone please fix this ASAP? We are locked at 3.0.5. Several of the plugins we use depend on later versions of the Git plugin and are now disabled: Static Analysis Utilities Static Analysis Collector Plug-in OWASP Dependency-Check Plugin Checkstyle Plug-in 
            Hide
            markewaite Mark Waite added a comment -

            Aaron Jensen I'm willing to spend the 20 hours of personal time that I think will be required to fix this, if you and/or your company are willing to commit to contribute 20 hours to help the Jenkins project.

            Refer to "Jenkins needs you" for suggestions and ideas that will help the project. As a specific example, there are many git plugin bugs which haven't yet been confirmed are actual bugs. As part of your 20 hours, you could review the description of bugs, confirm with testing that they are bugs, and note what you learned on the specific bug report.

            If you prefer a view that is more centered on code, consider the suggestions in a "Beginner's Guide to Contributing".

            Let me know if you're willing to commit to those 20 hours and I'll plan to start work on it during my Thanksgiving break.

            As an alternate, you could fork the git plugin, revert the change that is thought to cause the issue, build the plugin, and use your fork of the plugin (assuming that you don't need the use case covered by that change). You'll then need to do that same work each time you upgrade the plugin until the issue is resolved.

            Show
            markewaite Mark Waite added a comment - Aaron Jensen I'm willing to spend the 20 hours of personal time that I think will be required to fix this, if you and/or your company are willing to commit to contribute 20 hours to help the Jenkins project. Refer to " Jenkins needs you " for suggestions and ideas that will help the project. As a specific example, there are many git plugin bugs which haven't yet been confirmed are actual bugs. As part of your 20 hours, you could review the description of bugs, confirm with testing that they are bugs, and note what you learned on the specific bug report. If you prefer a view that is more centered on code, consider the suggestions in a " Beginner's Guide to Contributing ". Let me know if you're willing to commit to those 20 hours and I'll plan to start work on it during my Thanksgiving break. As an alternate, you could fork the git plugin, revert the change that is thought to cause the issue, build the plugin, and use your fork of the plugin (assuming that you don't need the use case covered by that change). You'll then need to do that same work each time you upgrade the plugin until the issue is resolved.
            markewaite Mark Waite made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            markewaite Mark Waite added a comment - - edited

            I've started exploring the issue. No estimate on a fix date, but the exploration has started, I've seen the problem in a repeatable fashion, and some ideas are implemented which may resolve the problem without reverting the behavior.

            If the ideas are correct, then new additions will be made to the git client plugin API and those new additions will be consumed by the git plugin. The current implementation ignores the correct location for those API's and rather places them directly inside the git plugin. That's not where they belong, but it allows evaluation of the ideas and compatibility testing.

            I have written several unit tests of the original code, and have run those unit tests with the new code. I've confirmed that one of the calls which was a problem before is significantly faster now.

            Interactive testing of my change shows improvement in a fresh install, but does not show an improvement when I'm upgrading an existing job. More diagnosis and debugging will be needed before it is ready for anyone else to test.

            Show
            markewaite Mark Waite added a comment - - edited I've started exploring the issue . No estimate on a fix date, but the exploration has started, I've seen the problem in a repeatable fashion, and some ideas are implemented which may resolve the problem without reverting the behavior. If the ideas are correct, then new additions will be made to the git client plugin API and those new additions will be consumed by the git plugin. The current implementation ignores the correct location for those API's and rather places them directly inside the git plugin. That's not where they belong, but it allows evaluation of the ideas and compatibility testing. I have written several unit tests of the original code, and have run those unit tests with the new code. I've confirmed that one of the calls which was a problem before is significantly faster now. Interactive testing of my change shows improvement in a fresh install, but does not show an improvement when I'm upgrading an existing job. More diagnosis and debugging will be needed before it is ready for anyone else to test.
            Hide
            markewaite Mark Waite added a comment - - edited

            Pull requests have been submitted for the git client plugin and the git plugin to resolve this bug without removing the functionality added in 31fedc.

            Please help with the evaluation of those pull requests by:

            1. Install pre-release git-client-plugin on your Jenkins server
            2. Install pre-release git plugin on your Jenkins server
            3. Restart the Jenkins server to complete the plugin installation
            4. Poll with a freestyle job which uses "advanced branch specifications" (for example, a branch name with a wildcard character)
            5. Review the polling log of freestyle jobs which use "advanced branch specifications", checking for calls to git rev-parse.  There should be few or none.  Polling should be significantly faster than before this change
            6. Build a Review the build log of freestyle jobs which use "advanced branch specifications", checking for calls to git rev-parse.  There should be few or none.  Checkout should be significantly faster than before this change

            If you're comfortable with Java, it would also help to review the pull requests (git client plugin and git plugin).

             

            Show
            markewaite Mark Waite added a comment - - edited Pull requests have been submitted for the git client plugin and the git plugin to resolve this bug without removing the functionality added in 31fedc . Please help with the evaluation of those pull requests by: Install pre-release git-client-plugin  on your Jenkins server Install pre-release git plugin  on your Jenkins server Restart the Jenkins server to complete the plugin installation Poll with a freestyle job which uses "advanced branch specifications" (for example, a branch name with a wildcard character) Review the polling log of freestyle jobs which use "advanced branch specifications", checking for calls to git rev-parse .  There should be few or none.  Polling should be significantly faster than before this change Build a Review the build log of freestyle jobs which use "advanced branch specifications", checking for calls to  git rev-parse .  There should be few or none.  Checkout should be significantly faster than before this change If you're comfortable with Java, it would also help to review the pull requests ( git client plugin and git plugin ).  
            markewaite Mark Waite made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            markewaite Mark Waite added a comment - - edited

            Aaron Jensen and Mykhaylo Yehorov are you able to test the git client plugin and git plugin pull requests that I have created? 

            They are working as expected for me.  I'd like someone directly impacted by the problem to confirm it is resolved in their case, in addition to my verification.

            Show
            markewaite Mark Waite added a comment - - edited Aaron Jensen  and Mykhaylo Yehorov are you able to test the git client plugin and git plugin pull requests that I have created?  They are working as expected for me.  I'd like someone directly impacted by the problem to confirm it is resolved in their case, in addition to my verification.
            Hide
            splatteredbits Aaron Jensen added a comment -

            Mark Waite I'm hoping to get this installed and tested sometime this week.

            Show
            splatteredbits Aaron Jensen added a comment - Mark Waite I'm hoping to get this installed and tested sometime this week.
            Hide
            sgabriel Samuel Gabriel added a comment -

            I can verify this fixed the issue for us. We actually had edited the git plugin to remove that commit. Glad it can be fixed in the main branch. 

            Show
            sgabriel Samuel Gabriel added a comment - I can verify this fixed the issue for us. We actually had edited the git plugin to remove that commit. Glad it can be fixed in the main branch. 
            Hide
            markewaite Mark Waite added a comment -

            Samuel Gabriel thanks very much!  I'm still looking forward to verification from at least one other user, and to a code review.  If code reviewers are too busy, I'll likely merge it early next week for inclusion in a release.  We have some other interesting fixes that seem to be approaching "done" that would be worth releasing a new plugin version.

            Show
            markewaite Mark Waite added a comment - Samuel Gabriel thanks very much!  I'm still looking forward to verification from at least one other user, and to a code review.  If code reviewers are too busy, I'll likely merge it early next week for inclusion in a release.  We have some other interesting fixes that seem to be approaching "done" that would be worth releasing a new plugin version.
            Hide
            froque Filipe Roque added a comment -

            I have tested this and everything seems OK

            Show
            froque Filipe Roque added a comment - I have tested this and everything seems OK
            Hide
            yehorov Mykhaylo Yehorov added a comment -

            I have tested pre-release plugins.
            Number of git rev-parse calls decreased from 71665 to 5.

            Show
            yehorov Mykhaylo Yehorov added a comment - I have tested pre-release plugins. Number of git rev-parse calls decreased from 71665 to 5.
            Hide
            splatteredbits Aaron Jensen added a comment -

            I have tested the pre-release plugins. It fixes the issue for us. (Dozens of builds polling every minute.)

            Show
            splatteredbits Aaron Jensen added a comment - I have tested the pre-release plugins. It fixes the issue for us. (Dozens of builds polling every minute.)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            pom.xml
            src/main/java/hudson/plugins/git/Branch.java
            src/main/java/hudson/plugins/git/GitObject.java
            src/main/java/hudson/plugins/git/Tag.java
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
            src/test/java/hudson/plugins/git/BranchTest.java
            src/test/java/hudson/plugins/git/GitObjectTest.java
            src/test/java/hudson/plugins/git/TagTest.java
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            http://jenkins-ci.org/commit/git-client-plugin/4dfc9368a7ce65bbb91294c22a0cd439fa7f77ea
            Log:
            Merge pull request #288 from MarkEWaite/add-getTags

            JENKINS-45447 Add GitClient.getTags()

            Compare: https://github.com/jenkinsci/git-client-plugin/compare/5526a61878c1...4dfc9368a7ce

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: pom.xml src/main/java/hudson/plugins/git/Branch.java src/main/java/hudson/plugins/git/GitObject.java src/main/java/hudson/plugins/git/Tag.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java src/test/java/hudson/plugins/git/BranchTest.java src/test/java/hudson/plugins/git/GitObjectTest.java src/test/java/hudson/plugins/git/TagTest.java src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java http://jenkins-ci.org/commit/git-client-plugin/4dfc9368a7ce65bbb91294c22a0cd439fa7f77ea Log: Merge pull request #288 from MarkEWaite/add-getTags JENKINS-45447 Add GitClient.getTags() Compare: https://github.com/jenkinsci/git-client-plugin/compare/5526a61878c1...4dfc9368a7ce
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/hudson/plugins/git/util/GitUtils.java
            http://jenkins-ci.org/commit/git-plugin/18bc3dc8ab0fa590725f09cd31443e0aee298bbf
            Log:
            JENKINS-45447 Improve poll and checkout performance for repos with many tags

            JENKINS-45447 reports that checkout of a repository with many tags
            from a freestyle job using a wildcard in the branch name is
            dramatically slower than earlier releases. Many other cases which use
            the advanced branch selection mechanism show the same performance
            problem.

            31fedce9c added tag checks to the evaluation loop for branch
            names. Unfortunately, tag evaluation needs both the tag name and the
            SHA1 of the commit identified by the tag. The original; implementation
            called revParse() to compute that SHA1 for each tag. With many tags in
            the repository (bug report example was 60,000 tags), the checkout time
            increased dramatically.

            I found that even the number of tags in the git plugin repository
            could add as much as 5 seconds for the computation of SHA1 hashes.
            Those computed SHA1 hashes were then immediately discarded because
            they did not satisfy the branch name selection criteria.

            Calling GitClient.getTags() returns more information than
            GitClient.getTagNames(), so it is possible that this change is still
            not fast enough.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/util/GitUtils.java http://jenkins-ci.org/commit/git-plugin/18bc3dc8ab0fa590725f09cd31443e0aee298bbf Log: JENKINS-45447 Improve poll and checkout performance for repos with many tags JENKINS-45447 reports that checkout of a repository with many tags from a freestyle job using a wildcard in the branch name is dramatically slower than earlier releases. Many other cases which use the advanced branch selection mechanism show the same performance problem. 31fedce9c added tag checks to the evaluation loop for branch names. Unfortunately, tag evaluation needs both the tag name and the SHA1 of the commit identified by the tag. The original; implementation called revParse() to compute that SHA1 for each tag. With many tags in the repository (bug report example was 60,000 tags), the checkout time increased dramatically. I found that even the number of tags in the git plugin repository could add as much as 5 seconds for the computation of SHA1 hashes. Those computed SHA1 hashes were then immediately discarded because they did not satisfy the branch name selection criteria. Calling GitClient.getTags() returns more information than GitClient.getTagNames(), so it is possible that this change is still not fast enough.
            Hide
            markewaite Mark Waite added a comment - - edited

            Fixed in git client plugin 2.7.0 and git plugin 3.7.0, released 21 Dec 2017.

            Show
            markewaite Mark Waite added a comment - - edited Fixed in git client plugin 2.7.0 and git plugin 3.7.0, released 21 Dec 2017.
            markewaite Mark Waite made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            markewaite Mark Waite made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              • Assignee:
                markewaite Mark Waite
                Reporter:
                yehorov Mykhaylo Yehorov
              • Votes:
                6 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: