Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25387

JGit in git client should support reference repositories

    Details

    • Similar Issues:

      Description

      Right now, the JGit implementation does not support reference repositories, because the clone and init commands do not support it directly. However, by using the repository builder directly, alternate directories can be set.

      I will provide a pull request for this.

        Attachments

          Activity

          Hide
          paux Stephan Pauxberger added a comment - - edited

          Created pull request 151

          Show
          paux Stephan Pauxberger added a comment - - edited Created pull request 151
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/hudson/plugins/git/Branch.java
          src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
          src/main/java/org/jenkinsci/plugins/gitclient/CloneCommand.java
          src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
          src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
          http://jenkins-ci.org/commit/git-client-plugin/f330cd9e4dc1301830b432e26cd313ca7b812aaf
          Log:
          Merge pull request #151 from pauxus/jgit-references

          JENKINS-25387 Allow reference repositories for JGit as well.

          Compare: https://github.com/jenkinsci/git-client-plugin/compare/6296a9a6fcd3...f330cd9e4dc1

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/Branch.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/CloneCommand.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java http://jenkins-ci.org/commit/git-client-plugin/f330cd9e4dc1301830b432e26cd313ca7b812aaf Log: Merge pull request #151 from pauxus/jgit-references JENKINS-25387 Allow reference repositories for JGit as well. Compare: https://github.com/jenkinsci/git-client-plugin/compare/6296a9a6fcd3...f330cd9e4dc1
          Hide
          markewaite Mark Waite added a comment -

          Fixed in git-client-plugin 1.11.1 released 7 Nov 2014

          Show
          markewaite Mark Waite added a comment - Fixed in git-client-plugin 1.11.1 released 7 Nov 2014
          Hide
          csilvers Craig Silverstein added a comment -

          Stephan, I have a kinda-crazy feature request: would it be possible to augment this functionality to also use reference repos for submodules? In our use case, we have a submodule that is as large as our 'main' module.

          git submodule update supports the `--reference` flag, so I think this would be replacing the current `git submodule update --init --recursive` with something like (bash-style psuedocode):

          git submodule init
          refdir=`cat .git/objects/info/alternates`
          for repo in `git submodule status | awk '{print $2}'`; do
             if [ -d "$refdir/../modules/$repo/objects ]; then
                git submodule update --reference $refdir/$repo --recursive -- $repo
             else
                git submodule update --recursive -- $repo
             fi
          done
          

          (Though I admit I'm confused by how git references play around with submodules, so maybe the above won't work. It seemed to in my testing, at least.)

          Thanks!
          craig

          Show
          csilvers Craig Silverstein added a comment - Stephan, I have a kinda-crazy feature request: would it be possible to augment this functionality to also use reference repos for submodules? In our use case, we have a submodule that is as large as our 'main' module. git submodule update supports the `--reference` flag, so I think this would be replacing the current `git submodule update --init --recursive` with something like (bash-style psuedocode): git submodule init refdir=`cat .git/objects/info/alternates` for repo in `git submodule status | awk '{print $2}' `; do if [ -d "$refdir/../modules/$repo/objects ]; then git submodule update --reference $refdir/$repo --recursive -- $repo else git submodule update --recursive -- $repo fi done (Though I admit I'm confused by how git references play around with submodules, so maybe the above won't work. It seemed to in my testing, at least.) Thanks! craig
          Hide
          markewaite Mark Waite added a comment -

          I don't think the code change which provided the reference implementation for JGit can be readily reused or reworked to support reference repositories for submodules.

          I think the core challenge is already indicated in your bash-style pseudocode. As far as I can tell, you assume the reference repository for each submodule must be at $refdir/$repo. My assumption is that is not general enough compared to the needs of the 50,000+ installations of git.

          I think adding reference support to submodules includes some of the same challenges as are represented in JENKINS-20941. That request asks for credential support.

          In the case of both credentials and reference repositories, we need to map a repository URL to something, and use that "something" in a later git command. I think git credentials with submodules might use the existing credentials system to attempt to choose a credential for the repository. That won't work all the time, but might be good enough. Reference repositories need to map a submodule repository URL to a local directory which contains the reference repository.

          Another approach would be to try to apply Jesse Glick's pattern of reference repository from the Mercurial plugin. I believe he creates a local clone of the repository on the Jenkins master server, then clones to slaves from the Jenkins master server. We might even want to extend it to clone a reference repository onto the slave node when a job runs.

          All those alternatives have some complications which will make them challenging.

          Show
          markewaite Mark Waite added a comment - I don't think the code change which provided the reference implementation for JGit can be readily reused or reworked to support reference repositories for submodules. I think the core challenge is already indicated in your bash-style pseudocode. As far as I can tell, you assume the reference repository for each submodule must be at $refdir/$repo. My assumption is that is not general enough compared to the needs of the 50,000+ installations of git. I think adding reference support to submodules includes some of the same challenges as are represented in JENKINS-20941 . That request asks for credential support. In the case of both credentials and reference repositories, we need to map a repository URL to something, and use that "something" in a later git command. I think git credentials with submodules might use the existing credentials system to attempt to choose a credential for the repository. That won't work all the time, but might be good enough. Reference repositories need to map a submodule repository URL to a local directory which contains the reference repository. Another approach would be to try to apply Jesse Glick's pattern of reference repository from the Mercurial plugin. I believe he creates a local clone of the repository on the Jenkins master server, then clones to slaves from the Jenkins master server. We might even want to extend it to clone a reference repository onto the slave node when a job runs. All those alternatives have some complications which will make them challenging.
          Hide
          csilvers Craig Silverstein added a comment -

          Mark, thanks for your response. You're right that I am assuming that the repo that we're referencing has the same directory structure as our repo – that is, that the submodules are in the same location in both cases. I think for jenkins, the common case is that you have to clone a repo in a number of different workspaces, and would like them all to --reference a common repo to speed up their fetches. In that case, all repos have the same structure, so using $refdir/$repo will work.

          (Note that $repo in my pseudocode isn't the name of a repo or anything like that, it's the directory that the submodule lives in inside the 'main' repo. I probably misnamed it. If all occurrences of $repo above were renamed to $submodule_dir, would that help? )

          Show
          csilvers Craig Silverstein added a comment - Mark, thanks for your response. You're right that I am assuming that the repo that we're referencing has the same directory structure as our repo – that is, that the submodules are in the same location in both cases. I think for jenkins, the common case is that you have to clone a repo in a number of different workspaces, and would like them all to --reference a common repo to speed up their fetches. In that case, all repos have the same structure, so using $refdir/$repo will work. (Note that $repo in my pseudocode isn't the name of a repo or anything like that, it's the directory that the submodule lives in inside the 'main' repo. I probably misnamed it. If all occurrences of $repo above were renamed to $submodule_dir, would that help? )

            People

            • Assignee:
              ndeloof Nicolas De Loof
              Reporter:
              paux Stephan Pauxberger
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: