Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-30318

Git plugin breaks usage of Git LFS due to lack of Git Clone use

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I'm currently involved in setting up a Jenkins-based build for our project and I've found that the popular Git plugin for Jenkins takes the slightly weird approach of git init+fetch instead of git clone when setting up the repo on agent machines.

      This appears to break Git LFS as all the binary files are still only the reference pointer files.

      The plugin does not currently appear to have any advanced options to force git cloning and there are a number of aged active tickets for the plugin to support it.

      I have raised an issue ticket over at Git LFS to discuss options with regards to this problem, which can be viewed here. It is not an issue for them to resolve, however they may be helpful in providing advice on how to best solve the problem for the plugin that provides minimal impact.

        Attachments

          Issue Links

            Activity

            Hide
            jekeller Jacob Keller added a comment -

            The original reasoning for this hack was due to credentials issues, which should be resolved after the 1.20.0-beta3 release, if I understand correctly. However, the decision to remove the init+fetch was not accepted at the time I suggested it.

            The reasoning for init+fetch was due to needing abillity to use the git credential store inside .git/config which does not exist when you perform a clone. This can be fixed using the GIT_SSH and GIT_ASKPASS programs as I did for fixing submodule credentials issues among other related problems.

            I would promote removal of the git init+fetch combination due to this, as the original reason for this workaround should no longer be valid.

            Show
            jekeller Jacob Keller added a comment - The original reasoning for this hack was due to credentials issues, which should be resolved after the 1.20.0-beta3 release, if I understand correctly. However, the decision to remove the init+fetch was not accepted at the time I suggested it. The reasoning for init+fetch was due to needing abillity to use the git credential store inside .git/config which does not exist when you perform a clone. This can be fixed using the GIT_SSH and GIT_ASKPASS programs as I did for fixing submodule credentials issues among other related problems. I would promote removal of the git init+fetch combination due to this, as the original reason for this workaround should no longer be valid.
            Hide
            alexagranov Alex Agranov added a comment -

            Can we raise the priority of this issue? It is currently not possible to add jobs with Git repos making use of git-lfs strictly via Jenkins UI.

            At best, the current workaround I've found is the following:

            • Via Jenkins Credentials, add a "Username with Password" credential
            • Create new job, making sure to use HTTPS url for repo and not GIT, along with credentials from previous step
            • Attempt first build, it will fail on git-lfs. Make note of `git checkout -f <tag>` that failed.
            • Manually ssh into Jenkins box, cd to job workspace, and re-attempt `git checkout -f <tag>`. When prompted, enter U/P
            • Subsequent builds for this job will run unattended
            Show
            alexagranov Alex Agranov added a comment - Can we raise the priority of this issue? It is currently not possible to add jobs with Git repos making use of git-lfs strictly via Jenkins UI. At best, the current workaround I've found is the following: Via Jenkins Credentials, add a "Username with Password" credential Create new job, making sure to use HTTPS url for repo and not GIT, along with credentials from previous step Attempt first build, it will fail on git-lfs. Make note of `git checkout -f <tag>` that failed. Manually ssh into Jenkins box, cd to job workspace, and re-attempt `git checkout -f <tag>`. When prompted, enter U/P Subsequent builds for this job will run unattended
            Hide
            alexk_motu alex karpinski added a comment -

            I can reproduce the same state Alex Agranov mentioned. But I really don't want to manually initialize a workspace any time I'm working with an LFS repo. I'd really prefer to have an appropriate way to work with LFS in Jenkins.

            Show
            alexk_motu alex karpinski added a comment - I can reproduce the same state Alex Agranov mentioned. But I really don't want to manually initialize a workspace any time I'm working with an LFS repo. I'd really prefer to have an appropriate way to work with LFS in Jenkins.
            Hide
            jekeller Jacob Keller added a comment -

            We can fix Git plugin, but there is an issue. The init+fetch workaround would allow us to avoid an issue with checking out into a non-empty directory (which currently we do a "wipe" of the workspace which breaks multiple-SCM plugin if the order of repositories is bad).

            However, I would propose going ahead with this fix, assuming the maintainer agrees.

            Show
            jekeller Jacob Keller added a comment - We can fix Git plugin, but there is an issue. The init+fetch workaround would allow us to avoid an issue with checking out into a non-empty directory (which currently we do a "wipe" of the workspace which breaks multiple-SCM plugin if the order of repositories is bad). However, I would propose going ahead with this fix, assuming the maintainer agrees.
            Hide
            ndeloof Nicolas De Loof added a comment -

            I'm +1 to use git clone if we have a correct workaround for credentials support.
            That being said, I don't get why git-lfs would not support init + fetch. Is there any special hook during a clone operation that git-lfs rely on to get large file downloaded ?

            Show
            ndeloof Nicolas De Loof added a comment - I'm +1 to use git clone if we have a correct workaround for credentials support. That being said, I don't get why git-lfs would not support init + fetch. Is there any special hook during a clone operation that git-lfs rely on to get large file downloaded ?
            Hide
            alexagranov2 Alex Agranov added a comment -

            I haven't tracked down the exact code that limits to clone/pull but their own man-page refers to their supported functionality: https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-install.1.ronn#L27-L30

            Show
            alexagranov2 Alex Agranov added a comment - I haven't tracked down the exact code that limits to clone/pull but their own man-page refers to their supported functionality: https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-install.1.ronn#L27-L30
            Hide
            strich Scott Richmond added a comment - - edited

            Hey guys. So. The reason this is an issue is because Git LFS uses various Git hooks to run its binary download processes. There hooks don't fire if one uses the granular Git commands that make up a git clone or git pull etc. Its not really a bug, its just Git design.

            If it really is a difficulty in using git clone, that is fine. The other option in this case is to simply explicitly support Git LFS by adding an option in the settings of this plugin such as Enable/Use Git LFS and to run git lfs pull when appropriate.

            Feel free to drop into the Gitter chat room (https://gitter.im/github/git-lfs) for Git LFS, where we have a fairly active community including the key maintainers of the project.

            Show
            strich Scott Richmond added a comment - - edited Hey guys. So. The reason this is an issue is because Git LFS uses various Git hooks to run its binary download processes. There hooks don't fire if one uses the granular Git commands that make up a git clone or git pull etc. Its not really a bug, its just Git design. If it really is a difficulty in using git clone , that is fine. The other option in this case is to simply explicitly support Git LFS by adding an option in the settings of this plugin such as Enable/Use Git LFS and to run git lfs pull when appropriate. Feel free to drop into the Gitter chat room ( https://gitter.im/github/git-lfs ) for Git LFS, where we have a fairly active community including the key maintainers of the project.
            Hide
            amaroo Ankit Maroo added a comment -

            HI, I am also getting stuck with this issue. Hope to see resolution soon. Thanks

            Show
            amaroo Ankit Maroo added a comment - HI, I am also getting stuck with this issue. Hope to see resolution soon. Thanks
            Hide
            leonov Pavel Leonov added a comment -

            Is there any update or workaround?

            Show
            leonov Pavel Leonov added a comment - Is there any update or workaround?
            Hide
            jekeller Jacob Keller added a comment -

            I can take a look tomorrow and see what it would take to use clone instead of fetch+init like we are now. I think with recent changes it might be possible now.

            Show
            jekeller Jacob Keller added a comment - I can take a look tomorrow and see what it would take to use clone instead of fetch+init like we are now. I think with recent changes it might be possible now.
            Hide
            strich Scott Richmond added a comment -

            Actually if you can implement it without clone it would be even better - Git LFS cannot download files in batches when using Git Clone and can be very slow, starting a new http request for each binary file. The more efficient way to implement Git LFS for repo cloning is to run git init/fetch as this plugin does right now, but to also include a `git lfs pull` afterwards. This will allow LFS to batch download binary files and increase the speed of the download by 10.

            Show
            strich Scott Richmond added a comment - Actually if you can implement it without clone it would be even better - Git LFS cannot download files in batches when using Git Clone and can be very slow, starting a new http request for each binary file. The more efficient way to implement Git LFS for repo cloning is to run git init/fetch as this plugin does right now, but to also include a `git lfs pull` afterwards. This will allow LFS to batch download binary files and increase the speed of the download by 10.
            Hide
            jekeller Jacob Keller added a comment -

            The problem is we can't depend on git-lfs, and I'd rather not have a checkbox for it. Also as I am unfamiliar with git-lfs I am not aware of how and what needs to be added everywhere. It would be better if it worked seamless.

            Switching to use git-clone is easier, and likely fixes other possible issues with the init+fetch combo we have now.

            Show
            jekeller Jacob Keller added a comment - The problem is we can't depend on git-lfs, and I'd rather not have a checkbox for it. Also as I am unfamiliar with git-lfs I am not aware of how and what needs to be added everywhere. It would be better if it worked seamless. Switching to use git-clone is easier, and likely fixes other possible issues with the init+fetch combo we have now.
            Hide
            strich Scott Richmond added a comment -

            I understand where you're coming from, but I would strongly suggest otherwise - Without some explicit support for LFS you will be forcing users to run the first clone with no parallelism. I wasn't kidding with how slow it is - A repo with 10k binary files can take around an hour using the default Git clone and under 10 minutes with some explicit LFS command/s.

            There is now a fairly new command in LFS - `git lfs clone` - It will perform the initial clone with parallelized file downloads. It is the recommended way to clone an LFS repo.

            Without some explicit support for LFS I fear that you'll only later get hammered by users to fix the above issue. That said, by moving to `git clone` I suppose at least LFS repos will work in the first place. Though you may want to consider adding a checkbox along the lines of 'Perform Git LFS Clone' that is optional and will simply swap all `git clone` commands with `git lfs clone` and thusly speed up LFS repos.

            Show
            strich Scott Richmond added a comment - I understand where you're coming from, but I would strongly suggest otherwise - Without some explicit support for LFS you will be forcing users to run the first clone with no parallelism. I wasn't kidding with how slow it is - A repo with 10k binary files can take around an hour using the default Git clone and under 10 minutes with some explicit LFS command/s. There is now a fairly new command in LFS - `git lfs clone` - It will perform the initial clone with parallelized file downloads. It is the recommended way to clone an LFS repo. Without some explicit support for LFS I fear that you'll only later get hammered by users to fix the above issue. That said, by moving to `git clone` I suppose at least LFS repos will work in the first place. Though you may want to consider adding a checkbox along the lines of 'Perform Git LFS Clone' that is optional and will simply swap all `git clone` commands with `git lfs clone` and thusly speed up LFS repos.
            Hide
            jekeller Jacob Keller added a comment -

            I would rather focus on just getting things working, and someone who is actively interested in using LFS is free to submit patches that fix the performance issues and they can be reviewed. Unfortunately I do not have the time to do that myself.

            Show
            jekeller Jacob Keller added a comment - I would rather focus on just getting things working, and someone who is actively interested in using LFS is free to submit patches that fix the performance issues and they can be reviewed. Unfortunately I do not have the time to do that myself.
            Hide
            jekeller Jacob Keller added a comment -

            Ok so I did some digging, and it is not feasible to return to a clone from an init+fetch, because of the various enhancements added to the clone command which are not feasible to remove (as that would be removing a feature that people may depend on).

            If one of the people here wishes to take the effort to add explicit git-lfs support, I will be happy to review it, but I don't have time myself to work on that patch.

            Show
            jekeller Jacob Keller added a comment - Ok so I did some digging, and it is not feasible to return to a clone from an init+fetch, because of the various enhancements added to the clone command which are not feasible to remove (as that would be removing a feature that people may depend on). If one of the people here wishes to take the effort to add explicit git-lfs support, I will be happy to review it, but I don't have time myself to work on that patch.
            Hide
            mcsf M Chon added a comment -

            I set up a Mac Mini slave, and installed git-lfs into /usr/local/bin/git
            Then I configured the slave to use /usr/local/bin/git instead of /usr/bin/git
            It seems to work just fine - LFS files are fetched seamlessly.

            Show
            mcsf M Chon added a comment - I set up a Mac Mini slave, and installed git-lfs into /usr/local/bin/git Then I configured the slave to use /usr/local/bin/git instead of /usr/bin/git It seems to work just fine - LFS files are fetched seamlessly.
            Hide
            markewaite Mark Waite added a comment - - edited

            A pre-release of large file support is being tested now. You can download the git client plugin and the git plugin from the lts-with-plugins branch of my docker instance.

            I deeply appreciate comments on the pull request which say "I've installed the pre-release of this plugin and am using it successfully". I also deeply appreciate comments which indicate when a problem has been encountered.

            Show
            markewaite Mark Waite added a comment - - edited A pre-release of large file support is being tested now. You can download the git client plugin and the git plugin from the lts-with-plugins branch of my docker instance. I deeply appreciate comments on the pull request which say "I've installed the pre-release of this plugin and am using it successfully". I also deeply appreciate comments which indicate when a problem has been encountered.
            Hide
            markewaite Mark Waite added a comment -

            Git plugin 3.1.0 released on 4 Mar 2017 now includes support for command line git large file support.

            Show
            markewaite Mark Waite added a comment - Git plugin 3.1.0 released on 4 Mar 2017 now includes support for command line git large file support.
            Hide
            alexagranov Alex Agranov added a comment -

            Just tested, after adding "Git LFS Pull After Checkout" as an Additional Behavior, and works like a charm!

            You. Da. Man.

            Show
            alexagranov Alex Agranov added a comment - Just tested, after adding "Git LFS Pull After Checkout" as an Additional Behavior, and works like a charm! You. Da. Man.
            Hide
            markewaite Mark Waite added a comment -

             Special thanks to Matt Hauck for the initial implementation and to creste for the final implementation and adaptations as needed.

            Show
            markewaite Mark Waite added a comment -  Special thanks to Matt Hauck for the initial implementation and to creste  for the final implementation and adaptations as needed.
            Hide
            matthauck Matt Hauck added a comment -

            Woot! :sparkles:

            Show
            matthauck Matt Hauck added a comment - Woot! :sparkles:
            Hide
            takekecy Jakub Vlk added a comment -

            I am still getting error "stderr: git: 'lfs' is not a git command. See 'git --help'." but just on OS X system. On the Windows I am fine. I am using ssh btw... Any advice?

            Thx, Jakub

            Show
            takekecy Jakub Vlk added a comment - I am still getting error "stderr: git: 'lfs' is not a git command. See 'git --help'." but just on OS X system. On the Windows I am fine. I am using ssh btw... Any advice? Thx, Jakub
            Hide
            markewaite Mark Waite added a comment -

            Jakub Vlk, you probably need to check your PATH settings to assure that git-lfs is installed as a program in the expected location on your MacOS machine.

            Show
            markewaite Mark Waite added a comment - Jakub Vlk , you probably need to check your PATH settings to assure that git-lfs is installed as a program in the expected location on your MacOS machine.
            Hide
            takekecy Jakub Vlk added a comment -

            Yeah I was thinking in the same way. But I can use git lfs from terminal by just writing git lfs. So it should be OK...

            I used echo for my $PATH and I can see this: 

            /opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

            and which git-lfs returns /usr/local/bin/git-lfs so it looks OK for me...

            Maybe there is a problem that git is installed in /usr/bin and git-lfs in /usr/local/bin/git-lfs? No sure though...

            Show
            takekecy Jakub Vlk added a comment - Yeah I was thinking in the same way. But I can use git lfs from terminal by just writing git lfs. So it should be OK... I used echo for my $PATH and I can see this:  /opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin and which git-lfs returns /usr/local/bin/git-lfs so it looks OK for me... Maybe there is a problem that git is installed in /usr/bin and git-lfs in /usr/local/bin/git-lfs? No sure though...
            Hide
            markewaite Mark Waite added a comment -

            Installing git-lfs in /usr/local/bin worked for me on several different Linux computers.

            You say that you used echo for your $PATH, but did you do that from within a Jenkins job? If so, then I have no idea what could be wrong.

            Show
            markewaite Mark Waite added a comment - Installing git-lfs in /usr/local/bin worked for me on several different Linux computers. You say that you used echo for your $PATH, but did you do that from within a Jenkins job? If so, then I have no idea what could be wrong.
            Hide
            takekecy Jakub Vlk added a comment -

            Oooh god, you're right! It was echo from terminal, where I am logged... Thank you very much! I added lfs path to env variable in Jenkins settings and it's working!

            Show
            takekecy Jakub Vlk added a comment - Oooh god, you're right! It was echo from terminal, where I am logged... Thank you very much! I added lfs path to env variable in Jenkins settings and it's working!

              People

              • Assignee:
                markewaite Mark Waite
                Reporter:
                strich Scott Richmond
              • Votes:
                20 Vote for this issue
                Watchers:
                33 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: