Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35687

Add explicit support for git lfs

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I would like to use git lfs and I would like to jenkins, and I would like for jenkins to use its same credentials already configured for git to fetch the git lfs resources. Rather than trying to change the init+fetch pattern, I would like to add a custom behavior to the git plugin that will execute `git lfs pull` after checkout using the same credentials as the fetch command.

      It seems that git lfs is the way forward for git repos with large files and I don't see a workaround right now for using git lfs with jenkins, short of sprinkling credentials everywhere so that `git lfs` can be run from the build context which doesn't seem like a good idea.

      I am looking into what it would take to add this. Would the maintainers be interested in taking in such a feature?

        Attachments

          Issue Links

            Activity

            Hide
            matthauck Matt Hauck added a comment -
            Show
            matthauck Matt Hauck added a comment - Added some simple pull requests to get this going: https://github.com/jenkinsci/git-client-plugin/pull/206 https://github.com/jenkinsci/git-plugin/pull/414
            Hide
            mcsf M Chon added a comment -

            I set up a Mac Mini slave, and installed git-lfs into /usr/local/bin/git
            Then I configured the slave to use /usr/local/bin/git instead of /usr/bin/git
            It seems to work just fine - LFS files are fetched seamlessly.

            Show
            mcsf M Chon added a comment - I set up a Mac Mini slave, and installed git-lfs into /usr/local/bin/git Then I configured the slave to use /usr/local/bin/git instead of /usr/bin/git It seems to work just fine - LFS files are fetched seamlessly.
            Hide
            swashbuck1r Spike Washburn added a comment -

            See the related discussion in the Git LFS issue tracker https://github.com/github/git-lfs/issues/640

            Show
            swashbuck1r Spike Washburn added a comment - See the related discussion in the Git LFS issue tracker https://github.com/github/git-lfs/issues/640
            Hide
            creste3 John Smith added a comment -
            Show
            creste3 John Smith added a comment - I created some pull requests that fixed a few issues with Matt Hauck 's PRs: https://github.com/jenkinsci/git-client-plugin/pull/232 https://github.com/jenkinsci/git-plugin/pull/463
            Hide
            markewaite Mark Waite added a comment - - edited

            I've added an unauthenticated verification job to my lts-with-plugins docker repository. That job passes with git client plugin PR232 and git plugin PR463.

            I've added an authenticated verification job to my private docker repo. That job also passes with git client plugin PR232 and git plugin PR463.

            I've added an authenticated verification job with Jenkins pipeline and it fails to checkout with an error. Refer to the comments on git plugin PR463 for the latest status.

            Show
            markewaite Mark Waite added a comment - - edited I've added an unauthenticated verification job to my lts-with-plugins docker repository. That job passes with git client plugin PR232 and git plugin PR463 . I've added an authenticated verification job to my private docker repo. That job also passes with git client plugin PR232 and git plugin PR463 . I've added an authenticated verification job with Jenkins pipeline and it fails to checkout with an error. Refer to the comments on git plugin PR463 for the latest status.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Matt Hauck
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            http://jenkins-ci.org/commit/git-client-plugin/e7ddf98e290de8c38b05807c579359dd1782f7de
            Log:
            JENKINS-35687 Add simple git lfs support

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Matt Hauck Path: src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java http://jenkins-ci.org/commit/git-client-plugin/e7ddf98e290de8c38b05807c579359dd1782f7de Log: JENKINS-35687 Add simple git lfs support
            Hide
            markewaite Mark Waite added a comment - - edited

            A pre-release of large file support is being tested now. You can download the git client plugin and the git plugin from the lts-with-plugins branch of my docker instance.

            I deeply appreciate comments on the pull request which say "I've installed the pre-release of this plugin and am using it successfully". I also deeply appreciate comments which indicate when a problem has been encountered.

            Show
            markewaite Mark Waite added a comment - - edited A pre-release of large file support is being tested now. You can download the git client plugin and the git plugin from the lts-with-plugins branch of my docker instance. I deeply appreciate comments on the pull request which say "I've installed the pre-release of this plugin and am using it successfully". I also deeply appreciate comments which indicate when a problem has been encountered.
            Hide
            monitorjbl Taylor Jones added a comment -

            Would it be possible to add support for remotes that are not the default LFS remote URL? It would make it a lot easier for teams that are using a separate LFS repo like Artifactory to create Jenkins jobs.

            Show
            monitorjbl Taylor Jones added a comment - Would it be possible to add support for remotes that are not the default LFS remote URL ? It would make it a lot easier for teams that are using a separate LFS repo like Artifactory to create Jenkins jobs.
            Hide
            markewaite Mark Waite added a comment - - edited

            Adding support for remotes that are not the default LFS remote URL will not be in the initial release.

            I don't plan to implement that support, even after the initial release of git LFS because I need to focus on submodule bugs and on authentication bugs once the basic LFS capability has been added.

            I'm willing to evaluate pull requests from others that propose to add such support, but I don't see it high enough on my list to do that myself. There are too many open pull requests with good ideas for me to spend much time writing improvements myself.

            Show
            markewaite Mark Waite added a comment - - edited Adding support for remotes that are not the default LFS remote URL will not be in the initial release. I don't plan to implement that support, even after the initial release of git LFS because I need to focus on submodule bugs and on authentication bugs once the basic LFS capability has been added. I'm willing to evaluate pull requests from others that propose to add such support, but I don't see it high enough on my list to do that myself. There are too many open pull requests with good ideas for me to spend much time writing improvements myself.
            Hide
            thetaiter Ian Tait added a comment -

            Mark Waite Is there a way to change the timeout time for the LFS pull? Currently it is set at 10 minutes, but we need it to be longer for the first pull to finish successfully.

            Show
            thetaiter Ian Tait added a comment - Mark Waite Is there a way to change the timeout time for the LFS pull? Currently it is set at 10 minutes, but we need it to be longer for the first pull to finish successfully.
            Hide
            markewaite Mark Waite added a comment -

            Since the LFS pull is performed by "git checkout", the timeout should be modified by adding the additional behavior "Advanced checkout behaviours" and set the timeout to the value you'd like.

            Show
            markewaite Mark Waite added a comment - Since the LFS pull is performed by "git checkout", the timeout should be modified by adding the additional behavior "Advanced checkout behaviours" and set the timeout to the value you'd like.
            Hide
            thetaiter Ian Tait added a comment -

            Ah there it is. Thank you for the help!

            Show
            thetaiter Ian Tait added a comment - Ah there it is. Thank you for the help!
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java
            http://jenkins-ci.org/commit/git-client-plugin/470f1257482e0b04c67fcadcbe66d9b4462d2f84
            Log:
            Merge pull request #232 from creste/JENKINS-35687

            JENKINS-35687 Add simple git lfs support

            Compare: https://github.com/jenkinsci/git-client-plugin/compare/9c12a007c5a0...470f1257482e

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/org/jenkinsci/plugins/gitclient/CheckoutCommand.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitClientTest.java http://jenkins-ci.org/commit/git-client-plugin/470f1257482e0b04c67fcadcbe66d9b4462d2f84 Log: Merge pull request #232 from creste/ JENKINS-35687 JENKINS-35687 Add simple git lfs support Compare: https://github.com/jenkinsci/git-client-plugin/compare/9c12a007c5a0...470f1257482e
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Matt Hauck
            Path:
            src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java
            src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java
            http://jenkins-ci.org/commit/git-plugin/a809052a9b8d59975477e52f106383d63b869830
            Log:
            JENKINS-35687 Add simple git lfs support

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Matt Hauck Path: src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java http://jenkins-ci.org/commit/git-plugin/a809052a9b8d59975477e52f106383d63b869830 Log: JENKINS-35687 Add simple git lfs support
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java
            src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java
            http://jenkins-ci.org/commit/git-plugin/ae6883b49bb3973e6139b666b7cea81464317274
            Log:
            Merge pull request #463 from creste/JENKINS-35687

            JENKINS-35687 Add simple git lfs support

            Compare: https://github.com/jenkinsci/git-plugin/compare/7fb16cbe8ff5...ae6883b49bb3

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/extensions/impl/GitLFSPull.java src/test/java/hudson/plugins/git/extensions/impl/CheckoutOptionTest.java http://jenkins-ci.org/commit/git-plugin/ae6883b49bb3973e6139b666b7cea81464317274 Log: Merge pull request #463 from creste/ JENKINS-35687 JENKINS-35687 Add simple git lfs support Compare: https://github.com/jenkinsci/git-plugin/compare/7fb16cbe8ff5...ae6883b49bb3
            Hide
            matthauck Matt Hauck added a comment -

            This can probably be marked as Resolved fixed now?

            Show
            matthauck Matt Hauck added a comment - This can probably be marked as Resolved fixed now?
            Hide
            markewaite Mark Waite added a comment -

            Not yet. I still need to release the git-plugin that includes the fix. I hope to do that within the next 24 hours.

            Show
            markewaite Mark Waite added a comment - Not yet. I still need to release the git-plugin that includes the fix. I hope to do that within the next 24 hours.
            Hide
            matthauck Matt Hauck added a comment -

            Ah, roger. Just upgraded and realized it wasn't there yet, haha.

            Show
            matthauck Matt Hauck added a comment - Ah, roger. Just upgraded and realized it wasn't there yet, haha.
            Hide
            markewaite Mark Waite added a comment -

            Git plugin 3.1.0 released on 4 Mar 2017 now includes support for command line git large file support.

            Show
            markewaite Mark Waite added a comment - Git plugin 3.1.0 released on 4 Mar 2017 now includes support for command line git large file support.
            Hide
            lantero Carlos Ruiz Lantero added a comment -

            Mark Waite that's a wonderful job for supporting git LFS, thanks a lot!

            Is there a way I can use it with pipelines? I've been trying to find examples and/or documentation with no luck. 

            Show
            lantero Carlos Ruiz Lantero added a comment - Mark Waite that's a wonderful job for supporting git LFS, thanks a lot! Is there a way I can use it with pipelines? I've been trying to find examples and/or documentation with no luck. 
            Hide
            markewaite Mark Waite added a comment - - edited

            The "pipeline syntax" link on most pipeline job pages will present a dropdown of pipeline steps.  One of those is "checkout".  Inside that "checkout" step, you can select git as the source control system, then add the "Additional Behaviour" for Git LFS pull.

            If you prefer to read a script example, refer to the Jenkinsfile in my jenkins-bugs repo. That shows an example using GitLFSPull in a pipeline definition.

            Show
            markewaite Mark Waite added a comment - - edited The "pipeline syntax" link on most pipeline job pages will present a dropdown of pipeline steps.  One of those is "checkout".  Inside that "checkout" step, you can select git as the source control system, then add the "Additional Behaviour" for Git LFS pull. If you prefer to read a script example, refer to the Jenkinsfile in my jenkins-bugs repo. That shows an example using GitLFSPull in a pipeline definition.

              People

              • Assignee:
                markewaite Mark Waite
                Reporter:
                matthauck Matt Hauck
              • Votes:
                14 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: