Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47531

git-lfs: allow specifying separate credentials

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.73.2
      Git plugin ver. 3.6.0
    • Similar Issues:

      Description

      I have a git repo, which itself is located on ssh:// remote, but whose [lfs] url points to https:// address.

      When I try to check it out, Jenkins job gets stuck at
      using GIT_SSH to set credentials
      > git lfs pull origin
      command. When I run it manually, I see that it is interactively asking for username and password. I don't see any additional settings for "Git LFS pull after checkout" behavior, so I wonder if it would be possible to allow specifying credentials there?

        Attachments

          Issue Links

            Activity

            mephi42 mephi42 created issue -
            markewaite Mark Waite made changes -
            Field Original Value New Value
            Assignee Mark Waite [ markewaite ]
            Hide
            markewaite Mark Waite added a comment -

            I think this is an uncommon use case. The git lfs implementation and documentation focuses on the https access path. I'm unlikely to spend any time adding this use case unless it becomes much more widely used.

            Show
            markewaite Mark Waite added a comment - I think this is an uncommon use case. The git lfs implementation and documentation focuses on the https access path. I'm unlikely to spend any time adding this use case unless it becomes much more widely used.
            Hide
            mephi42 mephi42 added a comment -

            In my case git lfs is on https server.

            I think my problem is that I need two sets of credentials in order to do the clone:

            • ssh public key to get git objects (which I can specify in Repositories->Credentials)
            • https password to get lfs blobs
            Show
            mephi42 mephi42 added a comment - In my case git lfs is on https server. I think my problem is that I need two sets of credentials in order to do the clone: ssh public key to get git objects (which I can specify in Repositories->Credentials) https password to get lfs blobs
            Hide
            tbull Tristan Bull added a comment -

            Mark Waite I am having the same issue, and I would just like to add my 2 cents. I do not think this is as uncommon as you might think. IMO, this is not related to SSH vs HTTPS. The issue is that (as far as I can tell) there is no way to have separate credentials for your git repo and LFS repo. I think this is a common use case for anyone who hosts their LFS files outside of github. In our case, we have our source code in github and our LFS files on a self-hosted nexus server. GitHub uses github credentials (obviously) while LFS authenticates against our LDAP server.

            Show
            tbull Tristan Bull added a comment - Mark Waite I am having the same issue, and I would just like to add my 2 cents. I do not think this is as uncommon as you might think. IMO, this is not related to SSH vs HTTPS. The issue is that (as far as I can tell) there is no way to have separate credentials for your git repo and LFS repo. I think this is a common use case for anyone who hosts their LFS files outside of github. In our case, we have our source code in github and our LFS files on a self-hosted nexus server. GitHub uses github credentials (obviously) while LFS authenticates against our LDAP server.
            Hide
            mephi42 mephi42 added a comment -

            I have implemented this functionality here:

            https://github.com/mephi42/git-client-plugin/commits/lfs-credentials

            https://github.com/mephi42/git-plugin/commits/lfs-credentials

            The patches are not that big or intrusive. The code works in my environment.

            Mark Waite, do you think it is acceptable to put this functionality into Jenkins git plugins? If yes, I can submit pull requests.

            Show
            mephi42 mephi42 added a comment - I have implemented this functionality here: https://github.com/mephi42/git-client-plugin/commits/lfs-credentials https://github.com/mephi42/git-plugin/commits/lfs-credentials The patches are not that big or intrusive. The code works in my environment. Mark Waite , do you think it is acceptable to put this functionality into Jenkins git plugins? If yes, I can submit pull requests.
            Hide
            markewaite Mark Waite added a comment -

            mephi42 that change will need a review by Stephen Connolly. The plan is to eventually update the credentials implementation in the git plugin to use the latest credentials API. I wouldn't want to bring this change into the plugins if we'll then need to remove it or revert it in the transition to the most recent credentials API.

            Stephen Connolly, your comments?

            Show
            markewaite Mark Waite added a comment - mephi42 that change will need a review by Stephen Connolly . The plan is to eventually update the credentials implementation in the git plugin to use the latest credentials API. I wouldn't want to bring this change into the plugins if we'll then need to remove it or revert it in the transition to the most recent credentials API. Stephen Connolly , your comments?
            Hide
            stephenconnolly Stephen Connolly added a comment -

            The git-client changes seem ok to me... I need to look at the git plugin changes

            Show
            stephenconnolly Stephen Connolly added a comment - The git-client changes seem ok to me... I need to look at the git plugin changes
            Hide
            stephenconnolly Stephen Connolly added a comment -

            The git-plugin changes are “close but no cigar”

            There’s some tweaks I’d like to see before i’d be Ok with merging the git-plugin changes. Probably better expressed by comments on a PR rather than on this ticket

            Show
            stephenconnolly Stephen Connolly added a comment - The git-plugin changes are “close but no cigar” There’s some tweaks I’d like to see before i’d be Ok with merging the git-plugin changes. Probably better expressed by comments on a PR rather than on this ticket
            Hide
            markewaite Mark Waite added a comment -

            mephi42 Stephen's review is enough to encourage a pull request to the git client plugin and the git plugin.

            Show
            markewaite Mark Waite added a comment - mephi42 Stephen's review is enough to encourage a pull request to the git client plugin and the git plugin.
            Show
            mephi42 mephi42 added a comment - Thanks! I have sent https://github.com/jenkinsci/git-client-plugin/pull/280 and https://github.com/jenkinsci/git-plugin/pull/546
            Hide
            markewaite Mark Waite added a comment -

            Git client plugin component is Included in git client plugin 2.6.0, released 27 Oct 2017

            Show
            markewaite Mark Waite added a comment - Git client plugin component is Included in git client plugin 2.6.0, released 27 Oct 2017
            Hide
            mephi42 mephi42 added a comment -

            Great!

            In the meantime, I have modified my git-plugin pull request (https://github.com/jenkinsci/git-plugin/pull/546) to reference the newly released git client plugin.

            There are failing jobs (only windows-8 passed), but it looks as if this is due to timeouts on builders when downloading maven artifacts.

            Show
            mephi42 mephi42 added a comment - Great! In the meantime, I have modified my git-plugin pull request ( https://github.com/jenkinsci/git-plugin/pull/546 ) to reference the newly released git client plugin. There are failing jobs (only windows-8 passed), but it looks as if this is due to timeouts on builders when downloading maven artifacts.
            Hide
            markewaite Mark Waite added a comment -

            I created two sample jobs that I can use to verify the functionality.  We'll need Stephen's comments before the change can be merged.

            Show
            markewaite Mark Waite added a comment - I created two sample jobs that I can use to verify the functionality.  We'll need Stephen's comments before the change can be merged.
            Hide
            thelq Leon Blakey added a comment -

            Any progress? We host our own simple git server and LFS server due to size costs.

            A workaround is to add .git-credentials manually on the build slave, then for piplines replace "LFS after checkout" option with a step that runs "git -c credential.helper=store lfs pull origin".

            Related: Jenkins will keep trying the same credentials until it times out in 10 minutes. GIT_TRACE=1 shows it's stuck in a loop of seeing Basic auth, setting GIT_ASKPASS, post, 401 error, repeat.

            Show
            thelq Leon Blakey added a comment - Any progress? We host our own simple git server and LFS server due to size costs. A workaround is to add .git-credentials manually on the build slave, then for piplines replace "LFS after checkout" option with a step that runs "git -c credential.helper=store lfs pull origin". Related: Jenkins will keep trying the same credentials until it times out in 10 minutes. GIT_TRACE=1 shows it's stuck in a loop of seeing Basic auth, setting GIT_ASKPASS, post, 401 error, repeat.
            Hide
            markewaite Mark Waite added a comment -

            Leon Blakey you could download the pull request build and test it in your environment. If it meets your needs and you note that in the pull request, it helps persuade me that the pull request benefits more than just the original submitter.

            Show
            markewaite Mark Waite added a comment - Leon Blakey you could download the pull request build and test it in your environment. If it meets your needs and you note that in the pull request, it helps persuade me that the pull request benefits more than just the original submitter.
            Hide
            mephi42 mephi42 added a comment -

            Stephen Connolly, could you please take another look at the pull request? I have some time now to work on improving the pull request in case it's still required.

            Show
            mephi42 mephi42 added a comment - Stephen Connolly , could you please take another look at the pull request? I have some time now to work on improving the pull request in case it's still required.
            Hide
            afunix Pavel Malyshev added a comment -

            One more "me too" here.
            We have git repos accessed over ssh and lfs is served by https..

            Show
            afunix Pavel Malyshev added a comment - One more "me too" here. We have git repos accessed over ssh and lfs is served by https..
            Hide
            markewaite Mark Waite added a comment -

            Pavel Malyshev if you install the latest release of the git client plugin from the update center and install the pull request build, you can check to see if the proposed change solves the case that is interesting to you.

            That helps me and others know if the proposed git plugin pull request meets the needs of more users than the original submitter.

            Show
            markewaite Mark Waite added a comment - Pavel Malyshev if you install the latest release of the git client plugin from the update center and install the pull request build , you can check to see if the proposed change solves the case that is interesting to you. That helps me and others know if the proposed git plugin pull request meets the needs of more users than the original submitter.
            Hide
            bienstock Gad Maor added a comment - - edited

            Hi Mark Waite and Stephen Connolly,
            We also have this issue with a lot of the projects we support - we use a corporate GitHub Enterprise instance but use a local Artifactory instance to store GitLFS objects, so the credentials are different than the GHE credentials.
            We recently had to scrap the SSH authentication to Artifactory, because of a weak cipher vulnerability discovered in its SSH server, so we had to turn to HTTPS authentication and for that we need to input username & API token.
            Currently, we are using an ugly workaround of setting the GitLFS smudge filter to "skip" and manually injecting the token to the Jenkins workspace via EnvInject then running "git lfs pull".

            Show
            bienstock Gad Maor added a comment - - edited Hi Mark Waite and Stephen Connolly , We also have this issue with a lot of the projects we support - we use a corporate GitHub Enterprise instance but use a local Artifactory instance to store GitLFS objects, so the credentials are different than the GHE credentials. We recently had to scrap the SSH authentication to Artifactory, because of a weak cipher vulnerability discovered in its SSH server, so we had to turn to HTTPS authentication and for that we need to input username & API token. Currently, we are using an ugly workaround of setting the GitLFS smudge filter to "skip" and manually injecting the token to the Jenkins workspace via EnvInject then running "git lfs pull".
            Hide
            markewaite Mark Waite added a comment - - edited

            Gad Maor instead of the workaround that you're using, you could download and install a plugin build with the proposed change to add support for separate LFS credentials. Report your results on that pull request so that others know what you observed.

            If that solves the problem then you've avoided the workaround and have helped the Jenkins community evaluate the pull request.

            If that doesn't solve the problem, you return to your workaround and have helped the Jenkins community evaluate the pull request.

            You'll need to install the git client plugin beta release from the experimental update center as well, which will also help the Jenkins community.

            Show
            markewaite Mark Waite added a comment - - edited Gad Maor instead of the workaround that you're using, you could download and install a plugin build with the proposed change to add support for separate LFS credentials. Report your results on that pull request so that others know what you observed. If that solves the problem then you've avoided the workaround and have helped the Jenkins community evaluate the pull request. If that doesn't solve the problem, you return to your workaround and have helped the Jenkins community evaluate the pull request. You'll need to install the git client plugin beta release from the experimental update center as well, which will also help the Jenkins community.
            Hide
            afunix Pavel Malyshev added a comment -

            For me the plugin fails to load, since I have an outdated Jenkins version.
            It's a bit hard to update a live server or a plugin on a live server which serves hundreds builds hourly.

            Show
            afunix Pavel Malyshev added a comment - For me the plugin fails to load, since I have an outdated Jenkins version. It's a bit hard to update a live server or a plugin on a live server which serves hundreds builds hourly.
            Hide
            markewaite Mark Waite added a comment -

            Pavel Malyshev the pre-release plugin depends on git client plugin 3.0.0 beta releases. Be sure that you've installed the updated git client plugin before you install the pre-release git plugin.

            Git client plugin 3.0.0 provides a much newer version of JGit and drops the support for Jenkins versions before 2.60.3.

            Git plugin 4.0.0 pre-release (like this case) should run on any Jenkins version 2.60.3 or newer.

            If your Jenkins version is older than 2.60.3, then you won't be able to use this enhancement anyway. It won't be included in a git plugin version before git plugin 4.0.0, and git plugin 4.0.0 will require at least Jenkins 2.60.3.

            Show
            markewaite Mark Waite added a comment - Pavel Malyshev the pre-release plugin depends on git client plugin 3.0.0 beta releases. Be sure that you've installed the updated git client plugin before you install the pre-release git plugin. Git client plugin 3.0.0 provides a much newer version of JGit and drops the support for Jenkins versions before 2.60.3. Git plugin 4.0.0 pre-release (like this case) should run on any Jenkins version 2.60.3 or newer. If your Jenkins version is older than 2.60.3, then you won't be able to use this enhancement anyway. It won't be included in a git plugin version before git plugin 4.0.0, and git plugin 4.0.0 will require at least Jenkins 2.60.3.
            Hide
            amatera Andrea Matera added a comment -

            Is possible to have some updates on this feature?

            Will be a very nice-to-have also for our company because we are using Bitbucket for git repos and JFrog Artifactory for git-lfs artifacs.

            Show
            amatera Andrea Matera added a comment - Is possible to have some updates on this feature? Will be a very nice-to-have also for our company because we are using Bitbucket for git repos and JFrog Artifactory for git-lfs artifacs .
            Hide
            markewaite Mark Waite added a comment - - edited

            Andrea Matera that pull request has been labeled as "Later". That means that it is one of over 60 pull requests which were not progressing because they were not high enough priority for me and were not receiving significant attention from others. I labeled those pull requests and closed them so that they would not spend time in the continuous integration servers. They will be reopened when time allows me to visit them again.

            My current priorities are:

            1. Resolve git plugin 4.0 pre-release critical regressions and git client plugin 3.0 pre-release critical regressions
            2. Test and release git plugin 4.0
            3. Resolve BuildData bloat bug (JENKINS-19022)
            4. Test and release BuildData bloat fix
            5. Improve notifyCommit handling
            6. Test and release notifyCommit handling improvements
            7. Improve submodule handling
            8. Test and release submodule handling improvements
            9. Improve caching for large git repositories
            10. Test and release git caching improvements
            11. Improve changelog handling
            12. Test and release changelog handling improvements
            13. Other improvements (like this credentials improvement)

            If you are available to help and would like to experiment with this proposed change, you could build it yourself, install it in your environment (development or production depending on how much risk you are willing to take), and report the results of that experiment to the pull request.

            If you'd like guidance on how to build that pull request locally, I'm happy to help. I'd prefer to provide that type of help in either the jenkinsci-dev mailing list or in the jenkins gitter channel, but I'm willing to help you build it yourself for testing and use. If you test it and use it, that meets your immediate need and helps me by adding more people that are testing the code.

            Show
            markewaite Mark Waite added a comment - - edited Andrea Matera that pull request has been labeled as " Later ". That means that it is one of over 60 pull requests which were not progressing because they were not high enough priority for me and were not receiving significant attention from others. I labeled those pull requests and closed them so that they would not spend time in the continuous integration servers. They will be reopened when time allows me to visit them again. My current priorities are: Resolve git plugin 4.0 pre-release critical regressions and git client plugin 3.0 pre-release critical regressions Test and release git plugin 4.0 Resolve BuildData bloat bug ( JENKINS-19022 ) Test and release BuildData bloat fix Improve notifyCommit handling Test and release notifyCommit handling improvements Improve submodule handling Test and release submodule handling improvements Improve caching for large git repositories Test and release git caching improvements Improve changelog handling Test and release changelog handling improvements Other improvements (like this credentials improvement) If you are available to help and would like to experiment with this proposed change, you could build it yourself, install it in your environment (development or production depending on how much risk you are willing to take), and report the results of that experiment to the pull request . If you'd like guidance on how to build that pull request locally, I'm happy to help. I'd prefer to provide that type of help in either the jenkinsci-dev mailing list or in the jenkins gitter channel, but I'm willing to help you build it yourself for testing and use. If you test it and use it, that meets your immediate need and helps me by adding more people that are testing the code.
            renescheibe René Scheibe made changes -
            Link This issue relates to JENKINS-59139 [ JENKINS-59139 ]
            renescheibe René Scheibe made changes -
            Remote Link This issue links to "git-plugin pr#546 (Web Link)" [ 24007 ]
            renescheibe René Scheibe made changes -
            Remote Link This issue links to "git-client-plugin pr#280 (Web Link)" [ 24008 ]
            Hide
            dominmuda Domingo Muñoz added a comment -

            Hello, is there any progress on this so far?

            Show
            dominmuda Domingo Muñoz added a comment - Hello, is there any progress on this so far?
            Hide
            markewaite Mark Waite added a comment -

            Domingo Muñoz the change has been delivered in git client plugin 2.6.0 and later. The change needed in git plugin has not been delivered and has not been reopened. If you'd like to take PR-546 as a basis for a new pull request, that would be great to have.

            The git LFS command line implementation has evolved further since the pull request was originally proposed. This will need to be evaluated with an even broader range of git LFS releases than previously.

            Show
            markewaite Mark Waite added a comment - Domingo Muñoz the change has been delivered in git client plugin 2.6.0 and later. The change needed in git plugin has not been delivered and has not been reopened. If you'd like to take PR-546 as a basis for a new pull request, that would be great to have. The git LFS command line implementation has evolved further since the pull request was originally proposed. This will need to be evaluated with an even broader range of git LFS releases than previously.
            Hide
            patricklang2 Patrick Lang added a comment - - edited

            I'm trying to revive this. I started from the past PR and addressed some feedback but it's not working quite yet. I'm using LFS on Artifactory, so I need a different credential supplied

            https://github.com/patricklangsonos/git-plugin/tree/git-4.2.2/JENKINS-47531

            This is currently branched from the 4.2.2 tag because I wanted to test the change in isolation without moving to the 4.3+ codebase. Once I'm happy with it I'll squash it and get a PR open against master.

            Show
            patricklang2 Patrick Lang added a comment - - edited I'm trying to revive this. I started from the past PR and addressed some feedback but it's not working quite yet. I'm using LFS on Artifactory, so I need a different credential supplied https://github.com/patricklangsonos/git-plugin/tree/git-4.2.2/JENKINS-47531 This is currently branched from the 4.2.2 tag because I wanted to test the change in isolation without moving to the 4.3+ codebase. Once I'm happy with it I'll squash it and get a PR open against master.
            Show
            patricklang2 Patrick Lang added a comment - PR open  https://github.com/jenkinsci/git-plugin/pull/930

              People

              • Assignee:
                Unassigned
                Reporter:
                mephi42 mephi42
              • Votes:
                6 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated: