Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56569

Prevent automatic pulling of lfs files if no GitLFSPull option is set

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: git-client-plugin
    • Labels:
      None
    • Environment:
      git-client-plugin 2.7.6
    • Similar Issues:

      Description

      Depending on the used method to install git on a system, git lfs is enabled per default on a system via these settings in the system-wide config file:

      [filter "lfs"]
          clean = git-lfs clean -- %f
          smudge = git-lfs smudge -- %f
          process = git-lfs filter-process
          required = true
      

      This means that even if no "git lfs pull" is performed explicitly, on checkout lfs files are pulled automatically due to the smudge filter.

      Problems

      1. If the remote server requires authentication and none is configured on the system on a Jenkins agent (e.g. ssh key in ~/.ssh/), the checkout fails.
      2. If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

      I therefore suggest to always set the environment variable GIT_LFS_SKIP_SMUDGE=1 even if no "GitLFSPull" option was specified.
      This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
      This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

        Attachments

          Issue Links

            Activity

            Hide
            flokli Florian Klink added a comment -

            If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

            Shouldn't then the clone itself fail, too?
            Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

            If lfs files shall not be pulled at all, there is no way to configure this in the plugin.

            Why don't you want your lfs pointers to be resolved while checking out? If that smudge filter is installed system-wide, why should jenkins override that behaviour at all, or silently flip its default?

            I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}}even if no "GitLFSPull" option was specified.
            This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication.
            This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option.

            Again, this really sounds more like a broken LFS backend on your side, than something that should be changed on Jenkins' side.

             

            This would also be problematic with declarative pipelines, for which there's currently no way to enable the GitLFSPull option.

             

            Show
            flokli Florian Klink added a comment - If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails. Shouldn't then the clone itself fail, too? Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too. If lfs files shall not be pulled at all, there is no way to configure this in the plugin. Why don't you want your lfs pointers to be resolved while checking out? If that smudge filter is installed system-wide, why should jenkins override that behaviour at all, or silently flip its default? I therefore suggest to always set the environment variable {{GIT_LFS_SKIP_SMUDGE=1}}even if no "GitLFSPull" option was specified. This makes sure that no "git lfs" commands that are not controlled by the git-client-plugin are launched and thus resolves the issue that the checkout fails in case the remote server requires authentication. This also prevents pulling lfs files regardless of a system's configuration and requires to explicitly pull lfs files via the "GitLFSPull" option. Again, this really sounds more like a broken LFS backend on your side, than something that should be changed on Jenkins' side.   This would also be problematic with declarative pipelines, for which there's currently no way to enable the GitLFSPull option.  
            Hide
            flokli Florian Klink added a comment -

            It looks like this is already implemented in https://github.com/jenkinsci/git-client-plugin/commit/853603cccd4434b116ef9b8e094c3f5b815aa75a - however, the git lfs pull doesn't seem to properly replace lfs pointers…

            Show
            flokli Florian Klink added a comment - It looks like this is already implemented in  https://github.com/jenkinsci/git-client-plugin/commit/853603cccd4434b116ef9b8e094c3f5b815aa75a  - however, the git lfs pull doesn't seem to properly replace lfs pointers…
            Hide
            flokli Florian Klink added a comment -

            Turns out, lfs pull did replace lfs pointers, but some other cleaning-related "additional behaviours" changed the resolved files back to pointers.

             

            I think this issue can be closed.

             

            I opened JENKINS-59139, as general clone performance has improved a lot recently.

            Show
            flokli Florian Klink added a comment - Turns out, lfs pull did replace lfs pointers, but some other cleaning-related "additional behaviours" changed the resolved files back to pointers.   I think this issue can be closed.   I opened  JENKINS-59139 , as general clone performance has improved a lot recently.
            Hide
            renescheibe René Scheibe added a comment - - edited

            Regarding this point:

            René Scheibe

            If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails.

            Florian Klink

            Shouldn't then the clone itself fail, too?
            Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too.

            In the git-client-plugin it's implemented that the following steps are performed

            1. git fetch (see here)
            2. git checkout (see here)
            3. in case of a specified "Git LFS pull after checkout" behaviour also git lfs pull (see here)

            If credentials are specified by the user they are only provided for git fetch and git lfs pull.
            Therefore if git-lfs filter-process is triggered automatically via git checkout the checkout fails because the credentials are not provided here.

            If one really wants to go the other direction to never use GIT_LFS_SKIP_SMUDGE as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this Mark Waite?

            Show
            renescheibe René Scheibe added a comment - - edited Regarding this point: René Scheibe If the remote server requires authentication and none is configured on the system (e.g. ssh key), the checkout fails. Florian Klink Shouldn't then the clone itself fail, too? Normal ssh clone setups nowadays execute `git-lfs-authenticate` to obtain http urls and authentication, even when purely cloning over ssh. Once you're able to clone, resolving LFS pointers should work too. In the git-client-plugin it's implemented that the following steps are performed git fetch (see here ) git checkout (see here ) in case of a specified "Git LFS pull after checkout" behaviour also git lfs pull (see here ) If credentials are specified by the user they are only provided for git fetch and git lfs pull . Therefore if git-lfs filter-process is triggered automatically via git checkout the checkout fails because the credentials are not provided here. If one really wants to go the other direction to never use GIT_LFS_SKIP_SMUDGE as described in JENKINS-59139 than the checkout should at least always be performed with providing the credentials. But JENKINS-47531 (to provide dedicated LFS credentials) is then still not covered. What's your opinion on this Mark Waite ?
            Hide
            markewaite Mark Waite added a comment -

            I think that JENKINS-47531 (dedicated LFS credentials) is less and less relevant to Jenkins users.

            I'm in favor of making the "Git LFS Pull" option truly mean that LFS is enabled. WIthout it, LFS would be disabled. That makes a very clear distinction between LFS pull enabled and disabled for a job.

            However, René Scheibe, wouldn't that mean being incompatible with users that explicitly enabled LFS with the configuration file settings that you mentioned?

            Show
            markewaite Mark Waite added a comment - I think that JENKINS-47531 (dedicated LFS credentials) is less and less relevant to Jenkins users. I'm in favor of making the "Git LFS Pull" option truly mean that LFS is enabled. WIthout it, LFS would be disabled. That makes a very clear distinction between LFS pull enabled and disabled for a job. However, René Scheibe , wouldn't that mean being incompatible with users that explicitly enabled LFS with the configuration file settings that you mentioned?
            Hide
            renescheibe René Scheibe added a comment -

            Right, that's a breaking change so to speak.

            If GIT_LFS_SKIP_SMUDGE=1 is always set to disable any enabled smudge filter and git lfs pull is only performed when the "Git LFS pull after checkout" behaviour is specified, users that currently have the smudge filter enabled via a system/global/local git config would have to change their job configuration.

            Show
            renescheibe René Scheibe added a comment - Right, that's a breaking change so to speak. If GIT_LFS_SKIP_SMUDGE=1 is always set to disable any enabled smudge filter and git lfs pull is only performed when the "Git LFS pull after checkout" behaviour is specified, users that currently have the smudge filter enabled via a system/global/local git config would have to change their job configuration.
            Hide
            renescheibe René Scheibe added a comment -

            Above I also linked JENKINS-45228 which is about git LFS at merge time. (This here is about git LFS at checkout time).

            Show
            renescheibe René Scheibe added a comment - Above I also linked JENKINS-45228 which is about git LFS at merge time. (This here is about git LFS at checkout time).
            Hide
            renescheibe René Scheibe added a comment -

            Above I also linked JENKINS-59516 which is about git LFS pull for submodules. In case it's decided here that the smudge filter is always disabled via GIT_LFS_SKIP_SMUDGE=1, then the referenced issue has to be worked on too. Otherwise it would not be possible at all to pull LFS files in submodules (even not with configuration outside of Jenkins).

            Show
            renescheibe René Scheibe added a comment - Above I also linked JENKINS-59516 which is about git LFS pull for submodules. In case it's decided here that the smudge filter is always disabled via GIT_LFS_SKIP_SMUDGE=1 , then the referenced issue has to be worked on too. Otherwise it would not be possible at all to pull LFS files in submodules (even not with configuration outside of Jenkins).

              People

              • Assignee:
                Unassigned
                Reporter:
                renescheibe René Scheibe
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: