Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38860

Git plugin using local config to update submodules

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      Jenkins v2.6, Git plugin version 3.0.0, git-client 2.2.1
    • Similar Issues:

      Description

      In order to get all of the submodules, the git plugin looks in the local configuration file:

       > git config --get-regexp ^submodule # timeout=10
       > git config --get submodule.externals/jzmq.url # timeout=10
       > git submodule update --remote externals/jzmq
       > git config --get submodule.protobuf-java-format.url # timeout=10
       > git submodule update --remote protobuf-java-format
      FATAL: Command "git submodule update --remote protobuf-java-format" returned status code 1:
      stdout: 
      stderr: error: pathspec 'protobuf-java-format' did not match any file(s) known to git.
      

      However, the .git/config file is local per repository, and old entries and sections do not get overriden. Therefore, when a submodule is no longer in use or has moved, the old configuration can cause an error. Anyway, the local configuration is not part of the remote repository, which is the correct clean repo which is the exclusive authority on git configs.

      A possible solution would be to call deninit in order to remove the old configuration.

      git submodule deinit
      

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment -

            I've constructed a public test which shows the problem and mirrored the same test into a private repository (with authenticated submodule access).

            Show
            markewaite Mark Waite added a comment - I've constructed a public test which shows the problem and mirrored the same test into a private repository (with authenticated submodule access).
            Hide
            jmcazaux Jean-Michel Cazaux added a comment - - edited

            Hi Mark,

            unfortunately, in my case this does not help.

             

            I do have submodules in private repose and others in public repos.

            I have wipped out the workspace.

            I have used "Advance submodule configuration" and checked to recurse into submodules and use parent module credentials, but it fails on the first submodule that happen to be public.

             
            The error is
            FATAL: Command "git config --get submodule.MySubModule.url" returned status code 1

            I am happy to try other workarounds.

             

            Show
            jmcazaux Jean-Michel Cazaux added a comment - - edited Hi Mark, unfortunately, in my case this does not help.   I do have submodules in private repose and others in public repos. I have wipped out the workspace. I have used "Advance submodule configuration" and checked to recurse into submodules and use parent module credentials, but it fails on the first submodule that happen to be public.   The error is FATAL: Command "git config --get submodule.MySubModule.url" returned status code 1 I am happy to try other workarounds.  
            Hide
            markewaite Mark Waite added a comment -

            Jean-Michel Cazaux if wipe workspace does not work, then it is likely that any fix I make will also not work. My goal with the change will be to assure that a repository cloned in a newly created job (with submodules enabled) looks the same as a repository which has been used previously but the submodule definitions in the repository have changed.

            Can you provide detailed instructions to show (in a public repository) the case that is failing for you?

            When you say "I have wiped out the workspace", that doesn't sound like you enabled the "Additional Behaviours" to "Wipe workspace". The work around requires that you enable that additional behavior. It is not enough to interactively wipe out the workspace.

            Show
            markewaite Mark Waite added a comment - Jean-Michel Cazaux if wipe workspace does not work, then it is likely that any fix I make will also not work. My goal with the change will be to assure that a repository cloned in a newly created job (with submodules enabled) looks the same as a repository which has been used previously but the submodule definitions in the repository have changed. Can you provide detailed instructions to show (in a public repository) the case that is failing for you? When you say "I have wiped out the workspace", that doesn't sound like you enabled the "Additional Behaviours" to "Wipe workspace". The work around requires that you enable that additional behavior. It is not enough to interactively wipe out the workspace.
            Hide
            ckitching Chris Kitching added a comment - - edited

            The error reported above:

            > FATAL: Command "git config --get submodule.MySubModule.url" returned status code 1

             

            Occurs for me whenever you have nested submodules. Consider a scenario in which repo A contains submodule B, which contains submodule C.

             

            If "Advanced submodule behaviour" with "Recursively update submodules" is not enabled, Jenkins will clone A, checkout/clone B, and fail to initialise/clone C. This is probably expected behaviour.

             

            If you enable "Recursively update submodules", you get the erro:

            > FATAL: Command "git config --get submodule.C.url" returned status code 1

             

            Here's some complete log output of this scenario, with repo names changed to A, B, and C to fit my example scenario:

             

            [0:native:Debug] Cloning the remote Git repository
            [0:native:Debug] Cloning with configured refspecs honoured and without tags
            [0:native:Debug] Cloning repository A
            [0:native:Debug] > git init /workspace/A_master_10/source # timeout=10
            [0:native:Debug] Fetching upstream changes from A
            [0:native:Debug] > git --version # timeout=10
            [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key
            [0:native:Debug] > git fetch --no-tags --progress A+refs/heads/master:refs/remotes/origin/master
            [0:native:Debug] > git config remote.origin.url A # timeout=10
            [0:native:Debug] > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
            [0:native:Debug] > git config remote.origin.url A # timeout=10
            [0:native:Debug] Fetching without tags
            [0:native:Debug] Fetching upstream changes from A
            [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key
            [0:native:Debug] > git fetch --no-tags --progress A+refs/heads/master:refs/remotes/origin/master
            [0:native:Debug] Checking out Revision 7b8f0cd431c42d62bd76ec23e879a95ea1f2e01f (master)
            [0:native:Debug] Commit message: "Bump B"
            [0:native:Debug] Enabling Git LFS pull
            [0:native:Debug] > git config core.sparsecheckout # timeout=10
            [0:native:Debug] > git checkout -f 7b8f0cd431c42d62bd76ec23e879a95ea1f2e01f
            [0:native:Debug] > git config --get remote.origin.url # timeout=10
            [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key
            [0:native:Debug] > git lfs pull origin
            [0:native:Debug] > git rev-list 7b8f0cd431c42d62bd76ec23e879a95ea1f2e01f # timeout=10
            [0:native:Debug] > git remote # timeout=10
            [0:native:Debug] > git submodule init # timeout=10
            [0:native:Debug] > git submodule sync # timeout=10
            [0:native:Debug] > git config --get remote.origin.url # timeout=10
            [0:native:Debug] > git submodule init # timeout=10
            [0:native:Debug] > git config -f .gitmodules --get-regexp ^submodule\.(.*)\.url # timeout=10
            [0:native:Debug] > git config --get submodule.B.url # timeout=10
            [0:native:Debug] > git remote # timeout=10
            [0:native:Debug] > git config --get remote.origin.url # timeout=10
            [0:native:Debug] > git config -f .gitmodules --get submodule.B.path # timeout=10
            [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key
            [0:native:Debug] > git submodule update --init --recursive B
            [0:native:Debug] > git config --get submodule.C.url # timeout=10

             

             

            That last command fails, because it is being executed inside A. The submodule.C.url config value exists only in B, not in A, so this fails (the similar command earlier for B works, however).

             

            This seems to be the core of the bug: you're not changing directory when executing this config --get command, causing this url-query step to fail when recursive submodule checkout is enabled.

             

            Show
            ckitching Chris Kitching added a comment - - edited The error reported above: > FATAL: Command "git config --get submodule.MySubModule.url" returned status code 1   Occurs for me whenever you have nested submodules. Consider a scenario in which repo A contains submodule B, which contains submodule C.   If "Advanced submodule behaviour" with "Recursively update submodules" is not enabled, Jenkins will clone A, checkout/clone B, and fail to initialise/clone C. This is probably expected behaviour.   If you enable "Recursively update submodules", you get the erro: > FATAL: Command "git config --get submodule.C.url" returned status code 1   Here's some complete log output of this scenario, with repo names changed to A, B, and C to fit my example scenario:   [0:native:Debug] Cloning the remote Git repository [0:native:Debug] Cloning with configured refspecs honoured and without tags [0:native:Debug] Cloning repository A [0:native:Debug] > git init /workspace/A_master_10/source # timeout=10 [0:native:Debug] Fetching upstream changes from A [0:native:Debug] > git --version # timeout=10 [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key [0:native:Debug] > git fetch --no-tags --progress A+refs/heads/master:refs/remotes/origin/master [0:native:Debug] > git config remote.origin.url A # timeout=10 [0:native:Debug] > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10 [0:native:Debug] > git config remote.origin.url A # timeout=10 [0:native:Debug] Fetching without tags [0:native:Debug] Fetching upstream changes from A [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key [0:native:Debug] > git fetch --no-tags --progress A+refs/heads/master:refs/remotes/origin/master [0:native:Debug] Checking out Revision 7b8f0cd431c42d62bd76ec23e879a95ea1f2e01f (master) [0:native:Debug] Commit message: "Bump B" [0:native:Debug] Enabling Git LFS pull [0:native:Debug] > git config core.sparsecheckout # timeout=10 [0:native:Debug] > git checkout -f 7b8f0cd431c42d62bd76ec23e879a95ea1f2e01f [0:native:Debug] > git config --get remote.origin.url # timeout=10 [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key [0:native:Debug] > git lfs pull origin [0:native:Debug] > git rev-list 7b8f0cd431c42d62bd76ec23e879a95ea1f2e01f # timeout=10 [0:native:Debug] > git remote # timeout=10 [0:native:Debug] > git submodule init # timeout=10 [0:native:Debug] > git submodule sync # timeout=10 [0:native:Debug] > git config --get remote.origin.url # timeout=10 [0:native:Debug] > git submodule init # timeout=10 [0:native:Debug] > git config -f .gitmodules --get-regexp ^submodule\.(.*)\.url # timeout=10 [0:native:Debug] > git config --get submodule.B.url # timeout=10 [0:native:Debug] > git remote # timeout=10 [0:native:Debug] > git config --get remote.origin.url # timeout=10 [0:native:Debug] > git config -f .gitmodules --get submodule.B.path # timeout=10 [0:native:Debug] using GIT_SSH to set credentials Jenkins GitHub SSH key [0:native:Debug] > git submodule update --init --recursive B [0:native:Debug] > git config --get submodule.C.url # timeout=10     That last command fails, because it is being executed inside A. The submodule.C.url config value exists only in B, not in A, so this fails (the similar command earlier for B works, however).   This seems to be the core of the bug: you're not changing directory when executing this config --get command, causing this url-query step to fail when recursive submodule checkout is enabled.  
            Hide
            rjanicki Rafal Janicki added a comment -

            Unfortunately, I ran into the exact issue as described by Chris Kitching. I tried to disable the recursive option of checking out, but that didn't work either - the plugin still tries to clone recursively.

            Show
            rjanicki Rafal Janicki added a comment - Unfortunately, I ran into the exact issue as described by Chris Kitching. I tried to disable the recursive option of checking out, but that didn't work either - the plugin still tries to clone recursively.

              People

              • Assignee:
                Unassigned
                Reporter:
                bdanilovich Boris Danilovich
              • Votes:
                16 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated: