Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21248

Add shallow update init for submodule

    Details

    • Similar Issues:

      Description

      Recent versions of cgit support shallow clone support in submodule. Like this
      git submodule update --init --depth 1

      But it doesn't support this new feature in current version of git-client-plugin.

        Attachments

          Activity

          Hide
          sverhoef Stefan Verhoeff added a comment -

          Tried this feature in beta git-4.0.0-beta9. But the plugin is refusing the allow the option:

          > git submodule init # timeout=10
          [WARNING] Git client older than 1.8.4 doesn't support shallow submodule updates. This flag is ignored.
          

          However my Git version should support this, see output of sh 'git --version' in my pipeline script:

          + git --version
          git version 2.11.0
          

          Bug in the version detection?

          Show
          sverhoef Stefan Verhoeff added a comment - Tried this feature in beta git-4.0.0-beta9 . But the plugin is refusing the allow the option: > git submodule init # timeout=10 [WARNING] Git client older than 1.8.4 doesn't support shallow submodule updates. This flag is ignored. However my Git version should support this, see output of sh 'git --version' in my pipeline script: + git --version git version 2.11.0 Bug in the version detection?
          Hide
          markewaite Mark Waite added a comment - - edited

          There could be a bug in the version detection logic, but you should probably first check that the version of git being used by your job is the same as the version reported in the pipeline script sh step.

          I've not seen any other bug reports of broken version detection. There are specific automated tests in the plugin to confirm that git version numbers in various formats are detected correctly. Those tests don't use 2.11.0 specifically, but they use 2.10.0 and other interesting values near it.

          If there is a bug in the detection logic, I'll need to update my acceptance test to exercise that bug.

          There seem to be some interesting cases in that acceptance test when running on command line git version 2.11.0, but the cases that I'm seeing are not related to version checking logic. In my case, the git submodule update fails with a warning that there are no common commits. The same message does not appear on at least some other command line git versions. I'll need to extend the acceptance test to execute with each of the versions of command line git in my test environment. That will then tell me if there are problems related to command line git version. Note that the issue I'm investigating shows that in my environment I'm able to perform the expected command line git operations. Thus, at least in my environment, the version check is working, since my investigation could not have started without passing that version check.

          Show
          markewaite Mark Waite added a comment - - edited There could be a bug in the version detection logic , but you should probably first check that the version of git being used by your job is the same as the version reported in the pipeline script sh step. I've not seen any other bug reports of broken version detection. There are specific automated tests in the plugin to confirm that git version numbers in various formats are detected correctly. Those tests don't use 2.11.0 specifically, but they use 2.10.0 and other interesting values near it. If there is a bug in the detection logic, I'll need to update my acceptance test to exercise that bug. There seem to be some interesting cases in that acceptance test when running on command line git version 2.11.0, but the cases that I'm seeing are not related to version checking logic. In my case, the git submodule update fails with a warning that there are no common commits . The same message does not appear on at least some other command line git versions. I'll need to extend the acceptance test to execute with each of the versions of command line git in my test environment. That will then tell me if there are problems related to command line git version. Note that the issue I'm investigating shows that in my environment I'm able to perform the expected command line git operations. Thus, at least in my environment, the version check is working, since my investigation could not have started without passing that version check.
          Hide
          markewaite Mark Waite added a comment -

          I've further investigated the JENKINS-21248 acceptance test failures in my environment.

          The initial submodule checkout succeeds on the following versions of command line git:

          • 2.21
          • 2.20
          • 2.17
          • 2.16

          The initial submodule checkout fails on the following versions of command line git:

          • 2.11
          • 2.10
          • 2.7
          • 2.1

          The stack trace in the checkout failure is:

          hudson.plugins.git.GitException: Command "git submodule update --reference /var/lib/git/mwaite/bugs/jenkins-bugs.git --depth=1 modules/JENKINS-46504.url" returned status code 1:
          stdout: 
          stderr: Cloning into 'modules/JENKINS-46504.url'...
          warning: no common commits
          fatal: reference is not a tree: 0736ba35a0d8c05236e3b71584bc4e149aa5f10a
          Unable to checkout '0736ba35a0d8c05236e3b71584bc4e149aa5f10a' in submodule path 'modules/JENKINS-46504.url'
          
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2298)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1910)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1334)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
          	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
          	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:75)
          	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:64)
          Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to debian8-mwaite
          		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
          		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
          		at hudson.remoting.Channel.call(Channel.java:957)
          		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
          		at jdk.internal.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
          		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
          		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
          		at com.sun.proxy.$Proxy119.execute(Unknown Source)
          		at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:148)
          		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1231)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
          		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          		at java.base/java.lang.Thread.run(Thread.java:834)
          Caused: hudson.plugins.git.GitException
          	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:87)
          	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:68)
          	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1337)
          	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
          	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          	at hudson.remoting.Request$2.run(Request.java:369)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          	at java.base/java.lang.Thread.run(Thread.java:834)
          Caused: java.io.IOException: Could not perform submodule update
          	at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:153)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1231)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          	at java.base/java.lang.Thread.run(Thread.java:834)
          

          Prior Failures Everywhere

          Failures were happening in that acceptance test prior to my creating a branch on the submodule repository which pointed to the SHA-1 referenced by the parent repository. Once that branch was created, then the newer command line git versions would consistently pass. That earlier failure message was:

          hudson.plugins.git.GitException: Command "git submodule update --reference /var/lib/git/mwaite/bugs/jenkins-bugs.git --depth=1 modules/JENKINS-46504.url" returned status code 1:
          stdout: 
          stderr: Cloning into '/mnt/xvdba/home/jagent/mark-pc2.markwaite.net-agent/workspace/ine-no-localbranch_JENKINS-21248/modules/JENKINS-46504.url'...
          warning: no common commits
          error: Server does not allow request for unadvertised object 0736ba35a0d8c05236e3b71584bc4e149aa5f10a
          Fetched in submodule path 'modules/JENKINS-46504.url', but it did not contain 0736ba35a0d8c05236e3b71584bc4e149aa5f10a. Direct fetching of that commit failed.
          
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2298)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1910)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1334)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
          	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
          	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:75)
          	at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:64)
          
          Show
          markewaite Mark Waite added a comment - I've further investigated the JENKINS-21248 acceptance test failures in my environment. The initial submodule checkout succeeds on the following versions of command line git: 2.21 2.20 2.17 2.16 The initial submodule checkout fails on the following versions of command line git: 2.11 2.10 2.7 2.1 The stack trace in the checkout failure is: hudson.plugins.git.GitException: Command "git submodule update --reference /var/lib/git/mwaite/bugs/jenkins-bugs.git --depth=1 modules/JENKINS-46504.url" returned status code 1: stdout: stderr: Cloning into 'modules/JENKINS-46504.url'... warning: no common commits fatal: reference is not a tree: 0736ba35a0d8c05236e3b71584bc4e149aa5f10a Unable to checkout '0736ba35a0d8c05236e3b71584bc4e149aa5f10a' in submodule path 'modules/JENKINS-46504.url' at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2298) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1910) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1334) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184) at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:75) at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:64) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to debian8-mwaite at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) at jdk.internal.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) at com.sun.proxy.$Proxy119.execute(Unknown Source) at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:148) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1231) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused: hudson.plugins.git.GitException at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:87) at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:68) at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1337) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused: java.io.IOException: Could not perform submodule update at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:153) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1231) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Prior Failures Everywhere Failures were happening in that acceptance test prior to my creating a branch on the submodule repository which pointed to the SHA-1 referenced by the parent repository. Once that branch was created, then the newer command line git versions would consistently pass. That earlier failure message was: hudson.plugins.git.GitException: Command "git submodule update --reference /var/lib/git/mwaite/bugs/jenkins-bugs.git --depth=1 modules/JENKINS-46504.url" returned status code 1: stdout: stderr: Cloning into '/mnt/xvdba/home/jagent/mark-pc2.markwaite.net-agent/workspace/ine-no-localbranch_JENKINS-21248/modules/JENKINS-46504.url'... warning: no common commits error: Server does not allow request for unadvertised object 0736ba35a0d8c05236e3b71584bc4e149aa5f10a Fetched in submodule path 'modules/JENKINS-46504.url', but it did not contain 0736ba35a0d8c05236e3b71584bc4e149aa5f10a. Direct fetching of that commit failed. at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2298) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1910) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1334) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184) at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:75) at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:64)
          Hide
          sverhoef Stefan Verhoeff added a comment -

          Thanks for the details Mark Waite. I'm happy to help with testing this useful feature.

          How can I check what version of Git is used by the checkout() command? I assume it would be the same I get when I run git --version just above it. Is there a different Git binary used? Is the checkout actually happening on the Master and the sh() command on the agent? Is JGit involved? I'm confused here.

          As per your acceptance test, does this point at an issue with the version parsing or not? I've checked the code around where the warning Git client older than 1.8.4 doesn't support shallow submodule updates. This flag is ignored. is raised, the shallow clone arguments are never actually added to the Git command because of the failed version check, so I'm not getting errors from Git on this.

          Show
          sverhoef Stefan Verhoeff added a comment - Thanks for the details Mark Waite . I'm happy to help with testing this useful feature. How can I check what version of Git is used by the  checkout() command? I assume it would be the same I get when I run git --version just above it. Is there a different Git binary used? Is the checkout actually happening on the Master and the  sh() command on the agent? Is JGit involved? I'm confused here. As per your acceptance test, does this point at an issue with the version parsing or not? I've checked the code around where the warning Git client older than 1.8.4 doesn't support shallow submodule updates. This flag is ignored. is raised, the shallow clone arguments are never actually added to the Git command because of the failed version check, so I'm not getting errors from Git on this.
          Hide
          markewaite Mark Waite added a comment -

          My acceptance test is strong evidence that there is not an issue with version parsing .

          Check the jenkins system configuration of the git tool and the agent configuration of the git tool

          Show
          markewaite Mark Waite added a comment - My acceptance test is strong evidence that there is not an issue with version parsing . Check the jenkins system configuration of the git tool and the agent configuration of the git tool

            People

            • Assignee:
              Unassigned
              Reporter:
              monaka Masaki Muranaka
            • Votes:
              22 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: