Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46344

During git checkout, jgit core crashes on a stackoverflow on an infinite openPackedFromSelfOrAlternate loop.

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: git-client-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      We are having intermittent problems checkout out some of our repositories. We're not entirely sure what happens, but it seems that when a particular commit is being present in the branch history, the checkout will fail in the following way (redacted):

      ---8<---------
      (initialisation)
      ...
      (checkout of other repositories)
      ...
      Cloning the remote Git repository
      Cloning repository git+ssh://git@git.company.com/reponame
      > C:\Program Files\Git\cmd\git.exe init ...\workspace\product\git\reponame # timeout=10
      Fetching upstream changes from git+ssh://git@git.company.com/reponame
      > C:\Program Files\Git\cmd\git.exe --version # timeout=10
      using GIT_SSH to set credentials 
       > C:\Program Files\Git\cmd\git.exe fetch --tags --progress git+ssh://git@git.company.com/reponame +refs/heads/*:refs/remotes/origin/* # timeout=20
      > C:\Program Files\Git\cmd\git.exe config remote.origin.url git+ssh://git@git.company.com/reponame # timeout=10
      > C:\Program Files\Git\cmd\git.exe config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
      > C:\Program Files\Git\cmd\git.exe config remote.origin.url git+ssh://git@git.company.com/reponame # timeout=10
      Fetching upstream changes from git+ssh://git@git.company.com/reponame
      using GIT_SSH to set credentials 
       > C:\Program Files\Git\cmd\git.exe fetch --tags --progress git+ssh://git@git.company.com/reponame +refs/heads/*:refs/remotes/origin/* # timeout=20
      > C:\Program Files\Git\cmd\git.exe rev-parse "origin/bugfix-branch^\{commit}" # timeout=10
      Checking out Revision 721397010a24a218e8619e07645d99afe0fabb52 (origin/bugfix-branch)
      java.lang.StackOverflowError
                      at java.lang.ThreadLocal.get(Unknown Source)
                      at sun.nio.fs.NativeBuffers.getNativeBufferFromCache(Unknown Source)
                      at sun.nio.fs.WindowsNativeDispatcher.asNativeBuffer(Unknown Source)
                      at sun.nio.fs.WindowsNativeDispatcher.GetFileAttributesEx(Unknown Source)
                      at sun.nio.fs.WindowsFileAttributes.get(Unknown Source)
                      at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(Unknown Source)
                      at sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(Unknown Source)
                      at sun.nio.fs.WindowsFileSystemProvider.readAttributes(Unknown Source)
                      at java.nio.file.Files.readAttributes(Unknown Source)
                      at java.nio.file.Files.getLastModifiedTime(Unknown Source)
                      at org.eclipse.jgit.util.FileUtils.lastModified(FileUtils.java:563)
                      at org.eclipse.jgit.util.FS.lastModified(FS.java:299)
                      at org.eclipse.jgit.internal.storage.file.FileSnapshot.isModified(FileSnapshot.java:164)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.searchPacksAgain(ObjectDirectory.java:688)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:435)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:390)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      ... (snip 999 repeats)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:394)
                      at ......remote call to Channel to /<ip-address>(Native Method)
                      at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1554)
                      at hudson.remoting.UserResponse.retrieve(UserRequest.java:281)
                      at hudson.remoting.Channel.call(Channel.java:839)
      Caused: java.io.IOException: Remote call on Channel to /<ip-address> failed
                      at hudson.remoting.Channel.call(Channel.java:847)
                      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
                      at com.sun.proxy.$Proxy97.withRepository(Unknown Source)
                      at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
                      at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1195)
                      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)
                      at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
                      at hudson.scm.SCM.checkout(SCM.java:495)
                      at hudson.model.AbstractProject.checkout(AbstractProject.java:1212)
                      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:560)
                      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
                      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:485)
                      at hudson.model.Run.execute(Run.java:1737)
                      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                      at hudson.model.ResourceController.execute(ResourceController.java:97)
                      at hudson.model.Executor.run(Executor.java:419)
      Sending e-mails to: developers@company.com
      [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
      Finished: FAILURE
      --->8---------
      

      So far the only way we've found to resolve the problem is to omit (note: not revert, that does not help) the commit in question from the branch's history. However, this commit contains a critical bug fix and is required to be released on this branch, so simply leaving it out is not an option for us. Moreover, we are worried that this problem may start cropping up on other branches too.

      We suspect that that the problem is with the version of JGit Core that is used by the Jenkins Git client plugin. Looking at the changes to JGit Core, we see the following change which fixes something which sounds suspiciously like the problem we are currently experiencing:

      However, we don't really see any 'alternates' files in the repository's .git folder so we're not 100% certain that this is really the problem, although we admit that we are not familiar with Git internals.

      PS. we found a workaround, we made a new reference checkout on the build machine with the problem and it the crashes have vanished. We do still have the old one so expect to be able to reproduce again.

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Since the issue seems to be in JGit handling of the alternates file, you might be able to avoid the problem by not using a reference repository. As far as I've observed, the alternates file is used with a reference repository.

          The JGit fix you referenced is first available in JGit 4.8.0. JGit 4.8.0 requires Java 8. The git plugin is intentionally maintaining compatibility with Java 7 until it makes the switch (in git client plugin 3.0 and git plugin 4.0) to require Java 8 and Jenkins 2.60 or later. Because the git plugin is used by so many Jenkins users, and in so many different versions, I rarely change the base Jenkins version on which it depends.

          A working branch is available now where I've been prototyping the switch to Java 8 and JGit 4.8.0. If you'd like to try it, you could clone the git plugin and the git client plugin, checkout the java-8 branches, and build the plugins from those branches. It might even be enough to only build git-client-plugin with the java-8 branch.

          It will likely be another 2-3 months before git client plugin 3.0 and git plugin 4.0 are released.

          Show
          markewaite Mark Waite added a comment - Since the issue seems to be in JGit handling of the alternates file, you might be able to avoid the problem by not using a reference repository. As far as I've observed, the alternates file is used with a reference repository. The JGit fix you referenced is first available in JGit 4.8.0. JGit 4.8.0 requires Java 8. The git plugin is intentionally maintaining compatibility with Java 7 until it makes the switch (in git client plugin 3.0 and git plugin 4.0) to require Java 8 and Jenkins 2.60 or later. Because the git plugin is used by so many Jenkins users, and in so many different versions, I rarely change the base Jenkins version on which it depends. A working branch is available now where I've been prototyping the switch to Java 8 and JGit 4.8.0. If you'd like to try it, you could clone the git plugin and the git client plugin, checkout the java-8 branches, and build the plugins from those branches. It might even be enough to only build git-client-plugin with the java-8 branch. It will likely be another 2-3 months before git client plugin 3.0 and git plugin 4.0 are released.
          Hide
          markewaite Mark Waite added a comment -

          A git client plugin pre-release build is available which uses JGit 4.10.0. It can be used with Jenkins 2.60.1 and later. I'd love to hear confirmation that the pre-release of that plugin resolves this issue.

          Show
          markewaite Mark Waite added a comment - A git client plugin pre-release build is available which uses JGit 4.10.0. It can be used with Jenkins 2.60.1 and later. I'd love to hear confirmation that the pre-release of that plugin resolves this issue.
          Hide
          markewaite Mark Waite added a comment -

          Fixed in git client plugin 3.0.0-beta1. Please test your use case (and other use cases) with git client 3.0.0-beta1 so that we build confidence the release does not regress from previous versions.

          Show
          markewaite Mark Waite added a comment - Fixed in git client plugin 3.0.0-beta1. Please test your use case (and other use cases) with git client 3.0.0-beta1 so that we build confidence the release does not regress from previous versions.
          Hide
          markewaite Mark Waite added a comment -

          Fixed in git client plugin 3.0.0

          Show
          markewaite Mark Waite added a comment - Fixed in git client plugin 3.0.0

            People

            • Assignee:
              Unassigned
              Reporter:
              wilcobt Wilco Belgraver Thissen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: