Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44323

"MissingObjectException: Missing commit" - no issues reported by git binary

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
    • Environment:
      2.46.2
      Master: Ubuntu 16.04.2
      Slave: OS X 10.12.4
    • Similar Issues:

      Description

      We are trying to move our build system to use the Github hook APIs rather than being based on a periodic build, however, we can't get our repository to check out when the hook is fired. 

       
      When the hook is fired, Jenkins checks for changes to the repo. This fails with the error:

      org.eclipse.jgit.errors.MissingObjectException: Missing commit [some commit hash]
      at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:159)
      at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:903)
      at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:155)
      at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:147)
      at org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:184)
      at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:435)

       

      This happens every time we get a hook call, and it fails on the same commit. I checked the commit to make sure it exists and that there was nothing weird about it. I ran `git fsck --full --strict` and there was nothing about this commit, just a few dangling ones. 

      However, when I changed the global configuration to change the git tool from `git` to `/usr/local/bin/git` (The homebrew version - I'm sure /usr/bin/git would also work), the check succeeds and changes are detected, kicking off a build. The problem is that when the build starts, and it does its git operations, I get the same crash as above in jgit.

      This is a total blocker for us since we can't even do the clone, let alone a build.

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          That sounds like you may have globally configured Jenkins (or that job) to use JGit as the implementation. You might try using command line git as the implementation for that job. Refer to the job configuration page and look for the "implementation" field where it may contain either "jgit" or "jgitapache". Switch it to "Default" to see if that provides you a working job.

          Show
          markewaite Mark Waite added a comment - That sounds like you may have globally configured Jenkins (or that job) to use JGit as the implementation. You might try using command line git as the implementation for that job. Refer to the job configuration page and look for the "implementation" field where it may contain either "jgit" or "jgitapache". Switch it to "Default" to see if that provides you a working job.
          Hide
          dalemy Dale Myers added a comment -

          In the "Global Tool Configuration" I have one named "Default" set to "/usr/local/bin/git", but there is no special configuration of any kind in the job.

          Show
          dalemy Dale Myers added a comment - In the "Global Tool Configuration" I have one named "Default" set to "/usr/local/bin/git", but there is no special configuration of any kind in the job.
          Hide
          markewaite Mark Waite added a comment -

          You may also want to check that the "refspec" for the job has not been modified from the default. Yet another check is to navigate into the folder structure of the agent executing the build, find the git repository, and check that the specific SHA1 is in that repository. I suspect it is not, though I don't have any immediate idea why it is not.

          Show
          markewaite Mark Waite added a comment - You may also want to check that the "refspec" for the job has not been modified from the default. Yet another check is to navigate into the folder structure of the agent executing the build, find the git repository, and check that the specific SHA1 is in that repository. I suspect it is not, though I don't have any immediate idea why it is not.
          Hide
          dalemy Dale Myers added a comment - - edited

          So it seems that this is caused by doing a shallow clone. When I change it to do a full clone (and specify the path to git), the issue goes away. It's still not clear why though?

          Edit: Seems like turning off the shallow clone only bought a temporary reprieve. It managed to get to that commit again and the builds failed.

          Show
          dalemy Dale Myers added a comment - - edited So it seems that this is caused by doing a shallow clone. When I change it to do a full clone (and specify the path to git), the issue goes away. It's still not clear why though? Edit: Seems like turning off the shallow clone only bought a temporary reprieve. It managed to get to that commit again and the builds failed.
          Hide
          markewaite Mark Waite added a comment -

          Shallow clone does not copy all the references from the remote repository to the local repository. Without all the remote references, there is no assurance that the reference you're seeking is in the repository.

          If your repository is large and you have some file system control of your agents, you could consider using a reference repository to reduce the data transfer time (and volume) between your central repository and the agent. The first 10 minutes of this youtube video provide an overview of some options to reduce git data transfer and clone time.

          Show
          markewaite Mark Waite added a comment - Shallow clone does not copy all the references from the remote repository to the local repository. Without all the remote references, there is no assurance that the reference you're seeking is in the repository. If your repository is large and you have some file system control of your agents, you could consider using a reference repository to reduce the data transfer time (and volume) between your central repository and the agent. The first 10 minutes of this youtube video provide an overview of some options to reduce git data transfer and clone time.
          Hide
          shiqiyang Shiqi Yang added a comment -

          this happened to me when I set the refspec to '/dev-' and have the build triggered by webhook
          if I squash and force push my dev branch (e.g. dev-shiqi) then jenkins will report that the sha it built last time is missing.

          Show
          shiqiyang Shiqi Yang added a comment - this happened to me when I set the refspec to ' /dev- ' and have the build triggered by webhook if I squash and force push my dev branch (e.g. dev-shiqi) then jenkins will report that the sha it built last time is missing.
          Hide
          markewaite Mark Waite added a comment -

          Shiqi Yang I suspect that the workspace copy of the repository is truly missing that commit for that SHA1 in your case.  I believe it is missing because the refspec you listed would not pull that SHA1 into the repository.

          The git plugin has a long-standing behavior that it ignores the refspec on initial clone (unless the "Advanced clone option" is set to "Honor refspec on initial clone"), but then it honors the refspec on updates.

          As far as I can tell from the git refspec man page, your refspec won't match the branch names you've described, because (quoting the refspec man page)

          You can’t use partial globs in the pattern

          If I've understood correctly, your case would be that the initial clone of the repository (which ignored the refspec and cloned all references), then later updates applied the refspec and pulled no additional commits (because none matched the refspec you provided).

          I'd further guess that the webhook you're calling may be passing a "sha1=" parameter which the git plugin (incorrectly) tries to build even when it detected no change for that sha1.

          Show
          markewaite Mark Waite added a comment - Shiqi Yang I suspect that the workspace copy of the repository is truly missing that commit for that SHA1 in your case.  I believe it is missing because the refspec you listed would not pull that SHA1 into the repository. The git plugin has a long-standing behavior that it ignores the refspec on initial clone (unless the "Advanced clone option" is set to "Honor refspec on initial clone"), but then it honors the refspec on updates. As far as I can tell from the git refspec man page , your refspec won't match the branch names you've described, because (quoting the refspec man page) You can’t use partial globs in the pattern If I've understood correctly, your case would be that the initial clone of the repository (which ignored the refspec and cloned all references), then later updates applied the refspec and pulled no additional commits (because none matched the refspec you provided). I'd further guess that the webhook you're calling may be passing a "sha1=" parameter which the git plugin (incorrectly) tries to build even when it detected no change for that sha1.
          Hide
          blaqqq Karl added a comment -

          got same issue. I used a git mirror repository for checkout and forgot to mount in the mirror into the docker container. if you run git log in your jenkins workspace, then error message should be obvious.  

           

          jenkins-slave@25c2c65b195c:~/workspace/pipeline-build-docker-image-JIM57YFX4XMVSBFUAN6C666QN3TVIALKUUMW7KNLVZPWSEOUCCZA$ git log
          error: object directory /home/jenkins-slave/workspace/git-mirror-products/products.git/objects does not exist; check .git/objects/info/alternates.
          fatal: bad object HEAD

           

          Show
          blaqqq Karl added a comment - got same issue. I used a git mirror repository for checkout and forgot to mount in the mirror into the docker container. if you run git log in your jenkins workspace, then error message should be obvious.     jenkins-slave@25c2c65b195c:~/workspace/pipeline-build-docker-image-JIM57YFX4XMVSBFUAN6C666QN3TVIALKUUMW7KNLVZPWSEOUCCZA$ git log error: object directory /home/jenkins-slave/workspace/git-mirror-products/products.git/objects does not exist; check .git/objects/info/alternates. fatal: bad object HEAD  

            People

            • Assignee:
              Unassigned
              Reporter:
              dalemy Dale Myers
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: