Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63549

First PR build always fails with "fatal: bad object" on a global pipeline git-clone

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.235.5
    • Similar Issues:

      Description

      We use a declarative pipeline that is stored on our Bitbucket server and fetch on every run (default Jenkins functionality).

      Every first PR build fails. There is unfortunately no further information on why it failed. We do see the following on every first PR build:
       

      fatal: bad object 97264b302405d60800f1fe3724963fa18afea158
      

      Any further build within the same PR does not include that message anymore.

      Switching the jenkins-master durability from MAX_SURVIVABILITY to PERFORMANCE_OPTIMIZED solves the issue for us. The "fatal bad object" message remains in the logs.

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment - - edited

          Thanks for reporting the issue.

          Does the issue resolve itself if you enable the "Preserve second fetch during checkout" global configuration? If it does, then we need the console log of a job before the upgrade and a console log of a job after the upgrade to understand if there is an undetected reliance on the redundant fetch in the case that you're seeing.

          What was the version of the git plugin before the upgrade?

          What is the refspec that is being used to fetch the commits from the remote?

          Is the declarative pipeline in a multibranch pipeline or a pipeline that is not multibranch?

          If it is a multibranch pipeline, is the multibranch pipeline using Bitbucket as the multibranch provider or is it using Git as the multibranch provider?

          Is the declarative pipeline being processed as part of a Bitbucket pull request or are they being run on a long-lived branch?

          Can you provide enough details of the configuration so that someone else can duplicate the problem?

          Show
          markewaite Mark Waite added a comment - - edited Thanks for reporting the issue. Does the issue resolve itself if you enable the " Preserve second fetch during checkout " global configuration? If it does, then we need the console log of a job before the upgrade and a console log of a job after the upgrade to understand if there is an undetected reliance on the redundant fetch in the case that you're seeing. What was the version of the git plugin before the upgrade? What is the refspec that is being used to fetch the commits from the remote? Is the declarative pipeline in a multibranch pipeline or a pipeline that is not multibranch? If it is a multibranch pipeline, is the multibranch pipeline using Bitbucket as the multibranch provider or is it using Git as the multibranch provider? Is the declarative pipeline being processed as part of a Bitbucket pull request or are they being run on a long-lived branch? Can you provide enough details of the configuration so that someone else can duplicate the problem?
          Hide
          lifeofguenter Gunter Grodotzki added a comment - - edited

          > Does the issue resolve itself if you enable the "Preserve second fetch during checkout" global configuration? 

           

          Awesome thanks, will try this out!

           

          > What was the version of the git plugin before the upgrade?

           

          we rebuild our jenkins-masters weekly based off the official jenkins-lts and we do not version pin-point plugins. So it would be: 

          git: 4.3.0

          git-client: 3.4.1

           

          > What is the refspec that is being used to fetch the commits from the remote?

           

           > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
           Avoid second fetch
           Checking out Revision 5f77a5e1868e2c332f6ff05743c9a25dcb7737da (master)
           > git config core.sparsecheckout # timeout=10
           > git checkout -f 5f77a5e1868e2c332f6ff05743c9a25dcb7737da # timeout=10
          
          

           

           

          thats at the top, a bit below the error:

          [Pipeline] { (Declarative: Checkout SCM)[Pipeline] checkoutThe recommended git tool is: NONE
           using credential git
           Cloning the remote Git repository
           Cloning with configured refspecs honoured and without tags
           Fetching without tags
           Checking out Revision 96a4f1960530748195a9f3f818ed40385fc83ef5 (PR-18)
           Commit message: "DRI-5696: enable prod"
           First time build. Skipping changelog.
           > git --version # timeout=10
           > git --version # 'git version 2.25.1'
           fatal: bad object 5f77a5e1868e2c332f6ff05743c9a25dcb7737da
          
          

           

          > Is the declarative pipeline in a multibranch pipeline or a pipeline that is not multibranch?

           

          multibranch / Bitbucket server organization folder

           

          > Is the declarative pipeline being processed as part of a Bitbucket pull request or are they being run on a long-lived branch?

           

          the declarative pipeline gets triggered on PRs and master-branch pushes (mainly) - we have so far only noticed this issue on PRs

           

          > Can you provide enough details of the configuration so that someone else can duplicate the problem?
           
          good point! it happens very intermittent so not sure how easy it will be to replicate but will see if I can somehow isolate it

          Show
          lifeofguenter Gunter Grodotzki added a comment - - edited > Does the issue resolve itself if you enable the " Preserve second fetch during checkout " global configuration?    Awesome thanks, will try this out!   > What was the version of the git plugin before the upgrade?   we rebuild our jenkins-masters weekly based off the official jenkins-lts and we do not version pin-point plugins. So it would be:  git: 4.3.0 git-client: 3.4.1   > What is the refspec that is being used to fetch the commits from the remote?   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 5f77a5e1868e2c332f6ff05743c9a25dcb7737da (master) > git config core.sparsecheckout # timeout=10 > git checkout -f 5f77a5e1868e2c332f6ff05743c9a25dcb7737da # timeout=10     thats at the top, a bit below the error: [Pipeline] { (Declarative: Checkout SCM)[Pipeline] checkoutThe recommended git tool is: NONE using credential git Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Fetching without tags Checking out Revision 96a4f1960530748195a9f3f818ed40385fc83ef5 (PR-18) Commit message: "DRI-5696: enable prod" First time build. Skipping changelog. > git --version # timeout=10 > git --version # 'git version 2.25.1' fatal: bad object 5f77a5e1868e2c332f6ff05743c9a25dcb7737da   > Is the declarative pipeline in a multibranch pipeline or a pipeline that is not multibranch?   multibranch / Bitbucket server organization folder   > Is the declarative pipeline being processed as part of a Bitbucket pull request or are they being run on a long-lived branch?   the declarative pipeline gets triggered on PRs and master-branch pushes (mainly) - we have so far only noticed this issue on PRs   > Can you provide enough details of the configuration so that someone else can duplicate the problem?   good point! it happens very intermittent so not sure how easy it will be to replicate but will see if I can somehow isolate it
          Hide
          markewaite Mark Waite added a comment - - edited

          The "fatal: bad object" in that log seems to be a reference to a commit on the master branch, likely in order to merge the master branch and the pull request. If you can locate one of the failing workspaces on an agent, I suspect you'll find that the refspec being used in that workspace does not include the master branch. I believe you can list the refspec in the workspace repository with the command `git config --list`.

          The message that says "The recommended git tool is: NONE" tells us that the Bitbucket branch source has not yet implemented the new git plugin extension to provide a size estimate for the remote repository. That's not an issue, but it does indicate that the new git tool chooser is not likely to be involved in this issue, since it won't make a recommendation of git implementation unless it has a source that can estimate the repository size.

          When pasting log examples, it helps them be more readable if you surround them with the 'noformat' tag in braces like

          {noformat}
          some log
          {noformat}

          Otherwise, Jira sees the '*' character in a refspec and renders it as bold text.

          Show
          markewaite Mark Waite added a comment - - edited The "fatal: bad object" in that log seems to be a reference to a commit on the master branch, likely in order to merge the master branch and the pull request. If you can locate one of the failing workspaces on an agent, I suspect you'll find that the refspec being used in that workspace does not include the master branch. I believe you can list the refspec in the workspace repository with the command `git config --list`. The message that says "The recommended git tool is: NONE" tells us that the Bitbucket branch source has not yet implemented the new git plugin extension to provide a size estimate for the remote repository. That's not an issue, but it does indicate that the new git tool chooser is not likely to be involved in this issue, since it won't make a recommendation of git implementation unless it has a source that can estimate the repository size. When pasting log examples, it helps them be more readable if you surround them with the 'noformat' tag in braces like {noformat} some log {noformat} Otherwise, Jira sees the '*' character in a refspec and renders it as bold text.
          Hide
          lifeofguenter Gunter Grodotzki added a comment -

          Ah ok.. so:

           

          • enabling "preserve second..." option globally did not help unfortunately
          • we also seen this now on master builds
          • our PRs are built on HEAD - master does not get involved
          • the global pipeline is on a separate repo, there should be zero cross-over with the project build - e.g. yes, it won't find that commit in the project repo or vice versa (which is why its failing? but why is it doing that now?)

           

           

          Show
          lifeofguenter Gunter Grodotzki added a comment - Ah ok.. so:   enabling "preserve second..." option globally did not help unfortunately we also seen this now on master builds our PRs are built on HEAD - master does not get involved the global pipeline is on a separate repo, there should be zero cross-over with the project build - e.g. yes, it won't find that commit in the project repo or vice versa (which is why its failing? but why is it doing that now?)    
          Hide
          markewaite Mark Waite added a comment -

          Were other plugins updated at the same time as the git plugin update to 4.4.0? For example, if the Bitbucket plugin changed the refspec it is using to fetch the repository in the agent workspace, it could exclude the fetch of the master branch when fetching into the agent workspace.

          If you manually upload git plugin 4.3.x to the Jenkins server, does that resolve the issue?

          Show
          markewaite Mark Waite added a comment - Were other plugins updated at the same time as the git plugin update to 4.4.0? For example, if the Bitbucket plugin changed the refspec it is using to fetch the repository in the agent workspace, it could exclude the fetch of the master branch when fetching into the agent workspace. If you manually upload git plugin 4.3.x to the Jenkins server, does that resolve the issue?
          Hide
          lifeofguenter Gunter Grodotzki added a comment - - edited

          for now we manually rolled back both git-client + git to the previous version - so far its ok, will wait a couple of days  

           

          @@ -14,7 +14,7 @@
           blueocean-config:1.23.2
           blueocean-core-js:1.23.2
           blueocean-dashboard:1.23.2
          -blueocean-display-url:2.3.1
          +blueocean-display-url:2.4.0
           blueocean-events:1.23.2
           blueocean-git-pipeline:1.23.2
           blueocean-github-pipeline:1.23.2
          @@ -51,9 +51,9 @@
           extended-read-permission:3.2
           external-monitor-job:1.7
           favorite:2.3.2
          -git-client:3.4.1
          +git-client:3.4.2
           git-server:1.9
          -git:4.3.0
          +git:4.4.0
           github-api:1.115
           github-branch-source:2.8.3
           github:1.31.0
          @@ -95,7 +95,7 @@
           pipeline-stage-tags-metadata:1.7.1
           pipeline-stage-view:2.14
           plain-credentials:1.7
          -plugin-util-api:1.2.2
          +plugin-util-api:1.2.4
           pubsub-light:1.13
           resource-disposer:0.14
           role-strategy:3.0

           

          Show
          lifeofguenter Gunter Grodotzki added a comment - - edited for now we manually rolled back both git-client + git to the previous version - so far its ok, will wait a couple of days     @@ -14,7 +14,7 @@ blueocean-config:1.23.2 blueocean-core-js:1.23.2 blueocean-dashboard:1.23.2 -blueocean-display-url:2.3.1 +blueocean-display-url:2.4.0 blueocean-events:1.23.2 blueocean-git-pipeline:1.23.2 blueocean-github-pipeline:1.23.2 @@ -51,9 +51,9 @@ extended-read-permission:3.2 external-monitor-job:1.7 favorite:2.3.2 -git-client:3.4.1 +git-client:3.4.2 git-server:1.9 -git:4.3.0 +git:4.4.0 github-api:1.115 github-branch-source:2.8.3 github:1.31.0 @@ -95,7 +95,7 @@ pipeline-stage-tags-metadata:1.7.1 pipeline-stage-view:2.14 plain-credentials:1.7 -plugin-util-api:1.2.2 +plugin-util-api:1.2.4 pubsub-light:1.13 resource-disposer:0.14 role-strategy:3.0  
          Hide
          lifeofguenter Gunter Grodotzki added a comment -

          even after downgrading I am still having the issue. So I don't think this is related to the git-plugin recent changes

           

          I recently did some changes in the jenkins pipeline which might have caused this. Strange though that its happening intermittently, but usually on the first PR.

          Show
          lifeofguenter Gunter Grodotzki added a comment - even after downgrading I am still having the issue. So I don't think this is related to the git-plugin recent changes   I recently did some changes in the jenkins pipeline which might have caused this. Strange though that its happening intermittently, but usually on the first PR.
          Hide
          lifeofguenter Gunter Grodotzki added a comment -

          updated the issue slightly. Something is wrong on another level - I unfortunately can not further figure out what.

          Show
          lifeofguenter Gunter Grodotzki added a comment - updated the issue slightly. Something is wrong on another level - I unfortunately can not further figure out what.

            People

            • Assignee:
              Unassigned
              Reporter:
              lifeofguenter Gunter Grodotzki
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: