Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55457

Concurrent git checkouts confuse SHAs between location of Jenkinsfile and the repos its building

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Labels:
      None
    • Environment:
      * Jenkins core 2.156
      * All experimental update center level plugins, including git and git-client betas
    • Similar Issues:

      Description

      Summary
      Declarative Pipeline jobs which use a set of parallel stages to concurrently download several branches of a different repository via the checkout step can fail. The appearance is that commit hashes get confused between the two repositories. The Jenkinsfile is stored in SCM, and Git Client is expecting the commit SHA for the Jenkinsfile, but instead reads the commit SHA for one of checkout steps in the pipeline itself.

      Frequency
      100% of the time. But this has been a real devil to try and explain...

      What this test is doing

      • This Jenkinsfile (which can be seen here) does a checkout of each individual branch of this repository.
      • Of the five branches that get checked out, three are using jgit as their checkout tool, and two are using Default.
      • Each branch that gets checked out is full of little more than a lot of ASCII gibberish. This is to make sure the checkouts take a long enough time that the pipeline stages will actually execute in parallel for more than a fraction of a second.

      Steps to recreate
      1. Create a new Pipeline job. Use the following settings:

      • Definition: Pipeline script from SCM
      • SCM: Git
      • Repository URL: https://github.com/kshultzCB/empty-repo.git
      • Credentials: Any credentials you already have set up for public GitHub repos should work fine. These repos are all public.
      • Branches to build: version-for-github-no-submodules . This is important, the other branches will do checkout s on a different repo.
      • The remaining settings are all left to default:
        • Git executable: Default
        • Repository browser: (Auto)
        • Additional behaviors: (None set)
        • Script path: Jenkinsfile
        • Lightweight checkout: checked

      2. Save the Pipeline

      3. Click "Build Now"

      4. Watch the build process via the streaming console log. The first sign of trouble looks to be a GitException, which reads:

      ERROR: Error fetching remote repo 'origin'
      hudson.plugins.git.GitException: Failed to fetch from https://github.com/kshultzCB/repo_without_submodules.git
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:903)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1130)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1161)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:51)
      	at hudson.security.ACL.impersonate(ACL.java:290)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:48)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/kshultzCB/repo_without_submodules.git +refs/heads/*:refs/remotes/origin/*" returned status code 1:
      stdout: 
      stderr: warning: no common commits
      

      5. The job continues to execute. Eventually, we land where it looks, to me, like the commit SHA for the Jenkinsfile is being confused with the commit SHA for one of the branches we are trying to check out. Take a look at this log snippet:

      From https://github.com/kshultzCB/repo_without_submodules
       * [new branch]      1-lots-of-stuff -> origin/1-lots-of-stuff
       * [new branch]      2-lots-of-stuff -> origin/2-lots-of-stuff
       * [new branch]      3-lots-of-stuff -> origin/3-lots-of-stuff
       * [new branch]      4-lots-of-stuff -> origin/4-lots-of-stuff
      error: Ref refs/remotes/origin/master is at 780de2489767b850181b1eff22051633ecc376e7 but expected 52fd1c9cf81f1f38e4cca47fb4f1c995967d6201
       ! 52fd1c9...780de24 master     -> origin/master  (unable to update local ref)
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2294)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1881)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:83)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:468)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	... 4 more
      

      This is the line that's interesting:

      Error: Ref refs/remotes/origin/master is at 780de2489767b850181b1eff22051633ecc376e7 but expected 52fd1c9cf81f1f38e4cca47fb4f1c995967d6201
       ! 52fd1c9...780de24 master     -> origin/master  (unable to update local ref)
      

      It took a lot of scratching my head, but here's what's going on.

      A complete log from running this test can be seen here.

      I've set the component to both git-plugin and git-client-plugin, since there are errors shown from both.

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Thanks for the report and for testing the plugin!

          The checkout step does not know about the shell step that performed the cd command. Thus all the checkout steps are being performed in parallel in the same working directory. Chaos ensues (interesting test of command line git locking a directory concurrently and jgit locking a directory concurrently).

          You need to place the checkout (and probably the predecessor steps) in a ws block or a dir block. For an example, refer to the examples in

          Show
          markewaite Mark Waite added a comment - Thanks for the report and for testing the plugin! The checkout step does not know about the shell step that performed the cd command. Thus all the checkout steps are being performed in parallel in the same working directory. Chaos ensues (interesting test of command line git locking a directory concurrently and jgit locking a directory concurrently). You need to place the checkout (and probably the predecessor steps) in a ws block or a dir block. For an example, refer to the examples in Jenkinsfile 1 Jenkinsfile 2
          Hide
          kshultz Karl Shultz added a comment -

          I had a boo-boo in my Jenkinsfile.

          Show
          kshultz Karl Shultz added a comment - I had a boo-boo in my Jenkinsfile.

            People

            • Assignee:
              markewaite Mark Waite
              Reporter:
              kshultz Karl Shultz
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: