Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51288

Builds can't checkout any more: Delete branch returned unexpected result LOCK_FAILURE

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: git-client-plugin
    • Labels:
      None
    • Environment:
      git-plugin: 3.9.0
      git-client-plugin: 2.7.2
      Jenkins: 2.121
    • Similar Issues:

      Description

      Since the Git updates today Multibranch Pipeline builds (at least, haven't tried other builds) fail with the exception:

      Setting origin to git@gitlab:xxx/yyy.git
      Fetching origin...
      org.eclipse.jgit.api.errors.JGitInternalException: Delete branch returned unexpected result LOCK_FAILURE
      	at org.eclipse.jgit.api.DeleteBranchCommand.call(DeleteBranchCommand.java:172)
      	at org.jenkinsci.plugins.gitclient.JGitAPIImpl$2.execute(JGitAPIImpl.java:620)
      	at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:394)
      	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:301)
      	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:298)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE
      

      Switching to the git cli implementation is not an option for me because that is not available on all the platforms I require (while JGit is).

        Attachments

          Activity

          Hide
          cobexer Ing. Christoph Obexer added a comment -

          Triggering lots of branch indexing including some caused by pushes I never observe any index.lock files (that a find every second would list) in any git cache repo.
          Branch indexing takes 1.6 seconds (also seen 3.8) so I should have seen such a lock file at least sometimes... unless I'm really unlucky

          Show
          cobexer Ing. Christoph Obexer added a comment - Triggering lots of branch indexing including some caused by pushes I never observe any index.lock files (that a find every second would list) in any git cache repo. Branch indexing takes 1.6 seconds (also seen 3.8) so I should have seen such a lock file at least sometimes... unless I'm really unlucky
          Hide
          cobexer Ing. Christoph Obexer added a comment -

          > Also, when you say that plugin downgrades did not help, can you explain further what steps you took after downgrading the plugins? Did you restart the Jenkins server after downgrading the plugins? What plugin versions were used after the downgrade, etc.?

          I first downgraded the git-client plugin (because that was in the callstack) to version 2.7.1, restarted jenkins and triggered branch indexing which threw the same errors. Then I downgraded the git plugin too (and restarted) but that did not fix the branch indexing errors and added the NPE I added above. I then removed all cached repositories, that fixed the branch indexing but existing branches were still throwing errors:

          Mai 14, 2018 10:22:36 AM jenkins.model.lazy.LazyBuildMixIn newBuild
          WARNUNG: A new build could not be created in job repo/master
          java.lang.IllegalStateException: JENKINS-27530: cannot create a build with number 1 since that (or higher) is already in use among [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
          34, 35, 36, 37, 38]
                  at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:393)
                  at hudson.model.RunMap.put(RunMap.java:192)
                  at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:185)
                  at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.createExecutable(ParameterizedJobMixIn.java:510)
                  at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.createExecutable(ParameterizedJobMixIn.java:320)
                  at hudson.model.Executor$1.call(Executor.java:365)
                  at hudson.model.Executor$1.call(Executor.java:347)
                  at hudson.model.Queue._withLock(Queue.java:1437)
                  at hudson.model.Queue.withLock(Queue.java:1298)
                  at hudson.model.Executor.run(Executor.java:347)
          

          I then upgraded both git plugins again (and restarted jenkins) and at that point my Jenkins started to throw the "IOException: yyy/feature-y #5 did not yet start" errors which I worked around by downgrading the workflow-cps plugin. Since then my builds work again.

          Show
          cobexer Ing. Christoph Obexer added a comment - > Also, when you say that plugin downgrades did not help, can you explain further what steps you took after downgrading the plugins? Did you restart the Jenkins server after downgrading the plugins? What plugin versions were used after the downgrade, etc.? I first downgraded the git-client plugin (because that was in the callstack) to version 2.7.1, restarted jenkins and triggered branch indexing which threw the same errors. Then I downgraded the git plugin too (and restarted) but that did not fix the branch indexing errors and added the NPE I added above. I then removed all cached repositories, that fixed the branch indexing but existing branches were still throwing errors: Mai 14, 2018 10:22:36 AM jenkins.model.lazy.LazyBuildMixIn newBuild WARNUNG: A new build could not be created in job repo/master java.lang.IllegalStateException: JENKINS-27530: cannot create a build with number 1 since that (or higher) is already in use among [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38] at jenkins.model.lazy.AbstractLazyLoadRunMap.proposeNewNumber(AbstractLazyLoadRunMap.java:393) at hudson.model.RunMap.put(RunMap.java:192) at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:185) at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.createExecutable(ParameterizedJobMixIn.java:510) at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.createExecutable(ParameterizedJobMixIn.java:320) at hudson.model.Executor$1.call(Executor.java:365) at hudson.model.Executor$1.call(Executor.java:347) at hudson.model.Queue._withLock(Queue.java:1437) at hudson.model.Queue.withLock(Queue.java:1298) at hudson.model.Executor.run(Executor.java:347) I then upgraded both git plugins again (and restarted jenkins) and at that point my Jenkins started to throw the "IOException: yyy/feature-y #5 did not yet start" errors which I worked around by downgrading the workflow-cps plugin. Since then my builds work again.
          Hide
          markewaite Mark Waite added a comment -

          Thanks for the additional details. Much appreciated.

          Show
          markewaite Mark Waite added a comment - Thanks for the additional details. Much appreciated.
          Hide
          cobexer Ing. Christoph Obexer added a comment -

          Are there any lingering index.lock or other lock files in the caches directory on the Jenkins master? If there are, is the problem resolved by removing those lock files? Is the Jenkins master a Windows machine, and if so, can those lock files be removed while Jenkins is running (in case a lock file is being held open)?

          So this happened again today... I do indeed have index.lock files in 2 out of 15 cached repositories.

          The only thing I did today is update Jenkins and plugins and then restart Jenkins and reboot the whole CentOS 7 vm.

          Removing the index.lock file does NOT immediately resolve the problem. Removing the cached repository resolves the Problem immediately.

          I made a backup of the cached repository this time - but I can't share that, I did not notice any files with unexpected names... any ideas?

          Show
          cobexer Ing. Christoph Obexer added a comment - Are there any lingering index.lock or other lock files in the caches directory on the Jenkins master? If there are, is the problem resolved by removing those lock files? Is the Jenkins master a Windows machine, and if so, can those lock files be removed while Jenkins is running (in case a lock file is being held open)? So this happened again today... I do indeed have index.lock files in 2 out of 15 cached repositories. The only thing I did today is update Jenkins and plugins and then restart Jenkins and reboot the whole CentOS 7 vm. Removing the index.lock file does NOT immediately resolve the problem. Removing the cached repository resolves the Problem immediately. I made a backup of the cached repository this time - but I can't share that, I did not notice any files with unexpected names... any ideas?
          Hide
          markewaite Mark Waite added a comment -

          Unfortunately, I don't have any other suggestions. The index.lock file is created and maintained by command line git.

          Show
          markewaite Mark Waite added a comment - Unfortunately, I don't have any other suggestions. The index.lock file is created and maintained by command line git.

            People

            • Assignee:
              Unassigned
              Reporter:
              cobexer Ing. Christoph Obexer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: