Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25353

Git operations fail due to "dead", but lingering lock file

    Details

    • Similar Issues:

      Description

      Time to time I see the following error. I suppose, it can be caused by job abort in git checkout stage.

      java.io.IOException: Could not checkout 154d73b8218b3a4c0db7808853565ca5ed0b8999
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:966)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1253)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
      	at hudson.model.Run.execute(Run.java:1745)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:89)
      	at hudson.model.Executor.run(Executor.java:240)
      Caused by: hudson.plugins.git.GitLockFailedException: Could not lock repository. Please try again
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1619)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:328)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at hudson.remoting.Engine$1$1.run(Engine.java:63)
      	at java.lang.Thread.run(Unknown Source)
      Caused by: hudson.plugins.git.GitException: Command "C:\Program Files (x86)\Git\bin\git.exe checkout -f 154d73b8218b3a4c0db7808853565ca5ed0b8999" returned status code 128:
      stdout: 
      stderr: fatal: Unable to create 'e:/jenkins/workspace/xxx/.git/index.lock': File exists.
      
      If no other git process is currently running, this probably means a
      git process crashed in this repository earlier. Make sure no other git
      process is running and remove the file manually to continue.
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1437)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1616)
      	... 12 more
      

        Attachments

          Issue Links

            Activity

            Hide
            bulb Jan Hudec added a comment -

            On Windows the lock file can't be deleted while still held, which is what makes unconditional removal safe. Unless somebody is messing up with the Jenkins workspaces by hand, there is no process running in them when build starts, so it still does fix the issue.

            Show
            bulb Jan Hudec added a comment - On Windows the lock file can't be deleted while still held, which is what makes unconditional removal safe . Unless somebody is messing up with the Jenkins workspaces by hand, there is no process running in them when build starts, so it still does fix the issue.
            Hide
            dragos_n_sava Nicolae Dragos Sava added a comment -

            Hi guys, so I manage to reproduce a posible cause for this bug (there could be other cases also):
            I am certain the build triggers are one of the problems: "pull scm" and "build periodically" collide. So the pull scm will work ok but when build periodically comes, the build is going to fail with the lock issue.
            Replicate: have pull scm at 5 minute and build periodically at 5 minute and look what happens.
            Note: I am using the latest plugins (as of 22 mar 2016).

            Show
            dragos_n_sava Nicolae Dragos Sava added a comment - Hi guys, so I manage to reproduce a posible cause for this bug (there could be other cases also): I am certain the build triggers are one of the problems: "pull scm" and "build periodically" collide. So the pull scm will work ok but when build periodically comes, the build is going to fail with the lock issue. Replicate: have pull scm at 5 minute and build periodically at 5 minute and look what happens. Note: I am using the latest plugins (as of 22 mar 2016).
            Hide
            f0zz matthew giardina added a comment -

            Is there any progress on this?

            Show
            f0zz matthew giardina added a comment - Is there any progress on this?
            Hide
            markewaite Mark Waite added a comment - - edited

            No, and not likely to be any progress in the near term.  

            You can install the "pre-scm build" plugin and and a pre-scm step to unconditionally remove lock files in the .git directory if you're seeing this frequently enough to justify changing your job definition.

            Show
            markewaite Mark Waite added a comment - - edited No, and not likely to be any progress in the near term.   You can install the "pre-scm build" plugin and and a pre-scm step to unconditionally remove lock files in the .git directory if you're seeing this frequently enough to justify changing your job definition.
            Hide
            trejkaz trejkaz added a comment -

            Copying from JENKINS-47652, commands leading up to the error were:

            [macosx]  > git rev-parse --is-inside-work-tree # timeout=10
            [macosx]  > git config remote.origin.url git@_git-server_:_app_/core.git # timeout=10
            [macosx]  > git --version # timeout=10
            [macosx]  > git fetch --tags --progress git@_git-server_:_app_/core.git +refs/pull/50/head:refs/remotes/origin/PR-50 +refs/heads/master:refs/remotes/origin/master --prune # timeout=40
            [macosx]  > git config core.sparsecheckout # timeout=10
            [macosx]  > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30
            [macosx]  > git merge 0abbc94cdff38bfe76dad45562011f6f703143ce # timeout=10
            [macosx]  > git config core.sparsecheckout # timeout=10
            [macosx]  > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30
            

            The sequence of commands seems odd to me, but surely it wasn't locked before SCM commands started, or the first `git checkout` would have failed, so I'm not sure a pre-SCM step would work either, unless there is a way to do it half way through the SCM step? Or are we saying that this log represents multiple SCM steps, so the pre would run before each? (Jenkins' logs make what's being done very unclear.)

            Show
            trejkaz trejkaz added a comment - Copying from JENKINS-47652 , commands leading up to the error were: [macosx] > git rev-parse --is-inside-work-tree # timeout=10 [macosx] > git config remote.origin.url git@_git-server_:_app_/core.git # timeout=10 [macosx] > git --version # timeout=10 [macosx] > git fetch --tags --progress git@_git-server_:_app_/core.git +refs/pull/50/head:refs/remotes/origin/PR-50 +refs/heads/master:refs/remotes/origin/master --prune # timeout=40 [macosx] > git config core.sparsecheckout # timeout=10 [macosx] > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30 [macosx] > git merge 0abbc94cdff38bfe76dad45562011f6f703143ce # timeout=10 [macosx] > git config core.sparsecheckout # timeout=10 [macosx] > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30 The sequence of commands seems odd to me, but surely it wasn't locked before SCM commands started, or the first `git checkout` would have failed, so I'm not sure a pre-SCM step would work either, unless there is a way to do it half way through the SCM step? Or are we saying that this log represents multiple SCM steps, so the pre would run before each? (Jenkins' logs make what's being done very unclear.)

              People

              • Assignee:
                ndeloof Nicolas De Loof
                Reporter:
                pbaranchikov Pavel Baranchikov
              • Votes:
                8 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: