-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.277.1 with git plugin 4.7.0 and other plugins as described in my docker-lfs repo dated 31 Mar 2021
The git plugin left a config.lock file in one of the caches folders The folder was two weeks old and was showing indications that it might have benefited from a git gc.
Because the config.lock file was left in that folder, many jobs failed (hundreds). They could not acquire a configuration lock on that repository. The plugin needs to either be less concerned about acquiring locks when reading the repository configuration or it needs to be more aggressive in cleaning locks and possibly detecting repositories that need to be garbage collected.
Might consider an Admin Monitor to report cached repositories that would benefit from garbage collection along with an operation that would perform garbage collection on repositories that were detected as needing cleanup.
The directory contents looked like this when the lock was held for 7 or more hours:
-rw-r--r-- 1 jenkins jenkins 0 Mar 22 11:24 FETCH_HEAD -rw-r--r-- 1 jenkins jenkins 164 Mar 22 11:24 config -rw-r--r-- 1 jenkins jenkins 0 Mar 22 11:24 config.lock drwxr-xr-x 8 jenkins jenkins 4096 Mar 31 19:11 . jenkins@bfe5372c3e46:~/caches/git-c3acd33583448548d14667df83b93ee1$ date Wed 31 Mar 2021 07:12:05 PM MDT
The build log using command line git reported:
15:04:49 > git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git # timeout=11 15:04:49 hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git" returned status code 255: 15:04:49 stdout: 15:04:49 stderr: error: could not lock config file .git/config: File exists 15:04:49 15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2608) 15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2538) 15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2534) 15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1920) 15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1932) 15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1546) 15:04:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161) 15:04:49 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357) 15:04:49 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330) 15:04:49 at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396) 15:04:49 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582) 15:04:49 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100) 15:04:49 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309) 15:04:49 at hudson.model.ResourceController.execute(ResourceController.java:97) 15:04:49 at hudson.model.Executor.run(Executor.java:429) 15:04:49 Finished: FAILURE
The build log using JGit reported:
5:03:01 Started by remote host Started from Groovy Console by Mark Waite with note: Groovy console started 15:03:01 Setting origin to https://github.com/MarkEWaite/jenkins-bugs.git 15:03:06 org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config. Ensure that no other process has an open file handle on the lock file /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config.lock, then you may delete the lock file and retry. 15:03:06 at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:220) 15:03:06 at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:945) 15:03:06 Caused: hudson.plugins.git.GitException 15:03:06 at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:947) 15:03:06 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357) 15:03:06 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330) 15:03:06 at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396) 15:03:06 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582) 15:03:06 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100) 15:03:06 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309) 15:03:06 at hudson.model.ResourceController.execute(ResourceController.java:97) 15:03:06 at hudson.model.Executor.run(Executor.java:429) 15:03:06 Finished: FAILURE