Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36833

Race Condition Populating Multiple Tag Parameters

    Details

    • Similar Issues:

      Description

      I'm observing a minor but somewhat annoying issue with the plugin: if a project has multiple tag parameters, and if at build time the tags need to be fetched, then multiple fetch operations are performed concurrently that may lead to failure due to git file locking. Here's some truncated log:

      Command "git -c core.askpass=true fetch --tags --progress <repo> +refs/heads/master:refs/remotes/origin/master --prune" returned status code 1:
      stdout: 
      stderr: remote: Counting objects: 1
      remote: Counting objects: 51, done
      remote: Finding sources:   3% (1/27)
      remote: Finding sources:   7% (2/27)
      ...
      remote: Finding sources:  96% (26/27)
      remote: Finding sources: 100% (27/27)
      remote: Total 27 (delta 12), reused 27 (delta 12)
      error: cannot lock ref 'refs/remotes/origin/master': Unable to create '/opt/jenkins_slave/workspace/deploy-production/.git/refs/remotes/origin/master.lock': File exists.
      
      Another git process seems to be running in this repository, e.g.
      an editor opened by 'git commit'. Please make sure all processes
      are terminated then try again. If it still fails, a git process
      may have crashed in this repository earlier:
      remove the file manually to continue.
      From <repo>
       ! 97af2f9..8a2a820  master     -> origin/master  (unable to update local ref)
       * [new tag]         <tag> -> <tag>
       Pleas look to log!
      

      It's possible to work around, as I can reload the Build page and tags will populate after the fetch has been done at least once. Our repo has many tags and branches though and fetching can take maybe 30+ seconds, after which having to reload the page can be a minor burden. Eventually the intention here is that engineers outside of our devops / infrastructure team should be able to trigger builds and deploys via Jenkins - these are potentially junior engineers / interns, so I'm trying to make the user experience as easy as possible.

        Attachments

          Activity

          Hide
          klimas7 Boguslaw Klimas added a comment -

          Hi,
          I reproduced yours and a similar problems: (sory for polish message )
          I created pull request with these changes https://github.com/jenkinsci/git-parameter-plugin/pull/28
          Could you check these changes? If is it fine and it works for you I merge PR and I will release new versions

          lip 21, 2016 12:21:42 AM net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition generateContents
          SEVERE: Unexpected error!
          hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/klimas7/Learn.git +refs/heads/*:refs/remotes/origin/* --prune" returned status code 1:
          stdout: 
          stderr: remote: Counting objects: 1, done.        
          remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0        
          error: cannot lock ref 'refs/remotes/origin/master': ref refs/remotes/origin/master is at 9294d3fcc582801a506df9459d563a065ee0a915 but expected b622558d89d356244d72f60ca2bb669f41cc2848
          From https://github.com/klimas7/Learn
           ! b622558..9294d3f  master     -> origin/master  (unable to update local ref)
          
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1459)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314)
                  at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.generateContents(GitParameterDefinition.java:250)
          
          lip 20, 2016 11:58:58 PM net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition generateContents
          SEVERE: Unexpected error!
          hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/klimas7/Learn.git +refs/heads/*:refs/remotes/origin/*" returned status code 1:
          stdout: 
          stderr: remote: Counting objects: 225, done.        
          remote: Total 225 (delta 0), reused 0 (delta 0), pack-reused 225        
          Receiving objects: 100% (225/225), 27.13 KiB | 0 bytes/s, done.
          Resolving deltas: 100% (36/36), done.
          error: cannot lock ref 'refs/remotes/origin/feature/test_1.0': Unable to create '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/refs/remotes/origin/feature/test_1.0.lock': Plik istnieje.
          
          If no other git process is currently running, this probably means a
          git process crashed in this repository earlier. Make sure no other git
          process is running and remove the file manually to continue.
          From https://github.com/klimas7/Learn
           ! [new branch]      feature/test_1.0 -> origin/feature/test_1.0  (unable to update local ref)
           * [new branch]      master     -> origin/master
           * [new tag]         Tag_1.0    -> Tag_1.0
          
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1459)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:506)
                  at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.clone(AbstractGitAPIImpl.java:67)
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clone(CliGitAPIImpl.java:63)
                  at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.initWorkspace(GitParameterDefinition.java:368)
          
          lip 20, 2016 11:58:58 PM net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition generateContents
          SEVERE: Unexpected error!
          hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/klimas7/Learn.git +refs/heads/*:refs/remotes/origin/*" returned status code 1:
          stdout: 
          stderr: remote: Counting objects: 225, done.        
          remote: Total 225 (delta 0), reused 0 (delta 0), pack-reused 225        
          Receiving objects: 100% (225/225), 27.13 KiB | 0 bytes/s, done.
          Resolving deltas: 100% (36/36), done.
          error: cannot lock ref 'refs/remotes/origin/feature/test_1.0': Unable to create '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/refs/remotes/origin/feature/test_1.0.lock': Plik istnieje.
          
          If no other git process is currently running, this probably means a
          git process crashed in this repository earlier. Make sure no other git
          process is running and remove the file manually to continue.
          From https://github.com/klimas7/Learn
           ! [new branch]      feature/test_1.0 -> origin/feature/test_1.0  (unable to update local ref)
          error: cannot lock ref 'refs/remotes/origin/master': Unable to create '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/refs/remotes/origin/master.lock': Plik istnieje.
          
          If no other git process is currently running, this probably means a
          git process crashed in this repository earlier. Make sure no other git
          process is running and remove the file manually to continue.
          
          Caused by: hudson.plugins.git.GitException: Command "git init /var/lib/jenkins/jobs/GitParameter_gitHub/workspace" returned status code 128:
          stdout: 
          stderr: fatal: cannot copy '/usr/share/git-core/templates/info/exclude' to '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/info/exclude': Plik istnieje
          
                  at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
          
          Show
          klimas7 Boguslaw Klimas added a comment - Hi, I reproduced yours and a similar problems: (sory for polish message ) I created pull request with these changes https://github.com/jenkinsci/git-parameter-plugin/pull/28 Could you check these changes? If is it fine and it works for you I merge PR and I will release new versions lip 21, 2016 12:21:42 AM net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition generateContents SEVERE: Unexpected error! hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/klimas7/Learn.git +refs/heads/*:refs/remotes/origin/* --prune" returned status code 1: stdout: stderr: remote: Counting objects: 1, done. remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 error: cannot lock ref 'refs/remotes/origin/master': ref refs/remotes/origin/master is at 9294d3fcc582801a506df9459d563a065ee0a915 but expected b622558d89d356244d72f60ca2bb669f41cc2848 From https://github.com/klimas7/Learn ! b622558..9294d3f master -> origin/master (unable to update local ref) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1459) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314) at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.generateContents(GitParameterDefinition.java:250) lip 20, 2016 11:58:58 PM net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition generateContents SEVERE: Unexpected error! hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/klimas7/Learn.git +refs/heads/*:refs/remotes/origin/*" returned status code 1: stdout: stderr: remote: Counting objects: 225, done. remote: Total 225 (delta 0), reused 0 (delta 0), pack-reused 225 Receiving objects: 100% (225/225), 27.13 KiB | 0 bytes/s, done. Resolving deltas: 100% (36/36), done. error: cannot lock ref 'refs/remotes/origin/feature/test_1.0': Unable to create '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/refs/remotes/origin/feature/test_1.0.lock': Plik istnieje. If no other git process is currently running, this probably means a git process crashed in this repository earlier. Make sure no other git process is running and remove the file manually to continue. From https://github.com/klimas7/Learn ! [new branch] feature/test_1.0 -> origin/feature/test_1.0 (unable to update local ref) * [new branch] master -> origin/master * [new tag] Tag_1.0 -> Tag_1.0 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1459) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:506) at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.clone(AbstractGitAPIImpl.java:67) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clone(CliGitAPIImpl.java:63) at net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.initWorkspace(GitParameterDefinition.java:368) lip 20, 2016 11:58:58 PM net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition generateContents SEVERE: Unexpected error! hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/klimas7/Learn.git +refs/heads/*:refs/remotes/origin/*" returned status code 1: stdout: stderr: remote: Counting objects: 225, done. remote: Total 225 (delta 0), reused 0 (delta 0), pack-reused 225 Receiving objects: 100% (225/225), 27.13 KiB | 0 bytes/s, done. Resolving deltas: 100% (36/36), done. error: cannot lock ref 'refs/remotes/origin/feature/test_1.0': Unable to create '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/refs/remotes/origin/feature/test_1.0.lock': Plik istnieje. If no other git process is currently running, this probably means a git process crashed in this repository earlier. Make sure no other git process is running and remove the file manually to continue. From https://github.com/klimas7/Learn ! [new branch] feature/test_1.0 -> origin/feature/test_1.0 (unable to update local ref) error: cannot lock ref 'refs/remotes/origin/master': Unable to create '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/refs/remotes/origin/master.lock': Plik istnieje. If no other git process is currently running, this probably means a git process crashed in this repository earlier. Make sure no other git process is running and remove the file manually to continue. Caused by: hudson.plugins.git.GitException: Command "git init /var/lib/jenkins/jobs/GitParameter_gitHub/workspace" returned status code 128: stdout: stderr: fatal: cannot copy '/usr/share/git-core/templates/info/exclude' to '/var/lib/jenkins/jobs/GitParameter_gitHub/workspace/.git/info/exclude': Plik istnieje at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
          Hide
          ltdowney Luke Downey added a comment -

          Hey,

          Thanks for the rapid response. The fix looks totally reasonable. I've been out of the office yesterday and today, but I will figure out how to package and install this on our Jenkins deployment on Monday. Once that's done it should be trivial to verify.

          Show
          ltdowney Luke Downey added a comment - Hey, Thanks for the rapid response. The fix looks totally reasonable. I've been out of the office yesterday and today, but I will figure out how to package and install this on our Jenkins deployment on Monday. Once that's done it should be trivial to verify.
          Hide
          ltdowney Luke Downey added a comment -

          Saw that you merged your PR over the weekend. I've just finished verifying the fix on our Jenkins deployment, and can confirm the issue is resolved. Thanks very much for the help!

          Show
          ltdowney Luke Downey added a comment - Saw that you merged your PR over the weekend. I've just finished verifying the fix on our Jenkins deployment, and can confirm the issue is resolved. Thanks very much for the help!
          Hide
          klimas7 Boguslaw Klimas added a comment -

          Thanks!

          Show
          klimas7 Boguslaw Klimas added a comment - Thanks!
          Hide
          klimas7 Boguslaw Klimas added a comment -

          release 0.6.2

          Show
          klimas7 Boguslaw Klimas added a comment - release 0.6.2

            People

            • Assignee:
              klimas7 Boguslaw Klimas
              Reporter:
              ltdowney Luke Downey
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: