Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14046

git operations occasionally hang in Jenkins on Windows

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: git-plugin
    • Environment:
    • Similar Issues:

      Description

      See elaborate description, dumps and discussion at:
      http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows

      Basically, there is a Jenkins environment with some jobs based on GIT with periodic polling.
      Each of the main GIT operations in the system (periodic polling, the pull at the beginning of a job, and the tag push at the end of a job) all sometimes hang, causing builds not to happen.

      I suspect that the GIT plugin is holding handles to some GIT internal files, and this locks out the GIT command-line program and causes it to hang at the beginning of a run.
      This suspicion is based on a different issue I encountered - I could not wipe the workspace for one of the jobs because a GIT pack file was locked. It turns out that the process locking the file was Tomcat itself and not the GIT process as I expected.

        Attachments

          Issue Links

            Activity

            Hide
            tmack0 Theral Mackey added a comment -

            We hit the same problem, project builds will randomly fail and once one project fails it will not build again until the processes holding the git pack files locked are killed off so the files can be removed. Other projects continue building fine. In our case, this involved one git.exe process and an ssh.exe process (win7-64 w/cygwin). Once removed the Jenkins process was restarted and the build that was failing built successfully.

            Show
            tmack0 Theral Mackey added a comment - We hit the same problem, project builds will randomly fail and once one project fails it will not build again until the processes holding the git pack files locked are killed off so the files can be removed. Other projects continue building fine. In our case, this involved one git.exe process and an ssh.exe process (win7-64 w/cygwin). Once removed the Jenkins process was restarted and the build that was failing built successfully.
            Hide
            vladdrussian VladDRussian added a comment -

            We've also hit the issue while on Jenkins 1.467 looking at changing to GIT plugin 1.1.19 - 1.1.22

            I think the below might be a duplicate of this one.
            https://issues.jenkins-ci.org/browse/JENKINS-14151

            Show
            vladdrussian VladDRussian added a comment - We've also hit the issue while on Jenkins 1.467 looking at changing to GIT plugin 1.1.19 - 1.1.22 I think the below might be a duplicate of this one. https://issues.jenkins-ci.org/browse/JENKINS-14151
            Hide
            gcollic Guillaume Collic added a comment -

            We've also hit the issue while on Jenkins "1.485", GIT plugin "1.1.24", and msysgit "1.7.11.msysgit.1", using git.cmd

            Show
            gcollic Guillaume Collic added a comment - We've also hit the issue while on Jenkins "1.485", GIT plugin "1.1.24", and msysgit "1.7.11.msysgit.1", using git.cmd
            Hide
            georgik Juraj Michalek added a comment - - edited

            Solution suggested by VonC works for me (http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows):

            • Setting the Path to Git executable to C:\Program Files (x86)\Git\cmd\git.exe (i.e. cmd and not bin!)
            Show
            georgik Juraj Michalek added a comment - - edited Solution suggested by VonC works for me ( http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows): Setting the Path to Git executable to C:\Program Files (x86)\Git\cmd\git.exe (i.e. cmd and not bin!)
            Hide
            ikedam ikedam added a comment -

            It also happens in my environment:
            OS: Windows7/64bit
            Git: msysgit 1.8.0-preview20121022(32bit)
            Path to git: cmd/git.exe
            Node: slave (master is linux)
            Scheme for git: ssh
            Authentication: key pairs without a passphrase(set HOME environment variable)
            When it happens: polling, SCM checkout in builds
            Note: It seems happen when multiple git.exe (and ssh.exe) runs.

            It may be caused not by git itself, but by ssh.
            I cannot see what is happening in backgrounds, for I cannot get outputs from git or ssh... it may be a kind of race conditions, for it happens when multiple git processes run.

            I'll try workarounds I've ever heard.

            • use plink.exe by setting GIT_SSH
            • use bin/git.exe instead of cmd/git.exe

            I also think these are worth to try:

            • Wrap git with a script for excluding execution.
            • Add a timeout feature to git plugin.
            Show
            ikedam ikedam added a comment - It also happens in my environment: OS: Windows7/64bit Git: msysgit 1.8.0-preview20121022(32bit) Path to git: cmd/git.exe Node: slave (master is linux) Scheme for git: ssh Authentication: key pairs without a passphrase(set HOME environment variable) When it happens: polling, SCM checkout in builds Note: It seems happen when multiple git.exe (and ssh.exe) runs. It may be caused not by git itself, but by ssh. I cannot see what is happening in backgrounds, for I cannot get outputs from git or ssh... it may be a kind of race conditions, for it happens when multiple git processes run. I'll try workarounds I've ever heard. use plink.exe by setting GIT_SSH use bin/git.exe instead of cmd/git.exe I also think these are worth to try: Wrap git with a script for excluding execution. Add a timeout feature to git plugin.
            Hide
            ikedam ikedam added a comment - - edited

            In my environment, hangs is not reproduced since introducing following configuration:

            • Create a batch file
              @echo off
              ssh -i C:\path\to\id_rsa -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=3 -oServerAliveInterval=1 -oServerAliveCountMax=3 %*
              
            • Set GIT_SSH to the path to the file above.

            My environemnt is:
            OS: Windows7/64bit
            Git: msysgit 1.8.0-preview20121022(32bit)
            Git plugin: 1.5.0

            I want to know whether this also works for others environments. Please try it.

            Show
            ikedam ikedam added a comment - - edited In my environment, hangs is not reproduced since introducing following configuration: Create a batch file @echo off ssh -i C:\path\to\id_rsa -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=3 -oServerAliveInterval=1 -oServerAliveCountMax=3 %* Set GIT_SSH to the path to the file above. My environemnt is: OS: Windows7/64bit Git: msysgit 1.8.0-preview20121022(32bit) Git plugin: 1.5.0 I want to know whether this also works for others environments. Please try it.
            Hide
            markewaite Mark Waite added a comment -

            Based on the significant changes in git-client-plugin and git-plugin to support credentials and to support JGit, and the lack of an activity on this bug for multiple months, I'm closing the bug.

            Show
            markewaite Mark Waite added a comment - Based on the significant changes in git-client-plugin and git-plugin to support credentials and to support JGit, and the lack of an activity on this bug for multiple months, I'm closing the bug.

              People

              • Assignee:
                ndeloof Nicolas De Loof
                Reporter:
                orantmp Oran t
              • Votes:
                5 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: