Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45581

All authenticated Git operations from Master node fail

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      When attempting to perform an authenticated Git operation from our Master node through Jenkins, it will always fail with the following error:

      hudson.plugins.git.GitException: Command "C:\Program Files\Git\bin\git.exe ls-remote -h <address-to-repo>.git" returned status code 128:
      stdout:
      stderr: 0 [main] ssh 8928 C:\Program Files\Git\usr\bin\ssh.exe: *** fatal error - add_item ("??\C:\Program Files\Git", "/", ...) failed, errno 1
      Stack trace:
      Frame Function Args
      000FFFF9B90 0018005CE9E (0018023D522, 0018021AC39, 00600010000, 000FFFF8B10)
      000FFFF9B90 00180046559 (000FFFFABC0, 000FFFF9B90, 1D2FF07C4E9D602, 00000000000)
      000FFFF9B90 00180046592 (000FFFF9B90, 00000000001, 00600010000, 505C3A435C3F3F5C)
      000FFFF9B90 001800CADBB (00000000000, 00140000024, 00000000000, 1D2FF07CDDAC80D)
      000FFFFCBE0 00180118755 (00000000000, 00000000000, 00000000000, 00000000000)
      000FFFFCCB0 00180046B65 (00000000000, 00000000000, 00000000000, 00000000000)
      00000000000 001800457C3 (00000000000, 00000000000, 00000000000, 00000000000)
      000FFFFFFF0 00180045874 (00000000000, 00000000000, 00000000000, 00000000000)
      End of stack trace
      fatal: Could not read from remote repository. Please make sure you have the correct access rights
      and the repository exists.

      This applies to both SSH and HTTP authentication. Performing the exact same operations directly on the server and/or using JGit works as expected. We had originally been using JGit for operations performed on the master node (Polling, SCM checkout of Pipelines, etc) as a workaround, but applying Git Polling from within a Pipeline script (Jenkinsfile) appears to override the JGit specification and use the CLI Git instead, resulting in polling always failing due to the error above.

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Unfortunately, I don't have any further ideas to offer. There are many git installations on Windows running the Jenkins git plugin. I suspect it is something in your environment, though I don't have any great guesses. Some things that I've used in the past have included:

          • Run Jenkins as a desktop app instead of running as a service (remove the complications of the local system account)
          • Remove as much as a possible from the environment (no GIT_* variables, no SSH_* variables, etc.)
          • Keep the Jenkins home directory in a path that includes no space characters
          • Keep the temporary directory in a path that includes no space characters
          Show
          markewaite Mark Waite added a comment - Unfortunately, I don't have any further ideas to offer. There are many git installations on Windows running the Jenkins git plugin. I suspect it is something in your environment, though I don't have any great guesses. Some things that I've used in the past have included: Run Jenkins as a desktop app instead of running as a service (remove the complications of the local system account) Remove as much as a possible from the environment (no GIT_* variables, no SSH_* variables, etc.) Keep the Jenkins home directory in a path that includes no space characters Keep the temporary directory in a path that includes no space characters
          Hide
          b_c Benjamin Coover added a comment - - edited

          Correct, Jenkins is installed and running as a service.

          Credentials manager is disabled (confirmed with git config --global --list, lack of "credential.helper=manager")

          Originally was not setting GIT_SSH, started doing so in hopes of fixing the issues (GIT_SSH=C:\Program Files\Git\usr\bin\ssh.exe; More on this below). No quotes includes.

          I was under the impression that HOME was set for the system account, but I discovered it was not. I have now gotten the Local System Account setup with a proper HOME variable (set to the jenkins user's home directory: C:\Users\jenkins). After this, I'm now able to replicate the error directly on the server in a cmd window running as the Local System Account.

          Based on some other discoveries I've made from this point, I've done the following:

          • uninstalled Git and reinstalled to "C:\Git" (was previously "C:\Program Files\Git")
          • Made sure to uncheck "Enable Credential Manager" during reinstallation. Simply unsetting the global config did not appear to be enough.
          • Changed GIT_SSH to point to C:\Git\usr\bin\ssh.exe

          By doing the above, I can now successfully clone a repository directly from the server (as the LSA), as well as via calling git during a job, such as in a Pipeline. However, Polling and Pipeline SCM steps using Git now hang at the "using GIT_SSH to set credentials Git SSH" step and eventually fail after the 10 minute timeout instead of giving the ssh error.

          EDIT: And just like that, I'm back to the error again. I restarted the Jenkins service with the intention of persisting the configurations I'd done on the server and then ran the job I was using to test all this. Error once again shows up in-job and direct on the server with no other changes in between.

          Show
          b_c Benjamin Coover added a comment - - edited Correct, Jenkins is installed and running as a service. Credentials manager is disabled (confirmed with git config --global --list, lack of "credential.helper=manager") Originally was not setting GIT_SSH, started doing so in hopes of fixing the issues (GIT_SSH=C:\Program Files\Git\usr\bin\ssh.exe; More on this below). No quotes includes. I was under the impression that HOME was set for the system account, but I discovered it was not. I have now gotten the Local System Account setup with a proper HOME variable (set to the jenkins user's home directory: C:\Users\jenkins). After this, I'm now able to replicate the error directly on the server in a cmd window running as the Local System Account. Based on some other discoveries I've made from this point, I've done the following: uninstalled Git and reinstalled to "C:\Git" (was previously "C:\Program Files\Git") Made sure to uncheck "Enable Credential Manager" during reinstallation. Simply unsetting the global config did not appear to be enough. Changed GIT_SSH to point to C:\Git\usr\bin\ssh.exe By doing the above, I can now successfully clone a repository directly from the server (as the LSA), as well as via calling git during a job, such as in a Pipeline. However, Polling and Pipeline SCM steps using Git now hang at the "using GIT_SSH to set credentials Git SSH" step and eventually fail after the 10 minute timeout instead of giving the ssh error. EDIT: And just like that, I'm back to the error again. I restarted the Jenkins service with the intention of persisting the configurations I'd done on the server and then ran the job I was using to test all this. Error once again shows up in-job and direct on the server with no other changes in between.
          Hide
          markewaite Mark Waite added a comment - - edited

          Based on your comments on bountysource, it appears that you're running Jenkins on your Windows machine as a service. Can you confirm that is still correct?

          Have you confirmed that the credentials manager is disabled in that git for windows installation?

          Have you reviewed the entire set of environment variables on that machine, and especially the environment variables defined for that service account? Watch for any environment variables that might affect the behavior of ssh.exe.

          Are you setting a value for GIT_SSH? If so, does it behave more reasonably without that value?

          If you're setting a value for GIT_SSH, are you mistakenly including double quotes in the value of GIT_SSH?

          Do you have a value for HOME in the system account where the process is running? If so, what is that value?

          Show
          markewaite Mark Waite added a comment - - edited Based on your comments on bountysource , it appears that you're running Jenkins on your Windows machine as a service. Can you confirm that is still correct? Have you confirmed that the credentials manager is disabled in that git for windows installation? Have you reviewed the entire set of environment variables on that machine, and especially the environment variables defined for that service account? Watch for any environment variables that might affect the behavior of ssh.exe. Are you setting a value for GIT_SSH? If so, does it behave more reasonably without that value? If you're setting a value for GIT_SSH, are you mistakenly including double quotes in the value of GIT_SSH? Do you have a value for HOME in the system account where the process is running? If so, what is that value?
          Hide
          b_c Benjamin Coover added a comment -

          I get the same result whether I define the Global Tool as "git" or if I give the full path "C:\Program Files\Git\bin\git.exe". The other executables (git-bash.exe, ssh.exe, etc) are not defined or even directly used anywhere except for the location of ssh.exe as GIT_SSH. I did this based on a StackOverflow answer to hopefully get somewhere based on the direct error that Git was reporting through the plugin.

          I have also done a fresh uninstall-reinstall of Git for Windows (multiple times) with no change.

          Show
          b_c Benjamin Coover added a comment - I get the same result whether I define the Global Tool as "git" or if I give the full path "C:\Program Files\Git\bin\git.exe". The other executables (git-bash.exe, ssh.exe, etc) are not defined or even directly used anywhere  except for the location of ssh.exe as GIT_SSH. I did this based on a StackOverflow answer  to hopefully get somewhere based on the direct error that Git was reporting through the plugin. I have also done a fresh uninstall-reinstall of Git for Windows (multiple times) with no change.
          Hide
          markewaite Mark Waite added a comment - - edited

          Since your master node is a Windows machine, I would guess that you've somehow incorrectly declared the git executable for that node. That configuration error could be either in the "Manage Jenkins" -> "Global Tool Configuration" -> "Git" section, or it might be in the agent configuration for the master agent.

          It is better to use "git" as the git executable, then adjust the system-wide PATH variable to include the path. The git for Windows installer will do that if its defaults are chosen.

          Don't use "git-bash.exe" or "git-cmd.exe" as the git executable. Those are helper programs included with the git for Windows distribution and they break the plugin due to their subtle differences compared to git.exe and git.cmd.

          It is much better to review these types of configuration questions with the jenkins users mailing list before submitting a bug report. There are many more readers of the jenkins users mailing list than there are readers of bug reports to the git plugin.

          Show
          markewaite Mark Waite added a comment - - edited Since your master node is a Windows machine, I would guess that you've somehow incorrectly declared the git executable for that node. That configuration error could be either in the "Manage Jenkins" -> "Global Tool Configuration" -> "Git" section, or it might be in the agent configuration for the master agent. It is better to use "git" as the git executable, then adjust the system-wide PATH variable to include the path. The git for Windows installer will do that if its defaults are chosen. Don't use "git-bash.exe" or "git-cmd.exe" as the git executable. Those are helper programs included with the git for Windows distribution and they break the plugin due to their subtle differences compared to git.exe and git.cmd. It is much better to review these types of configuration questions with the jenkins users mailing list before submitting a bug report. There are many more readers of the jenkins users mailing list than there are readers of bug reports to the git plugin.

            People

            • Assignee:
              Unassigned
              Reporter:
              b_c Benjamin Coover
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: