Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61408

Git executable is randomly set to empty string

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.204.5
      Git Plugin 4.1.1
    • Similar Issues:

      Description

      Once in a while, at least several times a week, the name of the Git executable ends up being unset and the Git plugin calls `  init` (whitespace) instead of `git init`.

      We use a custom installer for git and it works and install just fine and even with jobs with multiple parallel stages, only one stage hits the bug, randomly. So configuration on our side is just fine but we suspect that there is a bug in the codebase where the variable holding the git executable name can get invalidated, at the stage level.

      [2020-03-04T19:28:38.877Z] Failed to get git executable
      [2020-03-04T19:28:42.682Z] using credential xxxxx-github-rw
      [2020-03-04T19:28:42.691Z] Cloning the remote Git repository
      [2020-03-04T19:28:42.691Z] Using shallow clone with depth 500
      [2020-03-04T19:28:42.691Z] Avoid fetching tags
      [2020-03-04T19:28:42.691Z] Honoring refspec on initial clone
      [2020-03-04T19:28:42.933Z] ERROR: Error cloning remote repo 'origin'
      [2020-03-04T19:28:42.933Z] hudson.plugins.git.GitException: Could not init /home/jenkins/agent/workspace/GitHub_xxxx_master
      [2020-03-04T19:28:42.933Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:989)
      [2020-03-04T19:28:42.933Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:747)
      [2020-03-04T19:28:42.933Z] 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
      [2020-03-04T19:28:42.933Z] 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
      [2020-03-04T19:28:42.933Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      [2020-03-04T19:28:42.934Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      [2020-03-04T19:28:42.934Z] 	at hudson.remoting.Request$2.run(Request.java:369)
      [2020-03-04T19:28:42.934Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      [2020-03-04T19:28:42.934Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      [2020-03-04T19:28:42.934Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      [2020-03-04T19:28:42.934Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      [2020-03-04T19:28:42.934Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
      [2020-03-04T19:28:42.934Z] 	at java.lang.Thread.run(Thread.java:748)
      [2020-03-04T19:28:42.934Z] 	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-xx-xx-xx-xx.us-west-2.compute.internal/xx.xx.xx.xx:40980
      [2020-03-04T19:28:42.934Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
      [2020-03-04T19:28:42.934Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
      [2020-03-04T19:28:42.934Z] 		at hudson.remoting.Channel.call(Channel.java:955)
      [2020-03-04T19:28:42.934Z] 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
      [2020-03-04T19:28:42.934Z] 		at sun.reflect.GeneratedMethodAccessor811.invoke(Unknown Source)
      [2020-03-04T19:28:42.934Z] 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      [2020-03-04T19:28:42.934Z] 		at java.lang.reflect.Method.invoke(Method.java:498)
      [2020-03-04T19:28:42.934Z] 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
      [2020-03-04T19:28:42.934Z] 		at com.sun.proxy.$Proxy163.execute(Unknown Source)
      [2020-03-04T19:28:42.934Z] 		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1114)
      [2020-03-04T19:28:42.934Z] 		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)
      [2020-03-04T19:28:42.934Z] 		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
      [2020-03-04T19:28:42.934Z] 		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      [2020-03-04T19:28:42.934Z] 		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      [2020-03-04T19:28:42.934Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      [2020-03-04T19:28:42.934Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      [2020-03-04T19:28:42.934Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      [2020-03-04T19:28:42.934Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      [2020-03-04T19:28:42.934Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      [2020-03-04T19:28:42.934Z] 		... 1 more
      [2020-03-04T19:28:42.934Z] Caused by: hudson.plugins.git.GitException: Error performing git command:  init /home/jenkins/agent/workspace/GitHub_xxxx_master
      [2020-03-04T19:28:42.934Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2436)
      [2020-03-04T19:28:42.934Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2359)
      [2020-03-04T19:28:42.934Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2355)
      [2020-03-04T19:28:42.934Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1915)
      [2020-03-04T19:28:42.934Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:987)
      [2020-03-04T19:28:42.934Z] 	... 12 more
      [2020-03-04T19:28:42.934Z] Caused by: java.io.IOException: Cannot run program "" (in directory "/home/jenkins/agent/workspace/GitHub_xxxx_master"): error=2, No such file or directory
      [2020-03-04T19:28:42.934Z] 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
      [2020-03-04T19:28:42.934Z] 	at hudson.Proc$LocalProc.<init>(Proc.java:250)
      [2020-03-04T19:28:42.934Z] 	at hudson.Proc$LocalProc.<init>(Proc.java:219)
      [2020-03-04T19:28:42.934Z] 	at hudson.Launcher$LocalLauncher.launch(Launcher.java:937)
      [2020-03-04T19:28:42.934Z] 	at hudson.Launcher$ProcStarter.start(Launcher.java:455)
      [2020-03-04T19:28:42.934Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2422)
      [2020-03-04T19:28:42.934Z] 	... 16 more
      [2020-03-04T19:28:42.934Z] Caused by: java.io.IOException: error=2, No such file or directory
      [2020-03-04T19:28:42.934Z] 	at java.lang.UNIXProcess.forkAndExec(Native Method)
      [2020-03-04T19:28:42.934Z] 	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
      [2020-03-04T19:28:42.934Z] 	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
      [2020-03-04T19:28:42.934Z] 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
      [2020-03-04T19:28:42.934Z] 	... 21 more
      Error cloning remote repo 'origin'
      

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Steps to duplicate the problem?

          Show
          markewaite Mark Waite added a comment - Steps to duplicate the problem?
          Hide
          sodul Stephane Odul added a comment -

          Under Global Tool Configuration replace the Default Git installation with a custom one that uses a shell script.

          In our case we have 2 installers with filters:

          • master: the script is just `true` and Tool Home is `/usr/bin/git`
          • !master: the script is installing a simple shell wrapper to perform retry logic when the git commands fails due to transient network or server side glitches. The Tool Home is the path to the wrapper.

          We use the GitHub Organization Plugin to scan our repos and trigger our CI pipelines, and we use the Kubernetes plugin to launch ephemeral workers for each stage.

          The kubernetes plugin is configured with a default 'jnlp' pod definition which is nothing more that the default pod definition that comes with the kubernetes plugin.

          The Jenkinsfile has parallel stages, it does not matter what the stages do.

          Very randomly, and infrequently, with no apparent cause one of the stages will get an empty git executable name with the stack trace provided in the description.

           

          Sample Jenkinsfile:

           
          pipeline {
              agent none
              options {
                  ansiColor('xterm')
                  timestamps ()
                  buildDiscarder(logRotator(daysToKeepStr: '30'))
                  timeout(time: 5, unit: 'MINUTES')
              }
              stages {
                  stage('Tests') {
                      parallel {
                          stage('sleep 1') {
                              agent { kubernetes { label 'jnlp' } }
                              steps { sh 'sleep 1' }
                          }
                          stage('sleep 2') {
                              agent { kubernetes { label 'jnlp' } }
                              steps { sh 'sleep 2' }
                          }
                          stage('sleep 3') {
                              agent { kubernetes { label 'jnlp' } }
                              steps { sh 'sleep 3' }
                          }
                          stage('sleep 4') {
                              agent { kubernetes { label 'jnlp' } }
                              steps { sh 'sleep 4' }
                          }
                          stage('sleep 5') {
                              agent { kubernetes { label 'jnlp' } }
                              steps { sh 'sleep 5' }
                          }
                      }
                  }
              }
              post { always { chuckNorris() } }
          }
          
          Show
          sodul Stephane Odul added a comment - Under Global Tool Configuration replace the Default Git installation with a custom one that uses a shell script. In our case we have 2 installers with filters: master: the script is just `true` and Tool Home is `/usr/bin/git` !master: the script is installing a simple shell wrapper to perform retry logic when the git commands fails due to transient network or server side glitches. The Tool Home is the path to the wrapper. We use the GitHub Organization Plugin to scan our repos and trigger our CI pipelines, and we use the Kubernetes plugin to launch ephemeral workers for each stage. The kubernetes plugin is configured with a default 'jnlp' pod definition which is nothing more that the default pod definition that comes with the kubernetes plugin. The Jenkinsfile has parallel stages, it does not matter what the stages do. Very randomly, and infrequently, with no apparent cause one of the stages will get an empty git executable name with the stack trace provided in the description.   Sample Jenkinsfile: pipeline { agent none options { ansiColor('xterm') timestamps () buildDiscarder(logRotator(daysToKeepStr: '30')) timeout(time: 5, unit: 'MINUTES') } stages { stage('Tests') { parallel { stage('sleep 1') { agent { kubernetes { label 'jnlp' } } steps { sh 'sleep 1' } } stage('sleep 2') { agent { kubernetes { label 'jnlp' } } steps { sh 'sleep 2' } } stage('sleep 3') { agent { kubernetes { label 'jnlp' } } steps { sh 'sleep 3' } } stage('sleep 4') { agent { kubernetes { label 'jnlp' } } steps { sh 'sleep 4' } } stage('sleep 5') { agent { kubernetes { label 'jnlp' } } steps { sh 'sleep 5' } } } } } post { always { chuckNorris() } } }
          Hide
          sodul Stephane Odul added a comment -

          We have some additional data about the issue. The empty string is because our git installer did not have a fallback value and so the git command ended up being null.

          The root cause is `Failed to get git executable` which is probably returned by the installer but we have no information as to why it is failing.

          In our case the installer is a just a shell script that writes an other shell script which becomes a wrapper for the git commands so that we can have smarter retry logic and better tuned ref specs than possible with the plugin.

          Show
          sodul Stephane Odul added a comment - We have some additional data about the issue. The empty string is because our git installer did not have a fallback value and so the git command ended up being null. The root cause is `Failed to get git executable` which is probably returned by the installer but we have no information as to why it is failing. In our case the installer is a just a shell script that writes an other shell script which becomes a wrapper for the git commands so that we can have smarter retry logic and better tuned ref specs than possible with the plugin.

            People

            • Assignee:
              Unassigned
              Reporter:
              sodul Stephane Odul
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: