Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57542

git checkout over ssh fails on windows agents started from msys2 - missing bat file

    Details

    • Similar Issues:

      Description

      While setting up a new windows build slave on a rented virtual windows server (tried  server2012 as well as server2016) I keep running into this problem where Jenkins jobs cannot check out from git on the new agent:

      stderr: error: cannot run c:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7559664601978999582.bat: No such file or directory

      The required batch file is actually produced and quickly deleted again by the jenkins slave. A script I wrote was able to copy the batch file to a different location before it was deleted. It contains:

       

      @echo off
      
      "C:\msys64\usr\bin\ssh.exe" -i "C:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7913781371770019287.key" -l "git" -o "StrictHostKeyChecking=no" %*
      

      The referenced .key file also exists for a short time and contains the correct private key for the git repository.

      It is not clear to me if the file is deleted again before it is invoked, or if it is produced only after it was invoked. I would appreciate pointers how to debug this. Can I introduce wait times after file creation? Since this behaviour is dependent on the windows agent, Windows specific filesystem caching and timing may play a role.

      I was able to reproduce this with three different virtual windows servers: 2 windows cloud servers rented at ionos.de (2012 and 2016) and a 2016 server rented from a different provider. When executing this in a virtualbox windows 10 machine running on top of ubuntu on my own hardware, this issue does not show up.

      Steps to reproduce:

      • Rent a virtual windows server, e.g. from ionos.de
      • Install msys2 and install git and ssh through msys2
      • Install openjdk12 from zip, add to PATH and set JAVA_HOME
      • Define new build agent NEW-WINDOWS-SERVER in Jenkins
      • Start Jenkins slave on windows server like this:
      • Create test job in Jenkins, which checks out a git repository from a git repository over ssh (somethingj like ssh://git@example.com/example-repo)
      • The build steps do not matter, add something simple
      • Build now

      attaching the complete console output. Jenkins version: 2.177. Git client plugin version: 2.7.7

        Attachments

          Activity

          Hide
          t_herzke_2017 T Herzke added a comment - - edited

          The problem occurs under special circumstances that are probably outside of the scope of Jenkins. Closed as won't-fix.

          Show
          t_herzke_2017 T Herzke added a comment - - edited The problem occurs under special circumstances that are probably outside of the scope of Jenkins. Closed as won't-fix.
          Hide
          t_herzke_2017 T Herzke added a comment -

          Workaround recipe:

          1) Identify additional environment variables and their values when starting the msys2 bash via start menu vs by invoking bash.exe from cmd.exe.

          2) Create bash skript that sets these environment variables in multiple export NAME=value lines, also add the extended PATH value. Store as jenkins.env in msys2 root folder.

          3) In the Jenkinsfiles, define custom step windows_bash

          def windows_bash(command) {
            bat ('C:\\msys64\\usr\\bin\\bash -c "source /jenkins.env && ' + command + ' "')
          }
          

          4) In the Jenkinsfiles, define a delegating custom step that forwards to either sh or windows_bash depending on the operating system of the Agent

          def bash(command) {
            windows() ? windows_bash(command) : sh(command)
          }
          

          where windows() has to be defined to return true only for windows agents.

          5) In all places in the Jenkinsfile where a shell command could be executed either on windows or on Unix, use bash "command" instead of sh "command"

          Show
          t_herzke_2017 T Herzke added a comment - Workaround recipe: 1) Identify additional environment variables and their values when starting the msys2 bash via start menu vs by invoking bash.exe from cmd.exe. 2) Create bash skript that sets these environment variables in multiple export NAME=value lines, also add the extended PATH value. Store as jenkins.env in msys2 root folder. 3) In the Jenkinsfiles, define custom step windows_bash def windows_bash(command) { bat ( 'C:\\msys64\\usr\\bin\\bash -c "source /jenkins.env && ' + command + ' " ' ) } 4) In the Jenkinsfiles, define a delegating custom step that forwards to either sh or windows_bash depending on the operating system of the Agent def bash(command) { windows() ? windows_bash(command) : sh(command) } where windows() has to be defined to return true only for windows agents. 5) In all places in the Jenkinsfile where a shell command could be executed either on windows or on Unix, use bash "command" instead of sh "command"
          Hide
          t_herzke_2017 T Herzke added a comment -

          I can see how running jenkins agent from a unix shell on windows is unusual.

          Reason is my builds are cross-platform (linux, mac, windows) but require bash on every platform. I have found in the past that starting the jenkins agent as a child process of the shell I want to use gives me the least trouble to achieve this on windows. I have used, cygwin, msys, msys2, to achieve this so far with no trouble.

          Starting Jenkins agent from batch file is an alternative that I'm investigating now but the shell commands from Jenkinsfiles like sh "make sometarget" and their child processes show strange behaviour when I do this, like pretending they were called without parameters.

          Show
          t_herzke_2017 T Herzke added a comment - I can see how running jenkins agent from a unix shell on windows is unusual. Reason is my builds are cross-platform (linux, mac, windows) but require bash on every platform. I have found in the past that starting the jenkins agent as a child process of the shell I want to use gives me the least trouble to achieve this on windows. I have used, cygwin, msys, msys2, to achieve this so far with no trouble. Starting Jenkins agent from batch file is an alternative that I'm investigating now but the shell commands from Jenkinsfiles like sh "make sometarget" and their child processes show strange behaviour when I do this, like pretending they were called without parameters.
          Hide
          markewaite Mark Waite added a comment -

          The Windows agents where I test and develop the git plugin are run from either:

          • Windows batch file launched from a command processor window (a "JNLP agent")
          • Windows command processor launched through the OpenSSH server provided by Windows (an ssh agent)

          I prefer the ssh agent because it can be launched from the Jenkins server.

          I've never attempted to run the agent from a git bash shell on Windows. I have never heard of anyone else using that technique either. Is there a specific reason you chose to start the agent from a git bash shell instead of using the Windows command processor?

          Show
          markewaite Mark Waite added a comment - The Windows agents where I test and develop the git plugin are run from either: Windows batch file launched from a command processor window (a "JNLP agent") Windows command processor launched through the OpenSSH server provided by Windows (an ssh agent) I prefer the ssh agent because it can be launched from the Jenkins server. I've never attempted to run the agent from a git bash shell on Windows. I have never heard of anyone else using that technique either. Is there a specific reason you chose to start the agent from a git bash shell instead of using the Windows command processor?
          Hide
          t_herzke_2017 T Herzke added a comment - - edited

          Thanks for the suggestions. They are reasonable and I would follow up on them but I think I have found a workaround since reporting this issue:

          I have now observed that if I invoke the "java -jar slave.jar ..."  on the windows server agent not from the msys2 bash shell but from a git-bash shell, then this problem does not occur and the git checkout over ssh succeeds.

          I have now different problems because the actual Jenkins jobs require the invocation of tools from the msys2 environment which are difficult to invoke from the git-bash environment, but I think I will be able to solve this.

          The relevant difference for this issue that is introduced by the workaround is probably the different set of environment variables passed on to the java slave process by the different bash shells.

           

          Show
          t_herzke_2017 T Herzke added a comment - - edited Thanks for the suggestions. They are reasonable and I would follow up on them but I think I have found a workaround since reporting this issue: I have now observed that if I invoke the "java -jar slave.jar ..."  on the windows server agent not from the msys2 bash shell but from a git-bash shell, then this problem does not occur and the git checkout over ssh succeeds. I have now different problems because the actual Jenkins jobs require the invocation of tools from the msys2 environment which are difficult to invoke from the git-bash environment, but I think I will be able to solve this. The relevant difference for this issue that is introduced by the workaround is probably the different set of environment variables passed on to the java slave process by the different bash shells.  
          Hide
          markewaite Mark Waite added a comment - - edited

          You might be able to compile a custom version of the git client plugin which includes more diagnostic messages in the batch file with steps like:

          IF EXISTS "C:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7913781371770019287.key" ECHO Found key file
          

          in case something is causing those files to not be detected.

          You might also try google searches for specific cases where programs related to git report "No such file or directory" on Windows. That message seems to indicate in at least one case that a directory name was expected to be provided in a different format. Another hint indicated that message might be coming from bash rather than from the Windows command processor. If so, then there may be something incorrect in the git client plugin's detection of which operating system command (sh or bat) should be used to launch the process.

          Show
          markewaite Mark Waite added a comment - - edited You might be able to compile a custom version of the git client plugin which includes more diagnostic messages in the batch file with steps like: IF EXISTS "C:\msys64\home\jenkins\jenkins\workspace\test-tools\test@tmp\ssh7913781371770019287.key" ECHO Found key file in case something is causing those files to not be detected. You might also try google searches for specific cases where programs related to git report " No such file or directory " on Windows. That message seems to indicate in at least one case that a directory name was expected to be provided in a different format. Another hint indicated that message might be coming from bash rather than from the Windows command processor. If so, then there may be something incorrect in the git client plugin's detection of which operating system command ( sh or bat ) should be used to launch the process.

            People

            • Assignee:
              Unassigned
              Reporter:
              t_herzke_2017 T Herzke
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: