Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60297

Jenkins could not read from remote repository

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Not A Defect
    • Component/s: git-plugin
    • Environment:
    • Similar Issues:

      Description

      Jenkins fails sometimes with the error: Could not read from remote repository.

      Jenkins runs on a local Win10 machine with access rights to our local server. On the server there is the GIT repository stored. The WIN10 user account jenkins runs the jenkins-service with has access rights to the server. Communication to the server is stable.

      This happend after a update from jenkins 2.176.3 to 2.190.2 (and update of all plugins).

      The error occurs very often, if more than one build-processor is used (parallel builds). If only one build-processor is used, the error occurs less, but it still occurs.

       

      The following console output is generated:

      Build wurde zeitgesteuert ausgelöst.
      Running as SYSTEM
      Baue auf Master in Arbeitsbereich C:\jenkins\workspace\xxxxx
      No credentials specified
      > git.exe rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
      > git.exe config remote.origin.url //xxxx/xxxxx/xxxx/xxxx/xxxx/xxxx.git # timeout=10
      Cleaning workspace
      > git.exe rev-parse --verify HEAD # timeout=10
      Resetting working tree
      > git.exe reset --hard # timeout=10
      > git.exe clean -fdx # timeout=10
      > git.exe submodule foreach --recursive git reset --hard # timeout=10
      > git.exe submodule foreach --recursive git clean -fdx # timeout=10
      Fetching upstream changes from //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git
      > git.exe --version # timeout=10
      > git.exe fetch --tags --force --progress – //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git +refs/heads/:refs/remotes/origin/ # timeout=10
      ERROR: Error fetching remote repo 'origin'
      hudson.plugins.git.GitException: Failed to fetch from //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:899)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1114)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1145)
      at hudson.scm.SCM.checkout(SCM.java:504)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
      at hudson.model.Run.execute(Run.java:1815)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:97)
      at hudson.model.Executor.run(Executor.java:429)
      Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --force --progress – //xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git +refs/heads/:refs/remotes/origin/" returned status code 128:
      stdout:
      stderr: fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository
      fatal: Could not read from remote repository.

      Please make sure you have the correct access rights
      and the repository exists.

      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2372)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1985)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:80)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:563)
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:897)
      ... 11 more
      ERROR: Error fetching remote repo 'origin'
      [WARNINGS] Skipping publisher since build result is FAILURE
      Archiviere Artefakte
      Sending e-mails to: xxxxr@xxxx.com xxxx@xxxx.com
      Finished: FAILURE

       

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment - - edited

          The message from command line git is:

          fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository
          

          If you're accessing the remote git repository through https or ssh, that might indicate that the remote git repository is temporarily inaccessible or that remote server has some issue with its file system. In my case, I can generate that message when I use a a valid ssh private key credential to clone a repository which does not exist on a remote server which I host myself. My example looks like this:

          mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone mwaite@git.markwaite.net:git/bare/jenkins/not-really-a-repository.git
          Cloning into 'not-really-a-repository'...
          fatal: 'git/bare/jenkins/not-really-a-repository.git' does not appear to be a git repository
          fatal: Could not read from remote repository.
          
          Please make sure you have the correct access rights
          and the repository exists.
          

          However, when I attempt to use a valid ssh private key to clone a non-existent repository from github.com, I don't see that exact message. I see:

          mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone git@github.com:MarkEWaite/not-really-a-repository.git
          Cloning into 'not-really-a-repository'...
          ERROR: Repository not found.
          fatal: Could not read from remote repository.
          
          Please make sure you have the correct access rights
          and the repository exists.
          

          Based on that, I would guess that you're cloning a repository from a server that you're hosting. That internal server is temporarily unable to detect the repository, especially during times of high load (concurrent clones). You could test that theory by running multiple git clone commands on different machines at the same time.

          If you're accessing the remote git repository through file system access (for example through an NFS mount or a CIFS mount), that message might indicate that command line git was unable to read the directory it needed. It might also mean that it was unable to find a specific file or to acquire a lock on a specific file. If you're accessing a git repository through a remote file access protocol, you should reconsider that choice and use the remote protocols that are included with command line git (http/https and ssh).

          Show
          markewaite Mark Waite added a comment - - edited The message from command line git is: fatal: '//xxxx/xxxx/xxxx/xxxx/xxxx/xxxx.git' does not appear to be a git repository If you're accessing the remote git repository through https or ssh, that might indicate that the remote git repository is temporarily inaccessible or that remote server has some issue with its file system. In my case, I can generate that message when I use a a valid ssh private key credential to clone a repository which does not exist on a remote server which I host myself. My example looks like this: mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone mwaite@git.markwaite.net:git/bare/jenkins/not-really-a-repository.git Cloning into 'not-really-a-repository'... fatal: 'git/bare/jenkins/not-really-a-repository.git' does not appear to be a git repository fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. However, when I attempt to use a valid ssh private key to clone a non-existent repository from github.com, I don't see that exact message. I see: mwaite@mark-pc2:~/git/jenkins/git-plugin$ git clone git@github.com:MarkEWaite/not-really-a-repository.git Cloning into 'not-really-a-repository'... ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Based on that, I would guess that you're cloning a repository from a server that you're hosting. That internal server is temporarily unable to detect the repository, especially during times of high load (concurrent clones). You could test that theory by running multiple git clone commands on different machines at the same time. If you're accessing the remote git repository through file system access (for example through an NFS mount or a CIFS mount), that message might indicate that command line git was unable to read the directory it needed. It might also mean that it was unable to find a specific file or to acquire a lock on a specific file. If you're accessing a git repository through a remote file access protocol, you should reconsider that choice and use the remote protocols that are included with command line git (http/https and ssh).
          Hide
          benjaminloeffler Benjamin Loeffler added a comment -

          Thanks Mark Waite for this very fast response!

          But I cant resolve the issue.

          Yes, I clone a repository from a server we host. The error occures, when just a single clone is done and when nothing happens on the server (tried at night and on the early morning when noone is at work).

          I clone the GIT repo in the jenkins-job "Source-Code-Management" in "Repository URL" via //SERVERNAME/xxxxx/xxxxx/xxxxx/xxxxx/xxxx.git

          No credentials are used. (The error occures too if credentials are used: username and password of the user jenkins is running and which has access rights to the server)

          This should work without ssh or https right? Use of ssh or https is not needed, its a internal server and access is only via the LAN.

          So whats the poblem that it works sometimes and sometimes not?

          Show
          benjaminloeffler Benjamin Loeffler added a comment - Thanks Mark Waite for this very fast response! But I cant resolve the issue. Yes, I clone a repository from a server we host. The error occures, when just a single clone is done and when nothing happens on the server (tried at night and on the early morning when noone is at work). I clone the GIT repo in the jenkins-job "Source-Code-Management" in "Repository URL" via //SERVERNAME/xxxxx/xxxxx/xxxxx/xxxxx/xxxx.git No credentials are used. (The error occures too if credentials are used: username and password of the user jenkins is running and which has access rights to the server) This should work without ssh or https right? Use of ssh or https is not needed, its a internal server and access is only via the LAN. So whats the poblem that it works sometimes and sometimes not?
          Hide
          markewaite Mark Waite added a comment - - edited

          As far as I can tell, the issue is reported by command line git and can only be handled by command line git. The git plugin might be extended to retry on command line git errors, but retry on error is a complication that has not been added to the Jenkins git plugin yet.

          I assume that //SERVERNAME/path/to/repository is accessing the git repository through a Windows network file system protocol (likely CIFS). If so, then you'll need to request assistance from the git for Windows development team or the git development team. I have a personal suspicion that the git team does not spend significant effort assuring that the various implementations of CIFS (Windows 7, Windows 10, Windows Server 2008, Windows Server 2012, Windows Server 2016, Windows Server 2019, Samba in various versions on FreeNAS, FreeBSD, Linux, and macOS) provide the necessary file locking and data transfer semantics that command line git assumes.

          Refer to this stackoverflow article for information from one user that struggled with a git repository over CIFS.

          If you must use CIFS, then you may consider enabling core.ignoreStat in hopes of reducing issues.

          I don't plan to spend any further effort on this issue.

          Show
          markewaite Mark Waite added a comment - - edited As far as I can tell, the issue is reported by command line git and can only be handled by command line git. The git plugin might be extended to retry on command line git errors, but retry on error is a complication that has not been added to the Jenkins git plugin yet. I assume that //SERVERNAME/path/to/repository is accessing the git repository through a Windows network file system protocol (likely CIFS). If so, then you'll need to request assistance from the git for Windows development team or the git development team. I have a personal suspicion that the git team does not spend significant effort assuring that the various implementations of CIFS (Windows 7, Windows 10, Windows Server 2008, Windows Server 2012, Windows Server 2016, Windows Server 2019, Samba in various versions on FreeNAS, FreeBSD, Linux, and macOS) provide the necessary file locking and data transfer semantics that command line git assumes. Refer to this stackoverflow article for information from one user that struggled with a git repository over CIFS. If you must use CIFS, then you may consider enabling core.ignoreStat in hopes of reducing issues. I don't plan to spend any further effort on this issue.
          Hide
          markewaite Mark Waite added a comment -

          Not a defect in the git plugin. Command line git or the remote git server are the most likely places to look

          Show
          markewaite Mark Waite added a comment - Not a defect in the git plugin. Command line git or the remote git server are the most likely places to look

            People

            • Assignee:
              Unassigned
              Reporter:
              benjaminloeffler Benjamin Loeffler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: