Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56593

Jenkins git clone fails intermittently

    Details

    • Similar Issues:

      Description

      We clone three repos in our pipeline from BitBucket:

      1. Shared Libraries
      2. Jenkinsfile
      3. Source code

      The build fails intermittently with logs shown below:

      Started by an SCM change
       > git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to https://bitbucket-server/scm/xxxx/xxxx.git
       > git config remote.origin.url https://bitbucket-server/scm/xxxx/xxxx.git # timeout=10
      Fetching origin...
      Fetching upstream changes from origin
       > git --version # timeout=10
       > git config --get remote.origin.url # timeout=10
      using GIT_ASKPASS to set credentials 
       > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
      Seen branch in repository origin/develop
      Seen branch in repository origin/master
      Seen 2 remote branches
      Obtained Jenkinsfile from c310abe88924c711a29be22b659dd719ac680942
      Running in Durability level: MAX_SURVIVABILITY
      Loading library utilities@xxxx
      Attempting to resolve xxxx from remote references...
       > git --version # timeout=10
      using GIT_SSH to set credentials 
       > git ls-remote ssh://git@xxxx/xxxx/jenkins-libraries.git # timeout=10
      ERROR: Timeout after 10 minutes
      hudson.plugins.git.GitException: Command "git ls-remote ssh://git@xxxx:xxxx/xxxx/jenkins-libraries.git" returned status code 128:
      stdout: 
      stderr: Terminated
      fatal: Could not read from remote repository.
      
      Please make sure you have the correct access rights
      and the repository exists.
      
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1666)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1657)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2877)
          at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:806)
          at jenkins.scm.api.SCMSource.fetch(SCMSource.java:617)
          at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:83)
          at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:98)
          at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:83)
          at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
          at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
          at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
          at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
          at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
          at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
          at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
          at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
          at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
          at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
          at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135)
          at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:429)
      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      WorkflowScript: Loading libraries failed
      
      1 error
      
          at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
          at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
          at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
          at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
          at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
          at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
          at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
          at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
          at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135)
          at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE
      

      On triggering the same job manually after some time, it succeeds. I do not mean to say here that the poll scm is the issue. Sometimes it succeeds. We are using Jenkins in master - slave configuration. For the master, jenkins/jenkins:2.168 image is used and jenkins/jnlp-slave:3.10-1 for slave. This entire setup is running on Azure Kubernetes Service and deployed through Helm. For git, we use Git plugin version 3.9.3

      Edit 1:

      Please find below an excerpt of the logs:

      Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.711748] Killed process 30955 (git) total-vm:15612kB, anon-rss:144kB, file-rss:900kB, shmem-rss:0kB
      Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.725732] Memory cgroup out of memory: Kill process 26054 (java) score 1886 or sacrifice child
      Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.733299] Killed process 30969 (git) total-vm:15612kB, anon-rss:144kB, file-rss:876kB, shmem-rss:0kB
      Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.760196] Memory cgroup out of memory: Kill process 26054 (java) score 1887 or sacrifice child
      Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.770192] Killed process 26054 (java) total-vm:13656520kB, anon-rss:3995608kB, file-rss:19420kB, shmem-rss:0kB
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.728959] IPv6: ADDRCONF(NETDEV_UP): azv67869479f64: link is not ready
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.729980] azure0: port 14(azv67869479f64) entered blocking state
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.729982] azure0: port 14(azv67869479f64) entered disabled state
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.730045] device azv67869479f64 entered promiscuous mode
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760317] IPv6: ADDRCONF(NETDEV_CHANGE): azv67869479f64: link becomes ready
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760382] azure0: port 14(azv67869479f64) entered blocking state
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760384] azure0: port 14(azv67869479f64) entered forwarding state
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760894] eth0: renamed from azv67869479f642
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.780184] azure0: port 14(azv67869479f64) entered blocking state
      Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.780186] azure0: port 14(azv67869479f64) entered forwarding state
      Mar 26 12:35:08 aks-prodagtpool-18673252-1 kernel: [2626416.564062] device azv67869479f64 left promiscuous mode
      ---
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102751] java invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=928
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102752] java cpuset=efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe mems_allowed=0
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102758] CPU: 5 PID: 11193 Comm: java Not tainted 4.15.0-1037-azure #39~16.04.1-Ubuntu
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102759] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102760] Call Trace:
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102766]  dump_stack+0x63/0x82
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102770]  dump_header+0x77/0x285
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102772]  oom_kill_process+0x22e/0x450
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102774]  out_of_memory+0x11d/0x4c0
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102778]  mem_cgroup_out_of_memory+0x4b/0x80
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102780]  mem_cgroup_oom_synchronize+0x32a/0x350
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102782]  ? mem_cgroup_css_reset+0xe0/0xe0
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102784]  pagefault_out_of_memory+0x36/0x7b
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102788]  mm_fault_error+0x8f/0x190
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102792]  ? handle_mm_fault+0xcc/0x1c0
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102794]  __do_page_fault+0x4cd/0x500
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102797]  ? SyS_futex+0x7f/0x180
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102799]  do_page_fault+0x2e/0xf0
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102803]  ? page_fault+0x2f/0x50
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102805]  page_fault+0x45/0x50
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102807] RIP: 0033:0x7f00fc8cced2
      ---
      podf333c6d9-4642-11e9-bc91-468dba4b2a56/efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe killed as a result of limit of /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212950] memory: usage 4194304kB, limit 4194304kB, failcnt 359672
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212951] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212952] kmem: usage 72724kB, limit 9007199254740988kB, failcnt 0
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212952] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212961] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/16c243b50272a881b6786a8b0ef20ffe0e27d7876e8d1399f97227db2bdf5919: cache:0KB rss:44KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212970] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/a5c5b66f6ef785fd6a110226a041960ba7d4f6c1d8927a769d2c3b79d5973ea1: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212978] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/3ae2c3e5cac91453836f9df37ea67add55d9a04ad79eeed74a82d962a3632968: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
      Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212986] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe: cache:136KB rss:4121400KB rss_huge:2103296KB shmem:0KB mapped_file:4KB dirty:4KB writeback:0KB inactive_anon:0KB active_anon:4121400KB inactive_file:80KB active_file:56KB unevictable:0KB
      

      Jenkins is the only application running in the pod. You can take a look at screenshots for details about java versions.

       

      Edit 2:

      The Jenkins uses polling mechanism instead to get status of all the repositories

        Attachments

          Activity

          Hide
          csanchez Carlos Sanchez added a comment -

          in your pod template you can set the environment variable _JAVA_OPTIONS=-Xmx2048m

          Show
          csanchez Carlos Sanchez added a comment - in your pod template you can set the environment variable _JAVA_OPTIONS=-Xmx2048m
          Hide
          milinds747 Milind Sonawane added a comment -

          Sure. Will try that.

          Show
          milinds747 Milind Sonawane added a comment - Sure. Will try that.
          Hide
          milinds747 Milind Sonawane added a comment -

          The job fails with following message:

          Picked up _JAVA_OPTIONS: -Xmx2048
          Error occurred during initialization of VM
          Too small initial heap

          Increasing this param does not help, will go towards the same problem again.

          Show
          milinds747 Milind Sonawane added a comment - The job fails with following message: Picked up _JAVA_OPTIONS: -Xmx2048 Error occurred during initialization of VM Too small initial heap Increasing this param does not help, will go towards the same problem again.
          Hide
          csanchez Carlos Sanchez added a comment -

          you are missing the m -Xmx2048m

          Show
          csanchez Carlos Sanchez added a comment - you are missing the m -Xmx2048m
          Hide
          milinds747 Milind Sonawane added a comment -

          Update on this issue, following changes are made:

          1. Instead of poll SCM, the projects were configured to use webhooks (We have over 100 pipelines in Jenkins).
          2. Replaced SSH by HTTPS for cloning the git repos.

          The failures have not been seen for one week after applying this config.

          Show
          milinds747 Milind Sonawane added a comment - Update on this issue, following changes are made: Instead of poll SCM, the projects were configured to use webhooks (We have over 100 pipelines in Jenkins). Replaced SSH by HTTPS for cloning the git repos. The failures have not been seen for one week after applying this config.

            People

            • Assignee:
              Unassigned
              Reporter:
              milinds747 Milind Sonawane
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: