-
Bug
-
Resolution: Not A Defect
-
Minor
-
None
-
Dev
We clone three repos in our pipeline from BitBucket:
- Shared Libraries
- Jenkinsfile
- Source code
The build fails intermittently with logs shown below:
Started by an SCM change > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://bitbucket-server/scm/xxxx/xxxx.git > git config remote.origin.url https://bitbucket-server/scm/xxxx/xxxx.git # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* Seen branch in repository origin/develop Seen branch in repository origin/master Seen 2 remote branches Obtained Jenkinsfile from c310abe88924c711a29be22b659dd719ac680942 Running in Durability level: MAX_SURVIVABILITY Loading library utilities@xxxx Attempting to resolve xxxx from remote references... > git --version # timeout=10 using GIT_SSH to set credentials > git ls-remote ssh://git@xxxx/xxxx/jenkins-libraries.git # timeout=10 ERROR: Timeout after 10 minutes hudson.plugins.git.GitException: Command "git ls-remote ssh://git@xxxx:xxxx/xxxx/jenkins-libraries.git" returned status code 128: stdout: stderr: Terminated fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1666) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1657) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:2877) at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:806) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:617) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:83) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:98) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:83) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138) at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: Loading libraries failed 1 error at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Finished: FAILURE
On triggering the same job manually after some time, it succeeds. I do not mean to say here that the poll scm is the issue. Sometimes it succeeds. We are using Jenkins in master - slave configuration. For the master, jenkins/jenkins:2.168 image is used and jenkins/jnlp-slave:3.10-1 for slave. This entire setup is running on Azure Kubernetes Service and deployed through Helm. For git, we use Git plugin version 3.9.3
Edit 1:
Please find below an excerpt of the logs:
Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.711748] Killed process 30955 (git) total-vm:15612kB, anon-rss:144kB, file-rss:900kB, shmem-rss:0kB Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.725732] Memory cgroup out of memory: Kill process 26054 (java) score 1886 or sacrifice child Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.733299] Killed process 30969 (git) total-vm:15612kB, anon-rss:144kB, file-rss:876kB, shmem-rss:0kB Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.760196] Memory cgroup out of memory: Kill process 26054 (java) score 1887 or sacrifice child Mar 26 11:54:50 aks-prodagtpool-18673252-1 kernel: [2623998.770192] Killed process 26054 (java) total-vm:13656520kB, anon-rss:3995608kB, file-rss:19420kB, shmem-rss:0kB Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.728959] IPv6: ADDRCONF(NETDEV_UP): azv67869479f64: link is not ready Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.729980] azure0: port 14(azv67869479f64) entered blocking state Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.729982] azure0: port 14(azv67869479f64) entered disabled state Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.730045] device azv67869479f64 entered promiscuous mode Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760317] IPv6: ADDRCONF(NETDEV_CHANGE): azv67869479f64: link becomes ready Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760382] azure0: port 14(azv67869479f64) entered blocking state Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760384] azure0: port 14(azv67869479f64) entered forwarding state Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.760894] eth0: renamed from azv67869479f642 Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.780184] azure0: port 14(azv67869479f64) entered blocking state Mar 26 12:34:19 aks-prodagtpool-18673252-1 kernel: [2626367.780186] azure0: port 14(azv67869479f64) entered forwarding state Mar 26 12:35:08 aks-prodagtpool-18673252-1 kernel: [2626416.564062] device azv67869479f64 left promiscuous mode --- Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102751] java invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=928 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102752] java cpuset=efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe mems_allowed=0 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102758] CPU: 5 PID: 11193 Comm: java Not tainted 4.15.0-1037-azure #39~16.04.1-Ubuntu Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102759] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102760] Call Trace: Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102766] dump_stack+0x63/0x82 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102770] dump_header+0x77/0x285 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102772] oom_kill_process+0x22e/0x450 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102774] out_of_memory+0x11d/0x4c0 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102778] mem_cgroup_out_of_memory+0x4b/0x80 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102780] mem_cgroup_oom_synchronize+0x32a/0x350 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102782] ? mem_cgroup_css_reset+0xe0/0xe0 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102784] pagefault_out_of_memory+0x36/0x7b Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102788] mm_fault_error+0x8f/0x190 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102792] ? handle_mm_fault+0xcc/0x1c0 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102794] __do_page_fault+0x4cd/0x500 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102797] ? SyS_futex+0x7f/0x180 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102799] do_page_fault+0x2e/0xf0 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102803] ? page_fault+0x2f/0x50 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102805] page_fault+0x45/0x50 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.102807] RIP: 0033:0x7f00fc8cced2 --- podf333c6d9-4642-11e9-bc91-468dba4b2a56/efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe killed as a result of limit of /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212950] memory: usage 4194304kB, limit 4194304kB, failcnt 359672 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212951] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212952] kmem: usage 72724kB, limit 9007199254740988kB, failcnt 0 Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212952] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212961] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/16c243b50272a881b6786a8b0ef20ffe0e27d7876e8d1399f97227db2bdf5919: cache:0KB rss:44KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:44KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212970] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/a5c5b66f6ef785fd6a110226a041960ba7d4f6c1d8927a769d2c3b79d5973ea1: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212978] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/3ae2c3e5cac91453836f9df37ea67add55d9a04ad79eeed74a82d962a3632968: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB Mar 26 11:54:48 aks-prodagtpool-18673252-1 kernel: [2623997.212986] Memory cgroup stats for /kubepods/burstable/podf333c6d9-4642-11e9-bc91-468dba4b2a56/efb08efb1c859a563aa13dd108c40a1d0572de9ad1363c7ea02fd9fc6e8b8ebe: cache:136KB rss:4121400KB rss_huge:2103296KB shmem:0KB mapped_file:4KB dirty:4KB writeback:0KB inactive_anon:0KB active_anon:4121400KB inactive_file:80KB active_file:56KB unevictable:0KB
Jenkins is the only application running in the pod. You can take a look at screenshots for details about java versions.
Edit 2:
The Jenkins uses polling mechanism instead to get status of all the repositories
