-
Bug
-
Resolution: Not A Defect
-
Minor
Recently Jenkins disconnected from one of our linux slaves. When we tried restarting it, it failed. Investigating found insufficient memory to start Java (runaway python processes). The Agent, however, decided this was a problem with java not being installed and tried to reinstall it.
More: The log in jenkins shows the ssh agent connecting to the slave, verifying environment settings, then attempting to get the Java Version. Due to low-memory, it can't start Java to get its version (Java err12). After this failure it tries a few other locations for the Java bin that we do not use, then determines Java must not be installed. It then attempts to download and install Java, which fails due to Oracle's credential requirement for downloads (reported in other tickets, but not a bug, we install Java by puppet/hand).
Expected: It finds that Java exists, but fails to start due to Memory (errno 12), reports this and exits. It does NOT try to install Java.
The OOM condition is unrelated to Java (runaway python processes on the same machine), and we are addressing the xxMaxPermSize attribute though it should not have contributed here either.
[04/24/18 09:47:25] [SSH] Opening SSH connection to ourslave.build.slaves:22. [04/24/18 09:47:25] [SSH] Authentication successful. [04/24/18 09:47:25] [SSH] The remote user's environment is: BASH=/bin/bash BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_EXECUTION_STRING=set BASH_LINENO=() BASH_SOURCE=() BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu") BASH_VERSION='4.3.30(1)-release' DIRSTACK=() EUID=1021 GROUPS=() HOME=/opt/jenkins HOSTNAME=ourslave HOSTTYPE=x86_64 IFS=$' \t\n' LANG=en_US.UTF-8 LOGNAME=jenkins-node MACHTYPE=x86_64-pc-linux-gnu MAIL=/var/mail/jenkins OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/local/bin:/usr/bin:/bin:/usr/games PIPESTATUS=([0]="0") PPID=2468 PS4='+ ' PWD=/opt/jenkins SHELL=/bin/bash SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=1 SSH_CLIENT='10.10.10.10 54669 22' SSH_CONNECTION='10.10.10.10 54669 10.10.10.11 22' TERM=dumb UID=1001 USER=jenkins _=']' [04/24/18 09:47:25] [SSH] Checking java version of java Couldn't figure out the Java version of java Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128M; support was removed in 8.0 # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 348913664 bytes for committing reserved memory. # An error report file with more information is saved as: # /opt/jenkins/hs_err_pid24687.log Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000640000000, 348913664, 0) failed; error='Cannot allocate memory' (errno=12) [04/24/18 09:47:25] [SSH] Checking java version of /usr/bin/java Couldn't figure out the Java version of /usr/bin/java Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128M; support was removed in 8.0 # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 348913664 bytes for committing reserved memory. # An error report file with more information is saved as: # /opt/jenkins/hs_err_pid24688.log Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000640000000, 348913664, 0) failed; error='Cannot allocate memory' (errno=12) [04/24/18 09:47:25] [SSH] Checking java version of /usr/java/default/bin/java Couldn't figure out the Java version of /usr/java/default/bin/java bash: /usr/java/default/bin/java: No such file or directory [04/24/18 09:47:25] [SSH] Checking java version of /usr/java/latest/bin/java Couldn't figure out the Java version of /usr/java/latest/bin/java bash: /usr/java/latest/bin/java: No such file or directory [04/24/18 09:47:25] [SSH] Checking java version of /usr/local/bin/java Couldn't figure out the Java version of /usr/local/bin/java bash: /usr/local/bin/java: No such file or directory [04/24/18 09:47:25] [SSH] Checking java version of /usr/local/java/bin/java Couldn't figure out the Java version of /usr/local/java/bin/java bash: /usr/local/java/bin/java: No such file or directory [04/24/18 09:47:25] [SSH] Checking java version of /opt/jenkins/jdk/bin/java Couldn't figure out the Java version of /opt/jenkins/jdk/bin/java bash: /opt/jenkins/jdk/bin/java: No such file or directory Linux ourslave 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux Installing JDK jdk-8u121-oth-JPR Downloading JDK from http://download.oracle.com/otn/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-x64.tar.gz Your Oracle account doesn't appear valid. Please specify a valid username/password java.io.IOException: Could not find any known supported java version in [java, /usr/bin/java, /usr/java/default/bin/java, /usr/java/latest/bin/java, /usr/local/bin/java, /usr/local/java/bin/java, /opt/jenkins/jdk/bin/java], and we also failed to install JDK as a fallback at hudson.plugins.sshslaves.SSHLauncher.resolveJava(SSHLauncher.java:918) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:809) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: hudson.AbortException: Unable to install JDK unless a valid username/password is provided. at hudson.tools.JDKInstaller.locate(JDKInstaller.java:537) at hudson.plugins.sshslaves.SSHLauncher.attemptToInstallJDK(SSHLauncher.java:1039) at hudson.plugins.sshslaves.SSHLauncher.resolveJava(SSHLauncher.java:916) ... 6 more [04/24/18 09:47:28] Launch failed - cleaning up connection [04/24/18 09:47:28] [SSH] Connection closed.