Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50973

Insufficient free memory triggers Java install attempt

    Details

    • Similar Issues:

      Description

      Recently Jenkins disconnected from one of our linux slaves. When we tried restarting it, it failed. Investigating found insufficient memory to start Java (runaway python processes). The Agent, however, decided this was a problem with java not being installed and tried to reinstall it.

      More: The log in jenkins shows the ssh agent connecting to the slave, verifying environment settings, then attempting to get the Java Version. Due to low-memory, it can't start Java to get its version (Java err12). After this failure it tries a few other locations for the Java bin that we do not use, then determines Java must not be installed. It then attempts to download and install Java, which fails due to Oracle's credential requirement for downloads (reported in other tickets, but not a bug, we install Java by puppet/hand).

      Expected: It finds that Java exists, but fails to start due to Memory (errno 12), reports this and exits. It does NOT try to install Java.

      The OOM condition is unrelated to Java (runaway python processes on the same machine), and we are addressing the xxMaxPermSize attribute though it should not have contributed here either.

       

      [04/24/18 09:47:25] [SSH] Opening SSH connection to ourslave.build.slaves:22.
      [04/24/18 09:47:25] [SSH] Authentication successful.
      [04/24/18 09:47:25] [SSH] The remote user's environment is:
      BASH=/bin/bash
      BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
      BASH_ALIASES=()
      BASH_ARGC=()
      BASH_ARGV=()
      BASH_CMDS=()
      BASH_EXECUTION_STRING=set
      BASH_LINENO=()
      BASH_SOURCE=()
      BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
      BASH_VERSION='4.3.30(1)-release'
      DIRSTACK=()
      EUID=1021
      GROUPS=()
      HOME=/opt/jenkins
      HOSTNAME=ourslave
      HOSTTYPE=x86_64
      IFS=$' \t\n'
      LANG=en_US.UTF-8
      LOGNAME=jenkins-node
      MACHTYPE=x86_64-pc-linux-gnu
      MAIL=/var/mail/jenkins
      OPTERR=1
      OPTIND=1
      OSTYPE=linux-gnu
      PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
      PIPESTATUS=([0]="0")
      PPID=2468
      PS4='+ '
      PWD=/opt/jenkins
      SHELL=/bin/bash
      SHELLOPTS=braceexpand:hashall:interactive-comments
      SHLVL=1
      SSH_CLIENT='10.10.10.10 54669 22'
      SSH_CONNECTION='10.10.10.10 54669 10.10.10.11 22'
      TERM=dumb
      UID=1001
      USER=jenkins
      _=']'
      [04/24/18 09:47:25] [SSH] Checking java version of java
      Couldn't figure out the Java version of java
      Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128M; support was removed in 8.0
      #
      # There is insufficient memory for the Java Runtime Environment to continue.
      # Native memory allocation (mmap) failed to map 348913664 bytes for committing reserved memory.
      # An error report file with more information is saved as:
      # /opt/jenkins/hs_err_pid24687.log
      Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000640000000, 348913664, 0) failed; error='Cannot allocate memory' (errno=12)
      
      [04/24/18 09:47:25] [SSH] Checking java version of /usr/bin/java
      Couldn't figure out the Java version of /usr/bin/java
      Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128M; support was removed in 8.0
      #
      # There is insufficient memory for the Java Runtime Environment to continue.
      # Native memory allocation (mmap) failed to map 348913664 bytes for committing reserved memory.
      # An error report file with more information is saved as:
      # /opt/jenkins/hs_err_pid24688.log
      Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000640000000, 348913664, 0) failed; error='Cannot allocate memory' (errno=12)
      
      [04/24/18 09:47:25] [SSH] Checking java version of /usr/java/default/bin/java
      Couldn't figure out the Java version of /usr/java/default/bin/java
      bash: /usr/java/default/bin/java: No such file or directory
      
      [04/24/18 09:47:25] [SSH] Checking java version of /usr/java/latest/bin/java
      Couldn't figure out the Java version of /usr/java/latest/bin/java
      bash: /usr/java/latest/bin/java: No such file or directory
      
      [04/24/18 09:47:25] [SSH] Checking java version of /usr/local/bin/java
      Couldn't figure out the Java version of /usr/local/bin/java
      bash: /usr/local/bin/java: No such file or directory
      
      [04/24/18 09:47:25] [SSH] Checking java version of /usr/local/java/bin/java
      Couldn't figure out the Java version of /usr/local/java/bin/java
      bash: /usr/local/java/bin/java: No such file or directory
      
      [04/24/18 09:47:25] [SSH] Checking java version of /opt/jenkins/jdk/bin/java
      Couldn't figure out the Java version of /opt/jenkins/jdk/bin/java
      bash: /opt/jenkins/jdk/bin/java: No such file or directory
      
      Linux ourslave 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux
      Installing JDK jdk-8u121-oth-JPR
      Downloading JDK from http://download.oracle.com/otn/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-x64.tar.gz
      
      Your Oracle account doesn't appear valid. Please specify a valid username/password
      java.io.IOException: Could not find any known supported java version in [java, /usr/bin/java, /usr/java/default/bin/java, /usr/java/latest/bin/java, /usr/local/bin/java, /usr/local/java/bin/java, /opt/jenkins/jdk/bin/java], and we also failed to install JDK as a fallback
          at hudson.plugins.sshslaves.SSHLauncher.resolveJava(SSHLauncher.java:918)
          at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:809)
          at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: hudson.AbortException: Unable to install JDK unless a valid username/password is provided.
          at hudson.tools.JDKInstaller.locate(JDKInstaller.java:537)
          at hudson.plugins.sshslaves.SSHLauncher.attemptToInstallJDK(SSHLauncher.java:1039)
          at hudson.plugins.sshslaves.SSHLauncher.resolveJava(SSHLauncher.java:916)
          ... 6 more
      [04/24/18 09:47:28] Launch failed - cleaning up connection
      [04/24/18 09:47:28] [SSH] Connection closed.
      

        Attachments

          Activity

          tmack0 Theral Mackey created issue -
          Hide
          ifernandezcalvo Ivan Fernandez Calvo added a comment -

          It is an unusual situation and the logs allow to diagnose what it is happening, the Agent is screw up whatever you try to make will fail

          Show
          ifernandezcalvo Ivan Fernandez Calvo added a comment - It is an unusual situation and the logs allow to diagnose what it is happening, the Agent is screw up whatever you try to make will fail
          ifernandezcalvo Ivan Fernandez Calvo made changes -
          Field Original Value New Value
          Status Open [ 1 ] Closed [ 6 ]
          Resolution Not A Defect [ 7 ]

            People

            • Assignee:
              ifernandezcalvo Ivan Fernandez Calvo
              Reporter:
              tmack0 Theral Mackey
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: