Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11355

CLI stops working. java.lang.OutOfMemoryError: unable to create new native thread

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      I run frequent scripts on a node called "utils" using the groovy CLI to communicate with Jenkins. After a couple of hours, the CLI stops working. I've found several java.lang.OutOfMemoryError: unable to create new native thread in the log file. I've checked the memory and it uses 500 mb out of 1.5gb allocated. I searched the net and found that it could be caused by too many threads.

      I've done a thread dump and found that the node called "utils" has 1074 threads that look like this:

      ComThread for Thread-1022

      "ComThread for Thread-1022" Id=1497 Group=main RUNNABLE (in native)
      at com4j.Win32Lock.suspend0(Native Method)
      at com4j.Win32Lock.suspend(Win32Lock.java:33)
      at com4j.ComThread.run0(ComThread.java:135)
      at com4j.ComThread.run(ComThread.java:125)

      Finally, Jenkins crashes with out of memory. Attached is the thread dump.

      Any ideas? It started around the time when I finally understood how the groovy CLI scripts work and added a few of them.

      -------------------

      The groovy scripts I run are:

       
      Thread.getAllStackTraces().keySet().each() { 
              item ->
              if (item.getName().contains("SCM polling") && item.getName().contains("waiting for hudson.remoting")) { 
                      println "Interrupting thread " + item.getId(); 
                      item.interrupt() 
              }
      }
      
       
      def slaves = hudson.model.Hudson.instance.getNodes()
      for (slave in slaves) {
        def comp = slave.getComputer() 
        def executors = comp.getExecutors() 
        if (executors != null ) {
          for (executor in executors) {
            def workunit = executor.getCurrentWorkUnit() 
            if (workunit != null) {
              def subtask = workunit.work
              println  "Slavename:" + slave.getDisplayName() + ";Task:BUSY:" + subtask.getDisplayName()
            }
            else {
              println  "Slavename:" + slave.getDisplayName() + ";Task:IDLE"
            }
          }
        }
      }
      
       
      def slaves = hudson.model.Hudson.instance.getNodes()
      for (slave in slaves) {
        def comp = slave.getComputer() 
        if (comp.isOnline()) {
          println "Name:" + comp.getName() + "; Status: ONLINE; Hostname:" + comp.getHostName() + "; Idle:" + comp.countIdle()
        }
        else{
          println "Name:" + comp.getName() + "; Status: OFFLINE;  Hostname:NA" + "; Idle:0"
        }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            mabahj Markus added a comment -

            This could of course be unrelated to the groovy CLI scripts, but on that note: I run these scripts at least every 30 seconds.

            Show
            mabahj Markus added a comment - This could of course be unrelated to the groovy CLI scripts, but on that note: I run these scripts at least every 30 seconds.
            Hide
            mabahj Markus added a comment -

            I have now disabled the groovy CLI scripts and there are only 94 "ComThread for Thread-xxx" threads. The server has been running about 3 days. This is to me a strong indication that the groovy CLI scripts are causing a thread leak.

            Show
            mabahj Markus added a comment - I have now disabled the groovy CLI scripts and there are only 94 "ComThread for Thread-xxx" threads. The server has been running about 3 days. This is to me a strong indication that the groovy CLI scripts are causing a thread leak.
            Hide
            tofuatjava Thomas Fürer added a comment -

            have nearly the same issue on the linux master with windows slaves. version i run is 1.436 but I think I have this problem also with previous versions. Currently I try to monitor this problem. maybe it depends also on SCTMExecutor or xUnit plugin.

            Show
            tofuatjava Thomas Fürer added a comment - have nearly the same issue on the linux master with windows slaves. version i run is 1.436 but I think I have this problem also with previous versions. Currently I try to monitor this problem. maybe it depends also on SCTMExecutor or xUnit plugin.
            Hide
            mabahj Markus added a comment -

            I've moved the server to linux but there I got "java.lang.OutOfMemoryError: PermGen space". I've therefore rewritten the scripts to use the XML API instead of groovy CLI scripts and it now seems to have stabilized.

            Show
            mabahj Markus added a comment - I've moved the server to linux but there I got "java.lang.OutOfMemoryError: PermGen space". I've therefore rewritten the scripts to use the XML API instead of groovy CLI scripts and it now seems to have stabilized.
            Hide
            tofuatjava Thomas Fürer added a comment -

            I also found the issue in my system. the problem was a distribution upgrade which greates a completly new configuration for my tomcat. so the heap gets a maxiumum size of 128m. changed Xmx to 1g and watching if the problem is solved.

            Show
            tofuatjava Thomas Fürer added a comment - I also found the issue in my system. the problem was a distribution upgrade which greates a completly new configuration for my tomcat. so the heap gets a maxiumum size of 128m. changed Xmx to 1g and watching if the problem is solved.

              People

              • Assignee:
                Unassigned
                Reporter:
                mabahj Markus
              • Votes:
                1 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated: