Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57725

Proc.executor inherits ContextClassLoader dynamically

    Details

    • Similar Issues:
    • Released As:
      Jenkins 2.181

      Description

      The threads in the thread pool used for Proc.joinWithTimeout inherit their context class loader from the thread that calls Proc.joinWithTimeout, which seems undesirable. While looking into JENKINS-50223, I found a case where one of the threads in the thread pool was holding onto a CleanGroovyClassLoader class loader from workflow-cps (see the attached screenshot), which is definitely undesirable.

      We should wrap this thread pool in ClassLoaderSanityThreadFactory to prevent this kind of issue from happening.

        Attachments

          Issue Links

            Activity

            Hide
            2bluesc Kyle Manna added a comment - - edited

            This was mentioned in the LTS changelog for v2.176.2, but I'm experiencing something related.  My Jenkins system with 20 executors will creep towards 20k Tasks (processes + threads) before it hits the systemd imposed DefaultTasksMax limit of 19660 on my system.

             

             A java heap dump shows that  4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader-> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader.


             

             Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with. Ubuntu 18.04

            Show
            2bluesc Kyle Manna added a comment - - edited This was mentioned in the  LTS changelog for v2.176.2 , but I'm experiencing something related.  My Jenkins system with 20 executors will creep towards 20k Tasks (processes + threads) before it hits the systemd imposed DefaultTasksMax limit of 19660 on my system.    A java heap dump shows that  4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader-> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader .    Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with. Ubuntu 18.04
            Hide
            2bluesc Kyle Manna added a comment - - edited

            Additional plot from netdata tasks section showing the task plot skyrocket before derailing the Jenkins process and making it unstable/undefined when the systemd CGroup imposes the TasksMax limit:

            SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
            java.lang.OutOfMemoryError: unable to create new native thread
            

            Show
            2bluesc Kyle Manna added a comment - - edited Additional plot from netdata tasks section showing the task plot skyrocket before derailing the Jenkins process and making it unstable/undefined when the systemd CGroup imposes the TasksMax limit: SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed java.lang.OutOfMemoryError: unable to create new native thread
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Kyle Manna It looks like you have a distinct issue with two aspects: One, something is causing a bunch of timer threads to be created and not cleaned up, and two, those timer threads are inheriting their context class loader dynamically. The first issue is more serious, to investigate it I would search the jenkinsci organization on GitHub for all uses of java.util.Timer, cross-check that with the set of plugins you have installed, and then examine the ways that the plugins you have installed are using java.util.Timer based on the search. My guess is that something is repeatedly calling new Timer rather than storing a single timer somewhere and reusing it (maybe also better to switch to using an ExecutorService if that is the problem). Either way I'd go ahead and open a separate issue for your problem.

            Show
            dnusbaum Devin Nusbaum added a comment - Kyle Manna It looks like you have a distinct issue with two aspects: One, something is causing a bunch of timer threads to be created and not cleaned up, and two, those timer threads are inheriting their context class loader dynamically. The first issue is more serious, to investigate it I would search the jenkinsci organization on GitHub for all uses of java.util.Timer , cross-check that with the set of plugins you have installed, and then examine the ways that the plugins you have installed are using java.util.Timer based on the search . My guess is that something is repeatedly calling new Timer rather than storing a single timer somewhere and reusing it (maybe also better to switch to using an ExecutorService if that is the problem). Either way I'd go ahead and open a separate issue for your problem.
            Hide
            2bluesc Kyle Manna added a comment -

            Devin Nusbaum thanks for the response, I've created a new issue @ JENKINS-58684 as you recommended and included all my plugins in the list. All of the plugins are pretty mainstream.

            Show
            2bluesc Kyle Manna added a comment - Devin Nusbaum thanks for the response, I've created a new issue @ JENKINS-58684 as you recommended and included all my plugins in the list. All of the plugins are pretty mainstream.

              People

              • Assignee:
                dnusbaum Devin Nusbaum
                Reporter:
                dnusbaum Devin Nusbaum
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: