Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61103

LinkageError on agent following RemotingSystemException(InterruptedException)

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: maven-plugin, remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.204.2 (remoting 3.36.1)
      Linux OpenJDK 1.8.0_242
    • Similar Issues:

      Description

      I know this sound like a duplicate of some previously reported bugs, but I'm opening this new issue to have a single one I can refer to in a PR I'm about to open.

      I'm trying to improve resilience of the agent's RemoteClassLoader when it receives a RemotingSystemException caused by an InterruptedException.

      The overall issue is that the class loader can't really recover after an exception has been raised while some classes were being loaded / initialized; later attempts at using a class that failed being loaded/initialized will raise a LinkageError.
      This is not something new, see JENKINS-36991 / PR #94. This one got fixed by adding some try/catch/retry logic to the affected part of the RemoteClassLoader#findClass code. This way, the interruption can be deferred until after the class has been successfully loaded.

      But there are other parts of the RemoteClassLoader code which are not protected yet, despite being on the critical path of classes loading/initialization.

      See for instance JENKINS-51854: the exception occurs in a call to RemoteClassLoader#getResource made during initialization of some Subversion plugin classes, and from there the plugin can't be used on this agent anymore.

      I've reproduced something similar with the Maven plugin, simply by interrupting (from the GUI) a Maven job while it was starting on a freshly started agent.

      • error (slave log) when this first Maven job execution gets interrupted:
        Feb 10, 2020 5:51:39 PM hudson.remoting.UserRequest perform
        WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@38569db8
        java.lang.ExceptionInInitializerError
        	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
        	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
        	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
        	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
        	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        	at hudson.remoting.Request$2.run(Request.java:369)
        	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        	at java.lang.Thread.run(Thread.java:748)
        Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
        	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
        	at com.sun.proxy.$Proxy5.getResource2(Unknown Source)
        	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426)
        	at java.lang.ClassLoader.getResource(ClassLoader.java:1090)
        	at java.lang.Class.getResource(Class.java:2267)
        	at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53)
        	at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45)
        	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107)
        	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:128)
        	at org.jvnet.localizer.Localizable.toString(Localizable.java:67)
        	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
        	at hudson.model.Run.<clinit>(Run.java:2535)
        	... 11 more
        Caused by: java.lang.InterruptedException
        	at java.lang.Object.wait(Native Method)
        	at hudson.remoting.Request.call(Request.java:177)
        	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
        	... 22 more
        

      see RemoteClassLoader.java:426

      • error (slave log) when trying to run any other Maven job on the same agent after that:
      Feb 10, 2020 5:53:38 PM hudson.remoting.UserRequest perform
      WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@5e66716c
      java.lang.NoClassDefFoundError: Could not initialize class hudson.maven.MavenModuleSetBuild
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      And with the same manual procedure, I also sometimes get the fatal exception in an other part of RemoteClassLoader#findClass which is not protected covered by the retry logic either (at least, not for RemotingSystemException):

      • error (slave log) when this first Maven job gets interrupted:
      2020-02-11T12:16:49.933+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@601f4e25
      java.lang.ExceptionInInitializerError
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
      	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
      	at com.sun.proxy.$Proxy5.fetch(Unknown Source)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
      	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:89)
      	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:79)
      	at org.eclipse.sisu.plexus.ClassRealmManager.<clinit>(ClassRealmManager.java:66)
      	... 18 more
      Caused by: java.lang.InterruptedException
      	at java.lang.Object.wait(Native Method)
      	at hudson.remoting.Request.call(Request.java:177)
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
      	... 25 more
      

      see RemoteClassLoader.java:315

      • error (slave log) when trying to run any other Maven job on the same agent after that:
      2020-02-11T12:17:41.347+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@69183d46
      java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.sisu.plexus.ClassRealmManager
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
      	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

       

      Both these cases could be fixed by adding more "try / catch / retry with deferred interruption" logic on the relevant parts of the RemoteClassLoader code,
      but instead I propose to do it at a slightly lower level, on methods of the IClassLoader proxy (that's the com.sun.proxy.$Proxy5.getResource2(Unknown Source)
      and com.sun.proxy.$Proxy5.fetch(Unknown Source) in the above stack traces).

      I will submit a PR implementing that (with the retry code in a dynamic proxy). With this change, when I try to interrupt the first execution of a Maven job, there aren't any LinkageError reported anymore. The build is still eventually interrupted of course, but then I can re-execute the job without triggering any NoClassDefFoundError, as one would expect.

        Attachments

          Issue Links

            Activity

            Hide
            tom_gl Thomas de Grenier de Latour added a comment -

            whoops, wrong issue link, fixed

            Show
            tom_gl Thomas de Grenier de Latour added a comment - whoops, wrong issue link, fixed
            Hide
            tom_gl Thomas de Grenier de Latour added a comment -

            For information, another occurrence of this issue we've had in production a few days ago (not involving the Maven plugin):

            • user launch a job, and then cancel it a few seconds later (edited Apache logs):
              IP.IP.IP.IP - xxxxxxxx [17/Feb/2020:16:17:24 +0100] "POST /jenkins/view/VIEW_NAME/job/JOB_NAME/build?delay=0sec HTTP/1.1" 201 241 "http://SOMETHING/jenkins/view/VIEW_NAME/job/JOB_NAME/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"
              IP.IP.IP.IP - xxxxxxxx [17/Feb/2020:16:17:32 +0100] "POST /jenkins/view/VIEW_NAME/job/JOB_NAME/1/stop HTTP/1.1" 302 267 "http://SOMETHING/jenkins/view/VIEW_NAME/job/JOB_NAME/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0"
              
            • slave.log at the time of interruption (it's similar to one of my previous examples, in RemoteClassLoader#findResource() / RemoteClassLoader.java:426):
              Feb 17, 2020 4:17:32 PM hudson.remoting.UserRequest perform
              WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@8a955ea
              java.lang.ExceptionInInitializerError
              	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:44)
              	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
              	at hudson.model.Computer.<clinit>(Computer.java:1761)
              	at hudson.util.ProcessTree.get(ProcessTree.java:414)
              	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
              	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
              	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
              	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
              	at hudson.remoting.Request$2.run(Request.java:369)
              	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              	at java.lang.Thread.run(Thread.java:748)
              Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
              	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
              	at com.sun.proxy.$Proxy5.getResource2(Unknown Source)
              	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426)
              	at java.lang.ClassLoader.getResource(ClassLoader.java:1096)
              	at java.lang.Class.getResource(Class.java:2267)
              	at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53)
              	at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45)
              	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107)
              	at org.jvnet.localizer.Localizable.toString(Localizable.java:67)
              	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
              	at hudson.security.Permission.<clinit>(Permission.java:328)
              	... 14 more
              Caused by: java.lang.InterruptedException
              	at java.lang.Object.wait(Native Method)
              	at hudson.remoting.Request.call(Request.java:177)
              	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
              	... 24 more
              
            • slave.log when trying to execute jobs again (we've then had to disconnect/reconnect the agent):
              Feb 17, 2020 4:20:21 PM hudson.remoting.UserRequest perform
              WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@2813c722
              java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
              	at hudson.util.ProcessTree.get(ProcessTree.java:414)
              	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
              	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
              	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
              	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
              	at hudson.remoting.Request$2.run(Request.java:369)
              	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              	at java.lang.Thread.run(Thread.java:748)
              
              Feb 17, 2020 4:23:41 PM hudson.remoting.UserRequest perform
              WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@73bb794a
              java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
              	at hudson.util.ProcessTree.get(ProcessTree.java:414)
              	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
              	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
              	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
              	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
              	at hudson.remoting.Request$2.run(Request.java:369)
              	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              	at java.lang.Thread.run(Thread.java:748)
              
            Show
            tom_gl Thomas de Grenier de Latour added a comment - For information, another occurrence of this issue we've had in production a few days ago (not involving the Maven plugin): user launch a job, and then cancel it a few seconds later (edited Apache logs): IP.IP.IP.IP - xxxxxxxx [17/Feb/2020:16:17:24 +0100] "POST /jenkins/view/VIEW_NAME/job/JOB_NAME/build?delay=0sec HTTP/1.1" 201 241 "http: //SOMETHING/jenkins/view/VIEW_NAME/job/JOB_NAME/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0" IP.IP.IP.IP - xxxxxxxx [17/Feb/2020:16:17:32 +0100] "POST /jenkins/view/VIEW_NAME/job/JOB_NAME/1/stop HTTP/1.1" 302 267 "http: //SOMETHING/jenkins/view/VIEW_NAME/job/JOB_NAME/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0" slave.log at the time of interruption (it's similar to one of my previous examples, in RemoteClassLoader#findResource() / RemoteClassLoader.java:426 ): Feb 17, 2020 4:17:32 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@8a955ea java.lang.ExceptionInInitializerError at hudson.security.PermissionGroup.<init>(PermissionGroup.java:44) at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64) at hudson.model.Computer.<clinit>(Computer.java:1761) at hudson.util.ProcessTree.get(ProcessTree.java:414) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299) at com.sun.proxy.$Proxy5.getResource2(Unknown Source) at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426) at java.lang. ClassLoader .getResource( ClassLoader .java:1096) at java.lang. Class .getResource( Class .java:2267) at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53) at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45) at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107) at org.jvnet.localizer.Localizable.toString(Localizable.java:67) at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64) at hudson.security.Permission.<clinit>(Permission.java:328) ... 14 more Caused by: java.lang.InterruptedException at java.lang. Object .wait(Native Method) at hudson.remoting.Request.call(Request.java:177) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) ... 24 more slave.log when trying to execute jobs again (we've then had to disconnect/reconnect the agent): Feb 17, 2020 4:20:21 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@2813c722 java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer at hudson.util.ProcessTree.get(ProcessTree.java:414) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Feb 17, 2020 4:23:41 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@73bb794a java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer at hudson.util.ProcessTree.get(ProcessTree.java:414) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)
            Hide
            bladend Adam Romanek added a comment -

            Me and my team have been experiencing this issue for more than a year now. In our environment it breaks our Amazon EC2 slaves.

            Here's the initial ExceptionInInitializerError found in remoting.log on one of our slaves:

            Mar 06, 2020 11:52:33 AM hudson.remoting.UserRequest perform
            WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@75f3b9a6
            java.lang.ExceptionInInitializerError
            	at hudson.util.ProcessTree.get(ProcessTree.java:414)
            	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
            	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
            	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
            	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
            	at hudson.remoting.Request$2.run(Request.java:369)
            	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
            	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
            	at com.sun.proxy.$Proxy5.fetch(Unknown Source)
            	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315)
            	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
            	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            	at hudson.model.Computer.<clinit>(Computer.java:1362)
            	... 11 more
            Caused by: java.lang.InterruptedException
            	at java.lang.Object.wait(Native Method)
            	at hudson.remoting.Request.call(Request.java:177)
            	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
            	... 16 more

            From there one can only see NoClassDefFoundError:

            Mar 06, 2020 11:56:46 AM hudson.remoting.UserRequest perform
            WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@1c3a1ce1
            java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer
            	at hudson.util.ProcessTree.get(ProcessTree.java:414)
            	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103)
            	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
            	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
            	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
            	at hudson.remoting.Request$2.run(Request.java:369)
            	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)

            The issue occurs when a build finishes and ProcessTreeKiller is requested by Jenkins Master to clean up any left-over processes running on a given slave.

            Unfortunately due to a bug in JDK the issue is rather hard to track down, as the logs in our builds have always had this NoClassDefFoundError exception logged, never the one causing all the trouble - ExceptionInInitializerError.

            I'm really looking forward to get it fixed. Please let me know if I could help somehow.

            Show
            bladend Adam Romanek added a comment - Me and my team have been experiencing this issue for more than a year now. In our environment it breaks our Amazon EC2 slaves. Here's the initial ExceptionInInitializerError found in remoting.log on one of our slaves: Mar 06, 2020 11:52:33 AM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@75f3b9a6 java.lang.ExceptionInInitializerError at hudson.util.ProcessTree.get(ProcessTree.java:414) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299) at com.sun.proxy.$Proxy5.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at hudson.model.Computer.<clinit>(Computer.java:1362) ... 11 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:177) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) ... 16 more From there one can only see NoClassDefFoundError: Mar 06, 2020 11:56:46 AM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.Launcher$RemoteLauncher$KillTask@1c3a1ce1 java.lang.NoClassDefFoundError: Could not initialize class hudson.slaves.SlaveComputer at hudson.util.ProcessTree.get(ProcessTree.java:414) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) The issue occurs when a build finishes and ProcessTreeKiller is requested by Jenkins Master to clean up any left-over processes running on a given slave. Unfortunately due to a bug in JDK the issue is rather hard to track down, as the logs in our builds have always had this NoClassDefFoundError exception logged, never the one causing all the trouble - ExceptionInInitializerError. I'm really looking forward to get it fixed. Please let me know if I could help somehow.
            Hide
            tom_gl Thomas de Grenier de Latour added a comment -

            Thanks Adam for the pointer to JDK bug 8051847, interesting reading. I didn't know about https://github.com/kohsuke/eiie-logger, that's pretty clever!

            Show
            tom_gl Thomas de Grenier de Latour added a comment - Thanks Adam for the pointer to JDK bug 8051847, interesting reading. I didn't know about https://github.com/kohsuke/eiie-logger , that's pretty clever!

              People

              • Assignee:
                jthompson Jeff Thompson
                Reporter:
                tom_gl Thomas de Grenier de Latour
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: