Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: maven-plugin, remoting
Labels:
None
Environment:
Jenkins 2.204.2 (remoting 3.36.1)
Linux OpenJDK 1.8.0_242

Similar Issues:

Show
URL:
https://github.com/jenkinsci/remoting/pull/379

I know this sound like a duplicate of some previously reported bugs, but I'm opening this new issue to have a single one I can refer to in a PR I'm about to open.

I'm trying to improve resilience of the agent's RemoteClassLoader when it receives a RemotingSystemException caused by an InterruptedException.

The overall issue is that the class loader can't really recover after an exception has been raised while some classes were being loaded / initialized; later attempts at using a class that failed being loaded/initialized will raise a LinkageError.
This is not something new, see ~~JENKINS-36991~~ / PR #94. This one got fixed by adding some try/catch/retry logic to the affected part of the RemoteClassLoader#findClass code. This way, the interruption can be deferred until after the class has been successfully loaded.

But there are other parts of the RemoteClassLoader code which are not protected yet, despite being on the critical path of classes loading/initialization.

See for instance JENKINS-51854: the exception occurs in a call to RemoteClassLoader#getResource made during initialization of some Subversion plugin classes, and from there the plugin can't be used on this agent anymore.

I've reproduced something similar with the Maven plugin, simply by interrupting (from the GUI) a Maven job while it was starting on a freshly started agent.

error (slave log) when this first Maven job execution gets interrupted:

Feb 10, 2020 5:51:39 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@38569db8
java.lang.ExceptionInInitializerError
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
	at com.sun.proxy.$Proxy5.getResource2(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426)
	at java.lang.ClassLoader.getResource(ClassLoader.java:1090)
	at java.lang.Class.getResource(Class.java:2267)
	at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53)
	at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45)
	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107)
	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:128)
	at org.jvnet.localizer.Localizable.toString(Localizable.java:67)
	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
	at hudson.model.Run.<clinit>(Run.java:2535)
	... 11 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	... 22 more

see RemoteClassLoader.java:426

error (slave log) when trying to run any other Maven job on the same agent after that:

Feb 10, 2020 5:53:38 PM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@5e66716c
java.lang.NoClassDefFoundError: Could not initialize class hudson.maven.MavenModuleSetBuild
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

And with the same manual procedure, I also sometimes get the fatal exception in an other part of RemoteClassLoader#findClass which is not protected covered by the retry logic either (at least, not for RemotingSystemException):

error (slave log) when this first Maven job gets interrupted:

2020-02-11T12:16:49.933+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@601f4e25
java.lang.ExceptionInInitializerError
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
	at com.sun.proxy.$Proxy5.fetch(Unknown Source)
	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:89)
	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:79)
	at org.eclipse.sisu.plexus.ClassRealmManager.<clinit>(ClassRealmManager.java:66)
	... 18 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method)
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	... 25 more

see RemoteClassLoader.java:315

error (slave log) when trying to run any other Maven job on the same agent after that:

2020-02-11T12:17:41.347+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@69183d46
java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.sisu.plexus.ClassRealmManager
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Both these cases could be fixed by adding more "try / catch / retry with deferred interruption" logic on the relevant parts of the RemoteClassLoader code,
but instead I propose to do it at a slightly lower level, on methods of the IClassLoader proxy (that's the com.sun.proxy.$Proxy5.getResource2(Unknown Source)
and com.sun.proxy.$Proxy5.fetch(Unknown Source) in the above stack traces).

I will submit a PR implementing that (with the retry code in a dynamic proxy). With this change, when I try to interrupt the first execution of a Maven job, there aren't any LinkageError reported anymore. The build is still eventually interrupted of course, but then I can re-execute the job without triggering any NoClassDefFoundError, as one would expect.

causes

JENKINS-56137 EC2-slave jar cache vs class initialization race.

Open

is related to

JENKINS-65621 Post-processing can be interrupted by the plugin when aborting running builds

Open

relates to

JENKINS-51854 LinkageError while performing UserRequest:hudson.scm.SubversionSCM

Open

JENKINS-36991 Unable to load class once the loading was interrupted

Resolved

Details

Description

Attachments

Issue Links

Activity

People

Dates