Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50730

NoClassDefFound errors in Cloud Slaves

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • kubernetes-plugin, remoting
    • None
    • Jenkins v2.89.2
      Kubernetes Plugin v1.3.3
    • Remoting 3.28

      While provisioning slaves from a private Kubernetes instance, we've found that a lot of slaves terminate with the following (or similar) stack trace on the slave's side:

       

      INFO: Setting up slave: kube1-medium-r9zf4
Apr 10, 2018 11:02:05 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Apr 10, 2018 11:02:05 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/<user>/workDir/remoting as a remoting work directory
Apr 10, 2018 11:02:06 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server ...
Apr 10, 2018 11:02:06 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, CLI2-connect, JNLP-connect, Ping, CLI-connect, JNLP2-connect]
Apr 10, 2018 11:02:06 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful <...>
pr 10, 2018 11:02:06 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Apr 10, 2018 11:02:06 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <Jenkins Master>
Apr 10, 2018 11:02:06 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Apr 10, 2018 11:02:07 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: <...>
Apr 10, 2018 11:02:07 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Apr 10, 2018 11:02:14 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Apr 10, 2018 11:02:14 AM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2@3e708317
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller$2$1
        at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2.call(JnlpSlaveRestarterInstaller.java:71)
        at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2.call(JnlpSlaveRestarterInstaller.java:53)
        at hudson.remoting.UserRequest.perform(UserRequest.java:207)
        at hudson.remoting.UserRequest.perform(UserRequest.java:53)
        at hudson.remoting.Request$2.run(Request.java:358)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at hudson.remoting.Engine$1$1.run(Engine.java:98)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1
        at java.net.URLClassLoader.findClass(Unknown Source)
        at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:159)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        ... 11 more

      The class that appears to not have been found isn't consistently the same. I've seen `FilePathFilter`, `LaunchConfiguration`, `StringBuilderWriter`, and some others being reported as well. Sometimes, there's also exceptions related to `JarCacheSupport` not being able to resolve jars (I don't have the exact stacktrace at hand - will post it if I find it again).

      On the master's side, these exceptions generally manifest as `ChannelClosedException`s, or weird Exception-less failures in pipeline branches.

      ERROR: Issue with creating launcher for agent kube1-medium-r9zf4. The agent has not been fully initialized yet

ERROR: Issue with creating launcher for agent kube1-medium-r9zf4. The agent has not been fully initialized yet

remote file operation failed: /home/<user>/workspace/<job_name> at hudson.remoting.Channel@6639429c:JNLP4-connect connection from <some-host-name>/<ip-address>:60326: hudson.remoting.ChannelClosedException: Remote call on JNLP4-connect connection from <some-host-name>/<ip-address>:60326 failed. The channel is closing down or has closed down

      I haven't been able to consistently reproduce the error, but it does manifest enough to be causing major pain to users (especially since we extensively use pipelines with a large number of parallel nodes, and a failure in any one of the nodes causes the entire pipeline to fail).

            jthompson Jeff Thompson
            karthikduddu Karthik Duddu
            Votes:
            3 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: