Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46386

Agent sometimes fails to launch in custom cloud provider

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      (previously posted about this on mailing list https://groups.google.com/forum/#!topic/jenkinsci-dev/Zte6nExHM8w was encouraged to file a ticket also)

      I'm developing a cloud provider plugin and have run into an intermittent issue with launching agents. In the computer launch log I get this series of errors:

       

      openjdk full version "1.8.0_141-b16"
       Copying slave.jar
       Launching slave agent: java -jar /tmp/slave.jar -workDir .
       Both error and output logs will be printed to ./remoting
       <===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 3.10
       This is a Unix agent
       Evacuated stdout
       java.io.IOException: Remote call on testnode failed
       at hudson.remoting.Channel.call(Channel.java:847)
       at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:32)
       at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:582)
       at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:390)
       at sfoster.jenkins.plugins.onecloud.OneComputerLauncher.launch(OneComputerLauncher.java:141)
       at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:262)
       at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Caused by: java.lang.LinkageError: Failed to load jenkins.util.SystemProperties
       at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:385)
       at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:293)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
       at hudson.FilePath.<clinit>(FilePath.java:2363)
       at sun.misc.Unsafe.ensureClassInitialized(Native Method)
       at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
       at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
       at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
       at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
       at java.lang.reflect.Field.getLong(Field.java:611)
       at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1707)
       at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484)
       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
       at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
       at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)
       at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)
       at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
       at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
       at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
       at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
       at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
       at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
       at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
       at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
       at hudson.remoting.UserRequest.deserialize(UserRequest.java:245)
       at hudson.remoting.UserRequest.perform(UserRequest.java:159)
       at hudson.remoting.UserRequest.perform(UserRequest.java:52)
       at hudson.remoting.Request$2.run(Request.java:336)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       at ......remote call to testnode(Native Method)
       at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1554)
       at hudson.remoting.UserResponse.retrieve(UserRequest.java:281)
       at hudson.remoting.Channel.call(Channel.java:839)
       ... 10 more
       Caused by: java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
       at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:381)
       at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:293)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
       at hudson.FilePath.<clinit>(FilePath.java:2363)
       at sun.misc.Unsafe.ensureClassInitialized(Native Method)
       at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
       at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
       at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
       at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
       at java.lang.reflect.Field.getLong(Field.java:611)
       at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1707)
       at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484)
       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
       at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
       at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)
       at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)
       at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
       at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
       at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
       at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
       at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
       at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
       at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
       at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
       at hudson.remoting.UserRequest.deserialize(UserRequest.java:245)
       at hudson.remoting.UserRequest.perform(UserRequest.java:159)
       at hudson.remoting.UserRequest.perform(UserRequest.java:52)
       at hudson.remoting.Request$2.run(Request.java:336)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
       ... 4 more
       Caused by: java.lang.ClassNotFoundException: javax.servlet.ServletContextListener
       at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch(RemoteClassLoader.java:788)
       at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:896)
       at hudson.remoting.Request$2.run(Request.java:336)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
       at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:19)
       at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
       at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       at ......remote call to channel(Native Method)
       at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1554)
       at hudson.remoting.Request.call(Request.java:172)
       at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260)
       at com.sun.proxy.$Proxy5.fetch(Unknown Source)
       at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
       ... 40 more
       ERROR: Connection terminated
       java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
       at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
       at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
       at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
       Caused: java.io.IOException: Unexpected termination of the channel
       at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)

       This doesn't happen every time. If the failure does happen, it seems to work if I manually relaunch the agent through the UI. In the remoting/logs on the agent itself, all it has is the ERROR: Connection terminated stuff towards the bottom

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Filed alternate hotfix as PR 3362.

            Show
            jglick Jesse Glick added a comment - Filed alternate hotfix as PR 3362.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/util/SystemProperties.java
            test/src/test/java/jenkins/util/SystemPropertiesTest.java
            war/src/main/webapp/WEB-INF/web.xml
            http://jenkins-ci.org/commit/jenkins/7948315142cd8ac82ee69ab42416ea8af574860b
            Log:
            JENKINS-46386 Make SystemProperties safe to call from agent JVMs.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/util/SystemProperties.java test/src/test/java/jenkins/util/SystemPropertiesTest.java war/src/main/webapp/WEB-INF/web.xml http://jenkins-ci.org/commit/jenkins/7948315142cd8ac82ee69ab42416ea8af574860b Log: JENKINS-46386 Make SystemProperties safe to call from agent JVMs.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/jenkins/util/SystemProperties.java
            test/src/test/java/jenkins/util/SystemPropertiesTest.java
            war/src/main/webapp/WEB-INF/web.xml
            http://jenkins-ci.org/commit/jenkins/63a58031624f196b18d93aef00c76a10543ec95f
            Log:
            Merge pull request #3362 from jglick/SystemProperties-JENKINS-46386

            JENKINS-46386 Make SystemProperties safe to call from agent JVMs

            Compare: https://github.com/jenkinsci/jenkins/compare/9e667a2f4fcd...63a58031624f

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/jenkins/util/SystemProperties.java test/src/test/java/jenkins/util/SystemPropertiesTest.java war/src/main/webapp/WEB-INF/web.xml http://jenkins-ci.org/commit/jenkins/63a58031624f196b18d93aef00c76a10543ec95f Log: Merge pull request #3362 from jglick/SystemProperties- JENKINS-46386 JENKINS-46386 Make SystemProperties safe to call from agent JVMs Compare: https://github.com/jenkinsci/jenkins/compare/9e667a2f4fcd...63a58031624f
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The patch from Jesse Glick has been released in 2.113.
            I am closing this issue, because now the general use-case is addressed

            Show
            oleg_nenashev Oleg Nenashev added a comment - The patch from Jesse Glick has been released in 2.113. I am closing this issue, because now the general use-case is addressed
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/util/SystemProperties.java
            test/src/test/java/jenkins/util/SystemPropertiesTest.java
            war/src/main/webapp/WEB-INF/web.xml
            http://jenkins-ci.org/commit/jenkins/615c3b1f616b9ba4aa5dd2c348b61b3546edf882
            Log:
            JENKINS-46386 Make SystemProperties safe to call from agent JVMs.

            (cherry picked from commit 7948315142cd8ac82ee69ab42416ea8af574860b)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/util/SystemProperties.java test/src/test/java/jenkins/util/SystemPropertiesTest.java war/src/main/webapp/WEB-INF/web.xml http://jenkins-ci.org/commit/jenkins/615c3b1f616b9ba4aa5dd2c348b61b3546edf882 Log: JENKINS-46386 Make SystemProperties safe to call from agent JVMs. (cherry picked from commit 7948315142cd8ac82ee69ab42416ea8af574860b)

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                stevenfoster Steven Foster
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: