-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Jenkins 2.401.1
Jenkins inbound agent 3131.vf2b_b_798b_ce99-3
Jenkins Kubernetes plugin 3971.v94b_4c914ca_75
Hello,
We upgraded our Jenkins environment from 2.375.1 and the agents created via the Kubernetes plugin suddenly became unable to connect back to the controller. We can see a lot of logs but nothing that clearly states what is going on.
The agent is connecting to the controller via WebSocket over a tunnel to the http port of the controller. It is able to connect and it's getting closed:
Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Performing onReconnect operation.
Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: onReconnect operation completed.
Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: WebSocket connection open
Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Jul 17, 2023 8:29:12 AM hudson.remoting.ProxyOutputStream$Chunk lambda$execute$0
WARNING: Failed to ack the stream
java.io.IOException: java.lang.IllegalStateException: The connection has been closed.
at hudson.remoting.Engine$1AgentEndpoint$Transport.write(Engine.java:661)
at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:335)
at hudson.remoting.Channel.send(Channel.java:765)
at hudson.remoting.ProxyOutputStream$Chunk.lambda$execute$0(ProxyOutputStream.java:275)
at hudson.remoting.PipeWriter$1.run(PipeWriter.java:159)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: The connection has been closed.
at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusSession.checkConnectionState(TyrusSession.java:545)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusSession.getAsyncRemote(TyrusSession.java:201)
at hudson.remoting.Engine$1AgentEndpoint$Transport.write(Engine.java:659)
... 13 more
Any idea what could be wrong ?
Logs from the controller:
Jul 17, 2023 1:01:01 PM WARNING hudson.util.ExceptionCatchingThreadFactory uncaughtExceptionThread Computer.threadPoolForRemoting [#365] terminated unexpectedly java.nio.channels.ClosedChannelException Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from localhost/127.0.0.1:42350 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784) at hudson.remoting.Request.call(Request.java:199) at hudson.remoting.Channel.call(Channel.java:999) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:714) at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:176) at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:337) at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:428) at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler.lambda$onChannel$0(JnlpProtocol4Handler.java:335) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51) Caused: hudson.remoting.RequestAbortedException at hudson.remoting.Request.abort(Request.java:346) at hudson.remoting.Channel.terminate(Channel.java:1080) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:177) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:279) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:244) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:196) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:700) at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:156) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:328) at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:335) at hudson.remoting.Channel.send(Channel.java:765) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:146) at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:112) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.JarLoaderImpl.writeJarTo(JarLoaderImpl.java:57) at jdk.internal.reflect.GeneratedMethodAccessor2287.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:924) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:902) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:377) at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList.lambda$applyDecorator$0(CallableDecoratorList.java:19) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Jul 17, 2023 1:01:01 PM WARNING hudson.remoting.Request$2 runFailed to send back a reply to the request UserRequest:UserRPCRequest:hudson.remoting.JarLoader.writeJarTo[long,long,java.io.OutputStream](3) java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:177) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:279) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:244) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:196) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:700) at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:156) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:328) at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:335) at hudson.remoting.Channel.send(Channel.java:765) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:146) at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:112) at hudson.remoting.Util.copy(Util.java:58) at hudson.remoting.JarLoaderImpl.writeJarTo(JarLoaderImpl.java:57) at jdk.internal.reflect.GeneratedMethodAccessor2287.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:924) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:902) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:377) Caused: hudson.remoting.ChannelClosedException: Channel "JNLP4-connect connection from localhost/127.0.0.1:42350": channel is already closed at hudson.remoting.Channel.send(Channel.java:761) at hudson.remoting.Request$2.run(Request.java:390) at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList.lambda$applyDecorator$0(CallableDecoratorList.java:19) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Jul 17, 2023 1:01:09 PM WARNING hudson.remoting.AbstractByteBufferCommandTransport processCommandFailed to construct Command in channel xxxx java.io.UTFDataFormatException at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3728) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3672) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3476) at java.base/java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1212) at java.base/java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:629) at java.base/java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:961) at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1992) at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447) at hudson.remoting.Command.readFromObjectStream(Command.java:155) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:233) at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:220) at jenkins.agents.WebSocketAgents$Session.binary(WebSocketAgents.java:148) at jenkins.websocket.WebSockets$1.onWebSocketBinary(WebSockets.java:103) at jenkins.websocket.Jetty10Provider$2.onWebSocketBinary(Jetty10Provider.java:154) at org.eclipse.jetty.websocket.core.internal.messages.ByteArrayMessageSink.accept(ByteArrayMessageSink.java:89) at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.acceptMessage(JettyWebSocketFrameHandler.java:334) at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onContinuationFrame(JettyWebSocketFrameHandler.java:349) at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onFrame(JettyWebSocketFrameHandler.java:246) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$IncomingAdaptor.lambda$onFrame$1(WebSocketCoreSession.java:671) at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1468) at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1487) at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$IncomingAdaptor.onFrame(WebSocketCoreSession.java:671) at org.eclipse.jetty.websocket.core.internal.ExtensionStack.onFrame(ExtensionStack.java:120) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.onFrame(WebSocketCoreSession.java:481) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onFrame(WebSocketConnection.java:262) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.fillAndParse(WebSocketConnection.java:455) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onFillable(WebSocketConnection.java:340) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at java.base/java.lang.Thread.run(Thread.java:829)
Jul 17, 2023 1:01:09 PM WARNING jenkins.agents.WebSocketAgents$Session errornull org.eclipse.jetty.websocket.core.exception.ProtocolException: Unknown opcode: 14 at org.eclipse.jetty.websocket.core.internal.Parser.newFrame(Parser.java:271) at org.eclipse.jetty.websocket.core.internal.Parser.parsePayload(Parser.java:352) at org.eclipse.jetty.websocket.core.internal.Parser.parse(Parser.java:213) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.fillAndParse(WebSocketConnection.java:448) Caused: org.eclipse.jetty.websocket.api.exceptions.ProtocolException: Unknown opcode: 14 at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.convertCause(JettyWebSocketFrameHandler.java:515) at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onError(JettyWebSocketFrameHandler.java:258) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$2(WebSocketCoreSession.java:284) at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1468) at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1487) at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.closeConnection(WebSocketCoreSession.java:284) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$sendFrame$7(WebSocketCoreSession.java:519) at org.eclipse.jetty.util.Callback$3.succeeded(Callback.java:155) at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.notifyCallbackSuccess(TransformingFlusher.java:197) at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:154) at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:232) at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:214) at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:522) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.close(WebSocketCoreSession.java:239) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.processConnectionError(WebSocketCoreSession.java:335) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.fillAndParse(WebSocketConnection.java:506) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onFillable(WebSocketConnection.java:340) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at java.base/java.lang.Thread.run(Thread.java:829)
