Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45910

Template namespace is ignored when deleting pod

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Component/s: kubernetes-plugin, remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.60.2
      Kubernetes plugin 0.12
      Kubernetes 1.7.2
    • Similar Issues:

      Description

      We are dynamically creating and destroying build slaves and they have now started to exit in a Error state.

      Although the build job completes successfully it seems that the slave tries to reconnect.

      Log from slave:

      Aug 01, 2017 1:37:46 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Terminated
      Aug 01, 2017 1:37:56 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
      INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@863f8a3
      Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main createEngine
      INFO: Setting up slave: jenkins-build-slave-qvdpn
      Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [http://jenkins-service]
      Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
        Agent address: jenkins-service
        Agent port:    50000
        Identity:      51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP4-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Remote identity confirmed: 51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
      Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
      INFO: [JNLP4-connect connection to jenkins-service/10.251.248.121:50000] Local headers refused by remote: Unknown client name: jenkins-build-slave-qvdpn
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP4-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
      	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:375)
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
      	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
      	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: java.nio.channels.ClosedChannelException
      		... 7 more
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Server reports protocol JNLP4-plaintext not supported, skipping
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP3-connect is not enabled, skipping
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP2-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP2-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at org.jenkinsci.remoting.engine.JnlpProtocol2Handler.sendHandshake(JnlpProtocol2Handler.java:134)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at org.jenkinsci.remoting.engine.JnlpProtocol1Handler.sendHandshake(JnlpProtocol1Handler.java:121)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: The server rejected the connection: None of the protocols were accepted
      java.lang.Exception: The server rejected the connection: None of the protocols were accepted
      	at hudson.remoting.Engine.onConnectionRejected(Engine.java:484)
      	at hudson.remoting.Engine.innerRun(Engine.java:448)
      	at hudson.remoting.Engine.run(Engine.java:287)
      

      Log from master:

      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Terminating Kubernetes instance for slave jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Disconnected computer jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
      
      Computer.threadPoolForRemoting [#324] for jenkins-build-slave-qvdpn terminated
      java.nio.channels.ClosedChannelException
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
      	at hudson.remoting.Channel.close(Channel.java:1295)
      	at hudson.remoting.Channel.close(Channel.java:1263)
      	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
      	at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
      	at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 3:37:46 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      
      api-gateway #269 completed: SUCCESS
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP4-connect connection #145 from 10.248.2.34/10.248.2.34:34644
      
      Aug 01, 2017 3:37:58 PM INFO org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
      
      [JNLP4-connect connection from 10.248.2.34/10.248.2.34:34644] Refusing headers from remote: Unknown client name: jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP2-connect connection #146 from /10.248.2.34:34646
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP-connect connection #147 from /10.248.2.34:34648
      

      From kubectl describe

      State:		Terminated
            Reason:		Error
            Exit Code:	255
      

        Attachments

        1. first-slave-describe
          4 kB
        2. first-slave-log
          7 kB
        3. kubernetes-plugin.log
          17 kB
        4. kubernetes-plugin.log
          7 kB
        5. okhttp3-log-recorder.png
          okhttp3-log-recorder.png
          25 kB
        6. second-slave-describe
          4 kB
        7. second-slave-log
          8 kB

          Issue Links

            Activity

              People

              • Assignee:
                csanchez Carlos Sanchez
                Reporter:
                vikgun Viktor Gunnarson
              • Votes:
                10 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: