Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45910

Template namespace is ignored when deleting pod

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: kubernetes-plugin, remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.60.2
      Kubernetes plugin 0.12
      Kubernetes 1.7.2
    • Similar Issues:

      Description

      We are dynamically creating and destroying build slaves and they have now started to exit in a Error state.

      Although the build job completes successfully it seems that the slave tries to reconnect.

      Log from slave:

      Aug 01, 2017 1:37:46 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Terminated
      Aug 01, 2017 1:37:56 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
      INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@863f8a3
      Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main createEngine
      INFO: Setting up slave: jenkins-build-slave-qvdpn
      Aug 01, 2017 1:37:57 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [http://jenkins-service]
      Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
        Agent address: jenkins-service
        Agent port:    50000
        Identity:      51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP4-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Remote identity confirmed: 51:ac:fd:f7:69:94:53:62:f9:ac:43:6b:72:ff:e7:c2
      Aug 01, 2017 1:37:58 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
      INFO: [JNLP4-connect connection to jenkins-service/10.251.248.121:50000] Local headers refused by remote: Unknown client name: jenkins-build-slave-qvdpn
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP4-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
      	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-build-slave-qvdpn
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:375)
      	at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
      	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
      	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: java.nio.channels.ClosedChannelException
      		... 7 more
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Server reports protocol JNLP4-plaintext not supported, skipping
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP3-connect is not enabled, skipping
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP2-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP2-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at org.jenkinsci.remoting.engine.JnlpProtocol2Handler.sendHandshake(JnlpProtocol2Handler.java:134)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-service:50000
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP-connect
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Protocol JNLP-connect encountered an unexpected exception
      java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at hudson.remoting.Engine.innerRun(Engine.java:385)
      	at hudson.remoting.Engine.run(Engine.java:287)
      Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
      	at org.jenkinsci.remoting.engine.JnlpProtocol1Handler.sendHandshake(JnlpProtocol1Handler.java:121)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
      	at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at hudson.remoting.Engine$1$1.run(Engine.java:94)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 1:37:58 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: The server rejected the connection: None of the protocols were accepted
      java.lang.Exception: The server rejected the connection: None of the protocols were accepted
      	at hudson.remoting.Engine.onConnectionRejected(Engine.java:484)
      	at hudson.remoting.Engine.innerRun(Engine.java:448)
      	at hudson.remoting.Engine.run(Engine.java:287)
      

      Log from master:

      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Terminating Kubernetes instance for slave jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Terminated Kubernetes instance for slave jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      
      Disconnected computer jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:46 PM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
      
      Computer.threadPoolForRemoting [#324] for jenkins-build-slave-qvdpn terminated
      java.nio.channels.ClosedChannelException
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
      	at hudson.remoting.Channel.close(Channel.java:1295)
      	at hudson.remoting.Channel.close(Channel.java:1263)
      	at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
      	at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
      	at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      Aug 01, 2017 3:37:46 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      
      api-gateway #269 completed: SUCCESS
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP4-connect connection #145 from 10.248.2.34/10.248.2.34:34644
      
      Aug 01, 2017 3:37:58 PM INFO org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
      
      [JNLP4-connect connection from 10.248.2.34/10.248.2.34:34644] Refusing headers from remote: Unknown client name: jenkins-build-slave-qvdpn
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP2-connect connection #146 from /10.248.2.34:34646
      
      Aug 01, 2017 3:37:58 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
      
      Accepted JNLP-connect connection #147 from /10.248.2.34:34648
      

      From kubectl describe

      State:		Terminated
            Reason:		Error
            Exit Code:	255
      

        Attachments

        1. first-slave-describe
          4 kB
        2. first-slave-log
          7 kB
        3. kubernetes-plugin.log
          17 kB
        4. kubernetes-plugin.log
          7 kB
        5. okhttp3-log-recorder.png
          okhttp3-log-recorder.png
          25 kB
        6. second-slave-describe
          4 kB
        7. second-slave-log
          8 kB

          Issue Links

            Activity

            Hide
            csanchez Carlos Sanchez added a comment -

            do you have a namespace in the cloud definition or in the template ?

            have you changed the cloud definition namespace ? it would require a restart until https://github.com/jenkinsci/kubernetes-plugin/pull/189 is released

            Show
            csanchez Carlos Sanchez added a comment - do you have a namespace in the cloud definition or in the template ? have you changed the cloud definition namespace ? it would require a restart until https://github.com/jenkinsci/kubernetes-plugin/pull/189 is released
            Hide
            juhtie01 Juha Tiensyrjä added a comment -

            I have the 'default' namespace in cloud definition. Tried adding the namespace to also the 'jnlp' pod, and it looks like now it's deleted from Kubernetes after running the same job which previously left it behind.

            Show
            juhtie01 Juha Tiensyrjä added a comment - I have the 'default' namespace in cloud definition. Tried adding the namespace to also the 'jnlp' pod, and it looks like now it's deleted from Kubernetes after running the same job which previously left it behind.
            Hide
            zmeggyesi Zalan Meggyesi added a comment -

            I have the specific namespace (not default) in the cloud definition. I can confirm that replicating the namespace on the pod templates will allow the plugin to remove the slave in question, though it won't remove any that are already in the Error state.

            Show
            zmeggyesi Zalan Meggyesi added a comment - I have the specific namespace (not default ) in the cloud definition. I can confirm that replicating the namespace on the pod templates will allow the plugin to remove the slave in question, though it won't remove any that are already in the  Error state.
            Hide
            vikgun Viktor Gunnarson added a comment -

            I have namespace 'default' in the cloud definition but no namespace in the pod template. I will test if it works by adding the namespace to the pod template.

            Show
            vikgun Viktor Gunnarson added a comment - I have namespace 'default' in the cloud definition but no namespace in the pod template. I will test if it works by adding the namespace to the pod template.
            Hide
            vikgun Viktor Gunnarson added a comment -

            Yes, adding the namespace to the pod template as well seems to do the trick. Then the DELETE request uses the namespace:

            Aug 10, 2017 4:19:16 PM INFO okhttp3.internal.platform.Platform log
            --> DELETE https://kubernetes.default/api/v1/namespaces/default/pods/jenkins-build-slave-pw0h7 http/1.1
            
            Show
            vikgun Viktor Gunnarson added a comment - Yes, adding the namespace to the pod template as well seems to do the trick. Then the DELETE request uses the namespace: Aug 10, 2017 4:19:16 PM INFO okhttp3.internal.platform.Platform log --> DELETE https: //kubernetes. default /api/v1/namespaces/ default /pods/jenkins-build-slave-pw0h7 http/1.1

              People

              • Assignee:
                csanchez Carlos Sanchez
                Reporter:
                vikgun Viktor Gunnarson
              • Votes:
                10 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: