Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54170

Kubernetes agent does not connect on first several attempts

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • kubernetes-plugin
    • None
    • Jenkins 2.138.2 running in Kubernetes using jenkins/jenkins:2.138.2 image
      Kubernetes plugin 1.13.0

      I'm seeing long wait times for kubernetes build slaves to run jobs. What appears to be happening is the Jenkins master is refusing connection, but eventually accepts, at which time the build slave registers and start running the job.

      Here's a representative example of logs from the JNLP container when this is happening.

      You will see several failed attempts to connect, then it accepts the connection. This is easily reproducible. Sometimes it connects relatively quickly, other times it takes 5 minutes or so before it finally accepts the connection. 

      Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Oct 20, 2018 4:57:41 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: jenkins-slave-w5dwq-sdx28
Oct 20, 2018 4:57:41 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Oct 20, 2018 4:57:41 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.23
Oct 20, 2018 4:57:41 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Oct 20, 2018 4:57:41 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://jenkins.mycompany.com/]
Oct 20, 2018 4:57:42 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Oct 20, 2018 4:57:42 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Oct 20, 2018 4:57:42 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
 Agent address: jenkins-jnlp-svc.mycompany.com
 Agent port: 50000
 Identity: 63:07:ea:38:e5:41:4c:24:db:e3:5f:d6:6a:2b:0c:e7
Oct 20, 2018 4:57:42 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Oct 20, 2018 4:57:42 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-jnlp-svc.mycompany.com:50000
Oct 20, 2018 4:57:52 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-jnlp-svc.mycompany.com:50000 (retrying:2)
java.io.IOException: Failed to connect to jenkins-jnlp-svc.mycompany.com:50000
 at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)
 at hudson.remoting.Engine.connect(Engine.java:691)
 at hudson.remoting.Engine.innerRun(Engine.java:552)
 at hudson.remoting.Engine.run(Engine.java:474)
Caused by: java.net.ConnectException: Connection refused
 at sun.nio.ch.Net.connect0(Native Method)
 at sun.nio.ch.Net.connect(Net.java:454)
 at sun.nio.ch.Net.connect(Net.java:446)
 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
 at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
 at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)
 ... 3 more

Oct 20, 2018 4:58:02 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-jnlp-svc.mycompany.com:50000 (retrying:3)
java.io.IOException: Failed to connect to jenkins-jnlp-svc.mycompany.com:50000
 at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)
 at hudson.remoting.Engine.connect(Engine.java:691)
 at hudson.remoting.Engine.innerRun(Engine.java:552)
 at hudson.remoting.Engine.run(Engine.java:474)
Caused by: java.net.ConnectException: Connection refused
 at sun.nio.ch.Net.connect0(Native Method)
 at sun.nio.ch.Net.connect(Net.java:454)
 at sun.nio.ch.Net.connect(Net.java:446)
 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
 at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
 at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)
 ... 3 more

Oct 20, 2018 4:58:12 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-jnlp-svc.mycompany.com:50000 (retrying:4)
java.io.IOException: Failed to connect to jenkins-jnlp-svc.mycompany.com:50000
 at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)
 at hudson.remoting.Engine.connect(Engine.java:691)
 at hudson.remoting.Engine.innerRun(Engine.java:552)
 at hudson.remoting.Engine.run(Engine.java:474)
Caused by: java.net.ConnectException: Connection refused
 at sun.nio.ch.Net.connect0(Native Method)
 at sun.nio.ch.Net.connect(Net.java:454)
 at sun.nio.ch.Net.connect(Net.java:446)
 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
 at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
 at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)
 ... 3 more

Oct 20, 2018 4:58:12 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Oct 20, 2018 4:58:12 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 63:07:ea:38:e5:41:4c:24:db:e3:5f:d6:6a:2b:0c:e7
Oct 20, 2018 4:58:13 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

      The master is configured only to accept JNLP 4.

      The Kubernetes plugin is configured to use a tunnel for JNLP, which in my case is jenkins-jnlp-svc.mycompany.com:50000

            csanchez Carlos Sanchez
            johnjeffers John Jeffers
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: