-
Bug
-
Resolution: Duplicate
-
Critical
-
None
hi,
first of all: i'm so sorry to open this (and another ticket). really appreciate your hard work and totally love it.
i hope the following is not a n00b-misconfiguration-error-and-a-again-duplicate-ticket of someone who did not read the manual.
but i'm out of ideas and this is blocking me (and around 400 developers) to go one step further and push our CICD to a more self managed platform.
here is the issue:
when running multiple concurrent builds of specific job (with declarative definition of the build pods jenkins seem to freak out and deny the connection from build slaves to the master:
(from the jnlp container):
client name: brandhub-container-builder-834z6-3bmbk at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223) at hudson.remoting.Engine.innerRun(Engine.java:614) at hudson.remoting.Engine.run(Engine.java:474) Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: brandhub-container-builder-834z6-3bmbk at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378) at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Thread.java:748) Suppressed: java.nio.channels.ClosedChannelException ... 7 more
when running only one build "it works".
the multiple concurrent builds may get finished after some time.
one need to be patient and on the lucky side.
i searched the internet (and this jira) already and for quite some time.
could not find help yet.
ACHTUNG: when putting some other pod templates inline/declarative i get the same error. but if i define them in the configuration ui part in jenkins the builds are more stable and i got more reliable and satisfying results. tried it also with this builds but that trick did not help.
...
here are some logs and content of related files, also some version:
- k8s: 1.11.x
- jenkins: jenkins/jenkins:2.84 (10Gi Memory, 2000m cpu)
- slave: jenkins/jnlp-slave:3.29-1 (3Gi and 200m cpu)
- plugin version - kubernetes:1.16.4
THE JENKINSFILE:
pipeline {
agent {
kubernetes {
label 'brandhub-container-builder'
yamlFile 'api/.build.yaml'
}
}
stages {
stage('build and push container') {
steps {
container(name: 'kaniko', shell: '/busybox/sh') {
withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
sh '''#!/busybox/sh
/kaniko/executor -f `pwd`/api/Dockerfile -c `pwd`/api --verbosity=warn \
--destination=reg.cicd.intra.xxxxxxx.xxxxxx.io/ow-apps-mbbrandhub/cicd-php-node-x:latest
'''
}
}
}
}
}
}
RELATED BUILD YAML:
apiVersion: v1 kind: Pod metadata: name: kaniko spec: containers: - name: kaniko image: gcr.io/kaniko-project/executor:debug imagePullPolicy: Always command: - /busybox/cat tty: true volumeMounts: - name: jenkins-docker-cfg mountPath: /kaniko/.docker volumes: - name: jenkins-docker-cfg secret: secretName: regcreds items: - key: config.json path: config.json
ALL THESE PODS ARE FAILING:
NAME READY STATUS RESTARTS AGE brandhub-container-builder-nlvvh-26hcb 1/2 Error 0 29s brandhub-container-builder-nlvvh-6wqbn 1/2 Error 0 1m brandhub-container-builder-nlvvh-703pq 1/2 Error 0 29s brandhub-container-builder-nlvvh-d7jxd 1/2 Error 0 1m brandhub-container-builder-nlvvh-gn1pp 1/2 Error 0 49s brandhub-container-builder-nlvvh-s383r 1/2 Error 0 1m brandhub-container-builder-nlvvh-tn9nk 1/2 Error 0 49s brandhub-container-builder-nlvvh-v5tck 1/2 Error 0 29s brandhub-container-builder-nlvvh-w9939 1/2 Error 0 49s efs-provisioner-75dcb94b57-s4smt 1/1 Running 0 14h jenkins-non-prod-6555477944-mfj7t 1/1 Running 0 15s nodejs-8-3fmm3 4/4 Running 0 44m nodejs-8-g2cpq 3/4 Running 0 26m pcf-19574 1/1 Running 0 11m pcf-1fxx7 1/1 Running 0 1d pcf-4q0n3 1/1 Running 0 11m pcf-8k17d 1/1 Running 0 4h pcf-gr87l 1/1 Running 0 52m pcf-xxtzw 1/1 Running 0 11m remove-faulty-pods-1562694000-dqfzk 0/1 Completed 0 1m
GET LOGS:
$ k logs -f brandhub-container-builder-nlvvh-g9m2h -c jnlp
JNLP POD LOGS:
Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior Jul 09, 2019 5:37:59 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up agent: brandhub-container-builder-nlvvh-g9m2h Jul 09, 2019 5:37:59 PM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Jul 09, 2019 5:38:00 PM hudson.remoting.Engine startEngine INFO: Using Remoting version: 3.29 Jul 09, 2019 5:38:00 PM hudson.remoting.Engine startEngine WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins-non-prod:8080/] Jul 09, 2019 5:38:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping, JNLP3-connect] Jul 09, 2019 5:38:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: jenkins-non-prod-agent Agent port: 50004 Identity: 40:0f:dc:fe:7d:f1:f8:51:94:83:cc:2e:3e:c7:f5:7f Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins-non-prod-agent:50004 Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: 40:0f:dc:fe:7d:f1:f8:51:94:83:cc:2e:3e:c7:f5:7f Jul 09, 2019 5:38:00 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv INFO: [JNLP4-connect connection to jenkins-non-prod-agent/100.65.158.111:50004] Local headers refused by remote: Unknown client name: brandhub-container-builder-nlvvh-g9m2h Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Protocol JNLP4-connect encountered an unexpected exception java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: brandhub-container-builder-nlvvh-g9m2h at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223) at hudson.remoting.Engine.innerRun(Engine.java:614) at hudson.remoting.Engine.run(Engine.java:474) Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: brandhub-container-builder-nlvvh-g9m2h at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378) at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Thread.java:748) Suppressed: java.nio.channels.ClosedChannelException ... 7 more Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins-non-prod-agent:50004 Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server reports protocol JNLP4-plaintext not supported, skipping Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Protocol JNLP3-connect is not enabled, skipping Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server reports protocol JNLP2-connect not supported, skipping Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server reports protocol JNLP-connect not supported, skipping Jul 09, 2019 5:38:00 PM hudson.remoting.jnlp.Main$CuiListener error SEVERE: The server rejected the connection: None of the protocols were accepted java.lang.Exception: The server rejected the connection: None of the protocols were accepted at hudson.remoting.Engine.onConnectionRejected(Engine.java:682) at hudson.remoting.Engine.innerRun(Engine.java:639) at hudson.remoting.Engine.run(Engine.java:474)
does anyone has an idea what's going on here?
or what could be tweaked to get it more stable?
any help is pretty much appreciated.
hope the provided information is enough and gives a picture.
if you guys need more infos for whatever (related) purpose - just let me know.
thanks in ahead
- duplicates
-
-
- Open
-