-
Bug
-
Resolution: Fixed
-
Major
-
- Kubernetes (container) using AKS, running Jenkins through a custom Helm chart.
- Azure Load Balancer listening on TCP 50008 and forwarding the traffic to the Jenkins controller container (pod) port 50008 which we use for JNLP.
---
Jenkins: 2.387.1
OS: Linux - 5.4.0-1103-azure
Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.18.0
ace-editor:1.1
additional-metrics:67.vcb_823d3999a_2
analysis-core:1.96
analysis-model-api:11.2.0
ansible:174.vfd5323d2b_9d8
ant:487.vd79d090d4ea_e
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
artifactory:3.18.1
authentication-tokens:1.53.v1c90fd9191a_b_
azure-artifact-manager:115.vb0d5da76bb49
azure-commons:1.1.3
azure-container-agents:253.vd2f5cd5c5040
azure-credentials:254.v64da_8176c83a
azure-credentials-ext:1.0
azure-keyvault:192.v8723b_823cd7a_
azure-sdk:132.v62b_48eb_6f32f
azure-vm-agents:851.v16b_dcb_e85c85
azure-vmss:0.2.4
blackduck-detect:8.0.0
blueocean:1.27.3
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.3
blueocean-commons:1.27.3
blueocean-config:1.27.3
blueocean-core-js:1.27.3
blueocean-dashboard:1.27.3
blueocean-display-url:2.4.2
blueocean-events:1.27.3
blueocean-git-pipeline:1.27.3
blueocean-github-pipeline:1.27.3
blueocean-i18n:1.27.3
blueocean-jira:1.27.3
blueocean-jwt:1.27.3
blueocean-personalization:1.27.3
blueocean-pipeline-api-impl:1.27.3
blueocean-pipeline-editor:1.27.3
blueocean-pipeline-scm-api:1.27.3
blueocean-rest:1.27.3
blueocean-rest-impl:1.27.3
blueocean-web:1.27.3
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.2-2
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-environment:1.7
build-history-metrics-plugin:97.vdcf3aa48af8d
build-pipeline-plugin:1.5.8
buildresult-trigger:0.18
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloud-stats:267.v577e3742c282
cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1
cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8
cloudbees-folder:6.815.v0dd5a_cb_40e0e
cobertura:1.17
code-coverage-api:4.4.0
command-launcher:100.v2f6722292ee8
commons-httpclient3-api:3.1-3
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
conditional-buildstep:1.4.2
config-file-provider:3.11.1
configuration-as-code:1625.v27444588cc3d
configurationslicing:430.v966357576543
confluence-publisher:156.vf3597ca_9cf27
copyartifact:698.v393f578eb_ddc
cppcheck:1.26
credentials:1236.v31e44e6060c0
credentials-binding:604.vb_64480b_c56ca_
cygpath:1.5
dashboard-view:2.472.v9ff2a_e6a_c529
data-tables-api:1.13.3-3
disk-usage:0.28
display-url-api:2.3.7
docker-build-publish:1.4.0
docker-build-step:2.9
docker-commons:419.v8e3cd84ef49c
docker-custom-build-environment:1.7.3
docker-java-api:3.2.13-68.va_875df25a_b_45
docker-plugin:1.3.0
docker-slaves:1.0.7
docker-traceability:1.2
docker-workflow:563.vd5d2e5c4007f
dtkit-api:3.0.2
durable-task:506.v1b_3e14b_6f5da_
echarts-api:5.4.0-3
email-ext:2.96
email-ext-recipients-column:27.vb_9404db_b_018d
emailext-template:1.5
external-monitor-job:203.v683c09d993b_9
favorite:2.4.1
font-awesome-api:6.3.0-2
forensics-api:2.1.0
generic-webhook-trigger:1.86.3
gerrit-code-review:0.4.7
gerrit-trigger:2.39.0
gerrit-verify-status-reporter:0.0.3
git:5.0.1
git-client:4.2.0
git-forensics:2.0.0
git-parameter:0.9.18
git-server:99.va_0826a_b_cdfa_d
github:1.37.0
github-api:1.303-417.ve35d9dd78549
github-branch-source:1703.vd5a_2b_29c6cdc
gitlab-api:5.2.0-86.v1ed41a_9cf486
gitlab-branch-source:659.va_685a_51fda_db_
global-build-stats:269.v214f74360b_3a_
gradle:2.6
greenballs:1.15.1
groovy:453.vcdb_a_c5c99890
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
http_request:1.16
instance-identity:142.v04572ca_5b_265
ionicons-api:45.vf54fca_5d2154
ivy:2.4
jackson2-api:2.15.0-334.v317a_165f9b_7c
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:233.vdc1a_ec702cff
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:66.vd8fa_64ee91b_d
jenkins-design-language:1.27.3
jersey2-api:2.39.1-1
jira:3.9
jira-ext:0.9
jira-steps:2.0.165.v8846cf59f3db
jira-trigger:1.0.2
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.16-2
job-direct-mail:1.5
jobConfigHistory:1207.vd28a_54732f92
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.6.4-1
jsch:0.2.8-65.v052c39de79b_2
junit:1198.ve38db_d1b_c975
ldap:673.v034ec70ec2b_b_
leastload:3.0.0
locale:262.ved03281fa_64f
lockable-resources:1150.v59db_2b_994618
log-parser:2.3.0
mailer:448.v5b_97805e3767
mathworks-polyspace:1.0.5
matlab:2.10.0
matrix-auth:3.1.7
matrix-project:789.v57a_725b_63c79
maven-plugin:3.22
mercurial:1260.vdfb_723cdcc81
metrics:4.2.13-420.vea_2f17932dd6
mina-sshd-api-common:2.9.2-62.v199162f0a_2f8
mina-sshd-api-core:2.9.2-62.v199162f0a_2f8
mission-control-view:0.9.16
momentjs:1.1.1
monitoring:1.94.0
msbuild:1.30
msginject:0.1.1
nodelabelparameter:1.11.0
nunit:380.va_067eddf00da_
okhttp-api:4.10.0-132.v7a_7b_91cef39c
pam-auth:1.10
parameterized-trigger:2.45
percentage-du-node-column:0.1.0
pipeline-aggregator-view:1.15
pipeline-build-step:491.v1fec530da_858
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:468.va_5db_051498a_4
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2131.vb_9788088fdb_5
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2131.vb_9788088fdb_5
pipeline-model-extensions:2.2131.vb_9788088fdb_5
pipeline-rest-api:2.32
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2131.vb_9788088fdb_5
pipeline-stage-view:2.32
pipeline-utility-steps:2.15.2
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.2.0
poll-mailbox-trigger-plugin:1.026
popper-api:1.16.1-3
popper2-api:2.11.6-2
powershell:2.0
prism-api:1.29.0-4
project-build-times:1.2.1
project-stats-plugin:23.v47fee1f77b_84
prometheus:2.2.2
publish-over:0.22
publish-over-cifs:0.16
pubsub-light:1.17
purge-build-queue-plugin:88.v23b_97b_f2c7a_d
purge-job-history:1.6
pyenv-pipeline:2.1.2
quality-gates:2.5
rebuild:320.v5a_0933a_e7d61
release:2.15
release-helper:1.3.3
repo:1.16.0
resource-disposer:0.22
robot:3.3.0
run-condition:1.5
saferestart:0.7
saml:4.403.v423b_3195a_9ec
scm-api:667.v8b_6e07cdc7f2
scoring-load-balancer:59.vf791549fa_989
script-security:1244.ve463715a_f89c
shiningpanda:0.24
simple-theme-plugin:160.vb_76454b_67900
skip-certificate-check:1.1
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sonar:2.15
sonar-gerrit:377.v8f3808963dc5
sonar-quality-gates:1.3.1
sse-gateway:1.26
ssh:2.6.1
ssh-agent:333.v878b_53c89511
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.275.v9e17c10f2571
startup-trigger-plugin:2.9.3
structs:324.va_f5d6774f3a_d
test-results-analyzer:0.4.0
testweaver:1.0.8
timestamper:1.25
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
uno-choice:2.6.5
validating-string-parameter:2.8
variant:59.vf075fe829ccb
warnings:5.0.2
warnings-ng:10.1.0
windows-azure-storage:386.v673495b0a5de
windows-exe-runner:35.vea_7d292d39a_2
windows-slaves:1.8.1
winrm-client:1.0
workflow-aggregator:596.v8c21c963d92d
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:1017.vb_45b_302f0cea_
workflow-cps:3659.v582dc37621d8
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1246.v5524618ea_097
workflow-job:1292.v27d8cc3e2602
workflow-multibranch:746.v05814d19c001
workflow-scm-step:408.v7d5b_135a_b_d49
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.45
xunit:3.1.2- Kubernetes (container) using AKS, running Jenkins through a custom Helm chart. - Azure Load Balancer listening on TCP 50008 and forwarding the traffic to the Jenkins controller container (pod) port 50008 which we use for JNLP. --- Jenkins: 2.387.1 OS: Linux - 5.4.0-1103-azure Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.18.0 ace-editor:1.1 additional-metrics:67.vcb_823d3999a_2 analysis-core:1.96 analysis-model-api:11.2.0 ansible:174.vfd5323d2b_9d8 ant:487.vd79d090d4ea_e antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 artifactory:3.18.1 authentication-tokens:1.53.v1c90fd9191a_b_ azure-artifact-manager:115.vb0d5da76bb49 azure-commons:1.1.3 azure-container-agents:253.vd2f5cd5c5040 azure-credentials:254.v64da_8176c83a azure-credentials-ext:1.0 azure-keyvault:192.v8723b_823cd7a_ azure-sdk:132.v62b_48eb_6f32f azure-vm-agents:851.v16b_dcb_e85c85 azure-vmss:0.2.4 blackduck-detect:8.0.0 blueocean:1.27.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.3 blueocean-commons:1.27.3 blueocean-config:1.27.3 blueocean-core-js:1.27.3 blueocean-dashboard:1.27.3 blueocean-display-url:2.4.2 blueocean-events:1.27.3 blueocean-git-pipeline:1.27.3 blueocean-github-pipeline:1.27.3 blueocean-i18n:1.27.3 blueocean-jira:1.27.3 blueocean-jwt:1.27.3 blueocean-personalization:1.27.3 blueocean-pipeline-api-impl:1.27.3 blueocean-pipeline-editor:1.27.3 blueocean-pipeline-scm-api:1.27.3 blueocean-rest:1.27.3 blueocean-rest-impl:1.27.3 blueocean-web:1.27.3 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.2-2 bouncycastle-api:2.27 branch-api:2.1071.v1a_188a_562481 build-environment:1.7 build-history-metrics-plugin:97.vdcf3aa48af8d build-pipeline-plugin:1.5.8 buildresult-trigger:0.18 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloud-stats:267.v577e3742c282 cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1 cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.815.v0dd5a_cb_40e0e cobertura:1.17 code-coverage-api:4.4.0 command-launcher:100.v2f6722292ee8 commons-httpclient3-api:3.1-3 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1625.v27444588cc3d configurationslicing:430.v966357576543 confluence-publisher:156.vf3597ca_9cf27 copyartifact:698.v393f578eb_ddc cppcheck:1.26 credentials:1236.v31e44e6060c0 credentials-binding:604.vb_64480b_c56ca_ cygpath:1.5 dashboard-view:2.472.v9ff2a_e6a_c529 data-tables-api:1.13.3-3 disk-usage:0.28 display-url-api:2.3.7 docker-build-publish:1.4.0 docker-build-step:2.9 docker-commons:419.v8e3cd84ef49c docker-custom-build-environment:1.7.3 docker-java-api:3.2.13-68.va_875df25a_b_45 docker-plugin:1.3.0 docker-slaves:1.0.7 docker-traceability:1.2 docker-workflow:563.vd5d2e5c4007f dtkit-api:3.0.2 durable-task:506.v1b_3e14b_6f5da_ echarts-api:5.4.0-3 email-ext:2.96 email-ext-recipients-column:27.vb_9404db_b_018d emailext-template:1.5 external-monitor-job:203.v683c09d993b_9 favorite:2.4.1 font-awesome-api:6.3.0-2 forensics-api:2.1.0 generic-webhook-trigger:1.86.3 gerrit-code-review:0.4.7 gerrit-trigger:2.39.0 gerrit-verify-status-reporter:0.0.3 git:5.0.1 git-client:4.2.0 git-forensics:2.0.0 git-parameter:0.9.18 git-server:99.va_0826a_b_cdfa_d github:1.37.0 github-api:1.303-417.ve35d9dd78549 github-branch-source:1703.vd5a_2b_29c6cdc gitlab-api:5.2.0-86.v1ed41a_9cf486 gitlab-branch-source:659.va_685a_51fda_db_ global-build-stats:269.v214f74360b_3a_ gradle:2.6 greenballs:1.15.1 groovy:453.vcdb_a_c5c99890 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 http_request:1.16 instance-identity:142.v04572ca_5b_265 ionicons-api:45.vf54fca_5d2154 ivy:2.4 jackson2-api:2.15.0-334.v317a_165f9b_7c jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:233.vdc1a_ec702cff javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:66.vd8fa_64ee91b_d jenkins-design-language:1.27.3 jersey2-api:2.39.1-1 jira:3.9 jira-ext:0.9 jira-steps:2.0.165.v8846cf59f3db jira-trigger:1.0.2 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.16-2 job-direct-mail:1.5 jobConfigHistory:1207.vd28a_54732f92 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.4-1 jsch:0.2.8-65.v052c39de79b_2 junit:1198.ve38db_d1b_c975 ldap:673.v034ec70ec2b_b_ leastload:3.0.0 locale:262.ved03281fa_64f lockable-resources:1150.v59db_2b_994618 log-parser:2.3.0 mailer:448.v5b_97805e3767 mathworks-polyspace:1.0.5 matlab:2.10.0 matrix-auth:3.1.7 matrix-project:789.v57a_725b_63c79 maven-plugin:3.22 mercurial:1260.vdfb_723cdcc81 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.9.2-62.v199162f0a_2f8 mina-sshd-api-core:2.9.2-62.v199162f0a_2f8 mission-control-view:0.9.16 momentjs:1.1.1 monitoring:1.94.0 msbuild:1.30 msginject:0.1.1 nodelabelparameter:1.11.0 nunit:380.va_067eddf00da_ okhttp-api:4.10.0-132.v7a_7b_91cef39c pam-auth:1.10 parameterized-trigger:2.45 percentage-du-node-column:0.1.0 pipeline-aggregator-view:1.15 pipeline-build-step:491.v1fec530da_858 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7 pipeline-input-step:468.va_5db_051498a_4 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2131.vb_9788088fdb_5 pipeline-model-declarative-agent:1.1.1 pipeline-model-definition:2.2131.vb_9788088fdb_5 pipeline-model-extensions:2.2131.vb_9788088fdb_5 pipeline-rest-api:2.32 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2131.vb_9788088fdb_5 pipeline-stage-view:2.32 pipeline-utility-steps:2.15.2 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.2.0 poll-mailbox-trigger-plugin:1.026 popper-api:1.16.1-3 popper2-api:2.11.6-2 powershell:2.0 prism-api:1.29.0-4 project-build-times:1.2.1 project-stats-plugin:23.v47fee1f77b_84 prometheus:2.2.2 publish-over:0.22 publish-over-cifs:0.16 pubsub-light:1.17 purge-build-queue-plugin:88.v23b_97b_f2c7a_d purge-job-history:1.6 pyenv-pipeline:2.1.2 quality-gates:2.5 rebuild:320.v5a_0933a_e7d61 release:2.15 release-helper:1.3.3 repo:1.16.0 resource-disposer:0.22 robot:3.3.0 run-condition:1.5 saferestart:0.7 saml:4.403.v423b_3195a_9ec scm-api:667.v8b_6e07cdc7f2 scoring-load-balancer:59.vf791549fa_989 script-security:1244.ve463715a_f89c shiningpanda:0.24 simple-theme-plugin:160.vb_76454b_67900 skip-certificate-check:1.1 snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sonar:2.15 sonar-gerrit:377.v8f3808963dc5 sonar-quality-gates:1.3.1 sse-gateway:1.26 ssh:2.6.1 ssh-agent:333.v878b_53c89511 ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.275.v9e17c10f2571 startup-trigger-plugin:2.9.3 structs:324.va_f5d6774f3a_d test-results-analyzer:0.4.0 testweaver:1.0.8 timestamper:1.25 token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 uno-choice:2.6.5 validating-string-parameter:2.8 variant:59.vf075fe829ccb warnings:5.0.2 warnings-ng:10.1.0 windows-azure-storage:386.v673495b0a5de windows-exe-runner:35.vea_7d292d39a_2 windows-slaves:1.8.1 winrm-client:1.0 workflow-aggregator:596.v8c21c963d92d workflow-api:1208.v0cc7c6e0da_9e workflow-basic-steps:1017.vb_45b_302f0cea_ workflow-cps:3659.v582dc37621d8 workflow-cps-global-lib:609.vd95673f149b_b workflow-durable-task-step:1246.v5524618ea_097 workflow-job:1292.v27d8cc3e2602 workflow-multibranch:746.v05814d19c001 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.45 xunit:3.1.2
Our team provides Jenkins as a managed service to internal teams. We have previously been running the following setup:
- Single VM per controller, with the controller in a container
- HTTPS reverse proxy
- JNLP connecting directly to the controller VM using "tunnel connection through" as the HTTPS reverse proxy didn't serve JNLP connections
We recently switched to running all of our Jenkins controller on Kubernetes, and we share a common Azure load balancer for them all. This load balancer listens for HTTP connections and performs host-based-routing, and also listens on a unique JNLP port for each controller instance.
We've had CI teams report issues concerning node disconnections. We looked into the issue and noticed that it happened on nodes that run long-running processes where nothing is printed to stdout for long periods of time. We were able to solve it for many users by increasing the TCP idle connection timeout setting on the load balancer from 4 to 30 minutes, but we still have builds that run longer than this without any output. Now the issue is pointing to a faulty (or lack of) TCP keepalive functionality in the agent.
I would expect the agent to send TCP keepalive packets to the controller every n seconds (configurable) in order to assure the load balancer that the connection is active.
The errors we see in the controller console log look like this:
03:19:33 Cannot contact NODE_NAME: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@3095990c:JNLP4-connect connection from NODE_IP:54165": Remote call on JNLP4-connect connection from NODE_IP:54165 failed. The channel is closing down or has closed down