Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48761

Jenkins Slaves do not connect after update from version 2.95 to version 2.99 and jobs fail

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: core, remoting
    • Labels:
    • Environment:
      CentOS root node and 2 macOS agents connecting to it through JNLP4.
    • Similar Issues:

      Description

      Seeing something that sounds similar to https://issues.jenkins-ci.org/browse/JENKINS-48754.

      "After having installed Jenkins Version 2.99, none of my slaves did connect to the server anymore. Even a restart of the java script on the slave did not help."

      Workaround: Update Remoting on the agent side to 3.15

      In a job:

      First time build. Skipping changelog.
       No emails were triggered.
       FATAL: command execution failed
       Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from mdbuildserver01217.gamesys.corp/10.194.194.147:60268
       at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1696)
       at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
       at hudson.remoting.Channel.call(Channel.java:909)
       at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1053)
       at hudson.Launcher$ProcStarter.start(Launcher.java:450)
       at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
       at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
       at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
       at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
       at hudson.model.Build$BuildExecution.build(Build.java:206)
       at hudson.model.Build$BuildExecution.doRun(Build.java:163)
       at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
       at hudson.model.Run.execute(Run.java:1727)
       at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
       at hudson.model.ResourceController.execute(ResourceController.java:97)
       at hudson.model.Executor.run(Executor.java:429)
       java.lang.NoSuchMethodError: hudson.Launcher$RemoteLaunchCallable.getOpenChannelOrFail()Lhudson/remoting/Channel;
       at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1292)
       at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1260)
       at hudson.remoting.UserRequest.perform(UserRequest.java:207)
       at hudson.remoting.UserRequest.perform(UserRequest.java:53)
       at hudson.remoting.Request$2.run(Request.java:358)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at hudson.remoting.Engine$1$1.run(Engine.java:98)
       at java.lang.Thread.run(Thread.java:748)
       Caused: java.io.IOException: Remote call on JNLP4-connect connection from mdbuildserver01217.gamesys.corp/10.194.194.147:60268 failed
       at hudson.remoting.Channel.call(Channel.java:917)
       at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1053)
       at hudson.Launcher$ProcStarter.start(Launcher.java:450)
       at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
       at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
       at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
       at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
       at hudson.model.Build$BuildExecution.build(Build.java:206)
       at hudson.model.Build$BuildExecution.doRun(Build.java:163)
       at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
       at hudson.model.Run.execute(Run.java:1727)
       at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
       at hudson.model.ResourceController.execute(ResourceController.java:97)
       at hudson.model.Executor.run(Executor.java:429)
       Build step 'Execute shell' marked build as failure

      Starting the agent up:

      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main createEngine
      INFO: Setting up slave: <SlaveName>
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Jan 02, 2018 7:55:08 PM hudson.remoting.Engine startEngine
      WARNING: No Working Directory. Using the legacy JAR Cache location: /Users/mdbuildserver/.jenkins/cache/jars
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [https://<jenkins URL>/]
      Jan 02, 2018 7:55:08 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
        Agent address: <jenkins URL>/
        Agent port:    42650
        Identity:      fb:f3:24:67:05:be:94:06:49:2a:aa:64:c2:83:32:c7
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to nps-jenkins.gamesys.co.uk:42650
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP4-connect
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Remote identity confirmed: fb:f3:24:67:05:be:94:06:49:2a:aa:64:c2:83:32:c7
      Jan 02, 2018 7:55:08 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connected
      Jan 02, 2018 7:55:08 PM hudson.remoting.UserRequest perform
      WARNING: LinkageError while performing UserRequest:hudson.slaves.ChannelPinger$SetUpRemotePing@2905
      java.lang.NoSuchMethodError: hudson.slaves.ChannelPinger$SetUpRemotePing.getOpenChannelOrFail()Lhudson/remoting/Channel;
          at hudson.slaves.ChannelPinger$SetUpRemotePing.call(ChannelPinger.java:137)
          at hudson.slaves.ChannelPinger$SetUpRemotePing.call(ChannelPinger.java:121)
          at hudson.remoting.UserRequest.perform(UserRequest.java:207)
          at hudson.remoting.UserRequest.perform(UserRequest.java:53)
          at hudson.remoting.Request$2.run(Request.java:358)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at hudson.remoting.Engine$1$1.run(Engine.java:98)
          at java.lang.Thread.run(Thread.java:748)
      
      Jan 02, 2018 7:55:08 PM hudson.remoting.UserRequest perform
      WARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@3b16348f
      java.lang.NoSuchMethodError: jenkins.slaves.StandardOutputSwapper$ChannelSwapper.getOpenChannelOrFail()Lhudson/remoting/Channel;
          at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:42)
          at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39)
          at hudson.remoting.UserRequest.perform(UserRequest.java:207)
          at hudson.remoting.UserRequest.perform(UserRequest.java:53)
          at hudson.remoting.Request$2.run(Request.java:358)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at hudson.remoting.Engine$1$1.run(Engine.java:98)
          at java.lang.Thread.run(Thread.java:748)
      
      Jan 02, 2018 7:55:09 PM com.youdevise.hudson.slavestatus.SlaveListener call
      INFO: Slave-status listener starting
      Jan 02, 2018 7:55:09 PM com.youdevise.hudson.slavestatus.SocketHTTPListener waitForConnection
      INFO: Slave-status listener ready on port 3141
      

      Agent is launched like so:

      java -jar agent.jar -jnlpUrl https://<Jenkins URL>/computer/<AgentName>/slave-agent.jnlp -secret <Agent Secret>
      

        Attachments

          Issue Links

            Activity

            Hide
            panajev Goffredo Marocchi added a comment - - edited

            May be related... Suggested workaround is to update remoting to 3.15 for the agents. Oleg Nenashev how would I do that?

            Remoting on the master should have happened with the 2.98 Jenkins update:

             Update Remoting from 3.14 to 3.15 to fix several issues. (full changelog, issue 37566, issue 37670, issue 38696, issue 46724, issue 47965, issue 48055, issue 48130, issue 48133, issue 48309) 
            
            Show
            panajev Goffredo Marocchi added a comment - - edited May be related... Suggested workaround is to update remoting to 3.15 for the agents. Oleg Nenashev how would I do that? Remoting on the master should have happened with the 2.98 Jenkins update: Update Remoting from 3.14 to 3.15 to fix several issues. (full changelog, issue 37566, issue 37670, issue 38696, issue 46724, issue 47965, issue 48055, issue 48130, issue 48133, issue 48309)
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            > Suggested workaround is to update remoting to 3.15 for the agents. Oleg Nenashev how would I do that?

            Depends on which Launcher you use...

            • SSH Slaves get updated automatically
            • Windows Slaves (Java Web Start ones) may need manual update of slave.jar OR service restart if automatic upgrade is enabled
            • Swarm Plugin - update to the latest Swarm Plugin Client
            • Other agent types - to be handled on a case-by-case basis
            Show
            oleg_nenashev Oleg Nenashev added a comment - > Suggested workaround is to update remoting to 3.15 for the agents. Oleg Nenashev how would I do that? Depends on which Launcher you use... SSH Slaves get updated automatically Windows Slaves (Java Web Start ones) may need manual update of slave.jar OR service restart if automatic upgrade is enabled Swarm Plugin - update to the latest Swarm Plugin Client Other agent types - to be handled on a case-by-case basis
            Hide
            panajev Goffredo Marocchi added a comment -

            Stupid me...

            wget https://<Jenkins CI>/jnlpJars/agent.jar
            

            ... results in a connected Agent .

            Show
            panajev Goffredo Marocchi added a comment - Stupid me... wget https: //<Jenkins CI>/jnlpJars/agent.jar ... results in a connected Agent .
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Well, it SHOULD be working with old agent versions

            Show
            oleg_nenashev Oleg Nenashev added a comment - Well, it SHOULD be working with old agent versions
            Hide
            panajev Goffredo Marocchi added a comment -

            Fair enough, thanks for working on it .

            Show
            panajev Goffredo Marocchi added a comment - Fair enough, thanks for working on it .
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/Launcher.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/slaves/ChannelPinger.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/security/MasterToSlaveCallable.java
            core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
            http://jenkins-ci.org/commit/jenkins/4eb314b32742a07dcc68e716b7de1d31f0f917c8
            Log:
            JENKINS-48761 - Restore binary compatibility with agents running old Remoting versions

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/Launcher.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/slaves/ChannelPinger.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/security/MasterToSlaveCallable.java core/src/main/java/jenkins/slaves/StandardOutputSwapper.java http://jenkins-ci.org/commit/jenkins/4eb314b32742a07dcc68e716b7de1d31f0f917c8 Log: JENKINS-48761 - Restore binary compatibility with agents running old Remoting versions
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/jenkins/security/MasterToSlaveCallable.java
            http://jenkins-ci.org/commit/jenkins/cf1589734f98d895ecfab0ffda18976527f8d507
            Log:
            JENKINS-48761 - Use pre-Remoting 3.0 ChannelClosedException APIs

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/jenkins/security/MasterToSlaveCallable.java http://jenkins-ci.org/commit/jenkins/cf1589734f98d895ecfab0ffda18976527f8d507 Log: JENKINS-48761 - Use pre-Remoting 3.0 ChannelClosedException APIs
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            test/pom.xml
            test/src/test/java/jenkins/slaves/OldRemotingAgentTest.java
            http://jenkins-ci.org/commit/jenkins/8a34983d4571bab3acce87bb84ecd05a8251e528
            Log:
            JENKINS-48761 - Create a smoke test for old agents

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: test/pom.xml test/src/test/java/jenkins/slaves/OldRemotingAgentTest.java http://jenkins-ci.org/commit/jenkins/8a34983d4571bab3acce87bb84ecd05a8251e528 Log: JENKINS-48761 - Create a smoke test for old agents
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/filter/resources/jenkins/slaves/remoting-info.properties
            core/src/main/java/hudson/Launcher.java
            core/src/main/java/hudson/TcpSlaveAgentListener.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/slaves/ChannelPinger.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/security/MasterToSlaveCallable.java
            core/src/main/java/jenkins/slaves/RemotingVersionInfo.java
            core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
            core/src/test/java/jenkins/slaves/RemotingVersionInfoTest.java
            pom.xml
            test/pom.xml
            test/src/test/java/jenkins/slaves/OldRemotingAgentTest.java
            http://jenkins-ci.org/commit/jenkins/01b8ed8c71dee97d661bd5bf4b8e4bc5ecd636ae
            Log:
            JENKINS-48761 - Address review comments from @jglick

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/filter/resources/jenkins/slaves/remoting-info.properties core/src/main/java/hudson/Launcher.java core/src/main/java/hudson/TcpSlaveAgentListener.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/slaves/ChannelPinger.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/security/MasterToSlaveCallable.java core/src/main/java/jenkins/slaves/RemotingVersionInfo.java core/src/main/java/jenkins/slaves/StandardOutputSwapper.java core/src/test/java/jenkins/slaves/RemotingVersionInfoTest.java pom.xml test/pom.xml test/src/test/java/jenkins/slaves/OldRemotingAgentTest.java http://jenkins-ci.org/commit/jenkins/01b8ed8c71dee97d661bd5bf4b8e4bc5ecd636ae Log: JENKINS-48761 - Address review comments from @jglick
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/filter/resources/jenkins/slaves/remoting-info.properties
            core/src/main/java/hudson/Launcher.java
            core/src/main/java/hudson/TcpSlaveAgentListener.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/security/MasterToSlaveCallable.java
            core/src/main/java/jenkins/slaves/RemotingVersionInfo.java
            core/src/test/java/jenkins/slaves/RemotingVersionInfoTest.java
            pom.xml
            test/pom.xml
            test/src/test/java/jenkins/slaves/OldRemotingAgentTest.java
            http://jenkins-ci.org/commit/jenkins/8eb4254b0f56086f6576c108913fc5d066f4be9b
            Log:
            Merge pull request #3212 from oleg-nenashev/bug/JENKINS-48761-remoting-API

            JENKINS-48761 - Restore binary compatibility with agents running old Remoting versions

            Compare: https://github.com/jenkinsci/jenkins/compare/99ca10114332...8eb4254b0f56

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/filter/resources/jenkins/slaves/remoting-info.properties core/src/main/java/hudson/Launcher.java core/src/main/java/hudson/TcpSlaveAgentListener.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/security/MasterToSlaveCallable.java core/src/main/java/jenkins/slaves/RemotingVersionInfo.java core/src/test/java/jenkins/slaves/RemotingVersionInfoTest.java pom.xml test/pom.xml test/src/test/java/jenkins/slaves/OldRemotingAgentTest.java http://jenkins-ci.org/commit/jenkins/8eb4254b0f56086f6576c108913fc5d066f4be9b Log: Merge pull request #3212 from oleg-nenashev/bug/ JENKINS-48761 -remoting-API JENKINS-48761 - Restore binary compatibility with agents running old Remoting versions Compare: https://github.com/jenkinsci/jenkins/compare/99ca10114332...8eb4254b0f56
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            It should be resolved in Jenkins 2.100, the release is currently in the progress.
            Retrospective thread: https://groups.google.com/forum/#!topic/jenkinsci-dev/-VQ5YU0yYhg

            Show
            oleg_nenashev Oleg Nenashev added a comment - It should be resolved in Jenkins 2.100, the release is currently in the progress. Retrospective thread: https://groups.google.com/forum/#!topic/jenkinsci-dev/-VQ5YU0yYhg
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            content/_data/changelogs/weekly.yml
            http://jenkins-ci.org/commit/jenkins.io/ac42d83bf51e56e826991839a5471eccdd767543
            Log:
            Changelog: noting JENKINS-48761/JENKINS-48754 in Jenkins 2.100.

            The PR also adds warning banners to 2.98/2.99

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: content/_data/changelogs/weekly.yml http://jenkins-ci.org/commit/jenkins.io/ac42d83bf51e56e826991839a5471eccdd767543 Log: Changelog: noting JENKINS-48761 / JENKINS-48754 in Jenkins 2.100. The PR also adds warning banners to 2.98/2.99
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: R. Tyler Croy
            Path:
            content/_data/changelogs/weekly.yml
            http://jenkins-ci.org/commit/jenkins.io/99fe6d097fd1cc1f6797ad7b816c98dec3261a37
            Log:
            Merge pull request #1305 from oleg-nenashev/changelog/2.100

            Changelog: noting JENKINS-48761/JENKINS-48754 in Jenkins 2.100.

            Compare: https://github.com/jenkins-infra/jenkins.io/compare/62362762ff55...99fe6d097fd1

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: content/_data/changelogs/weekly.yml http://jenkins-ci.org/commit/jenkins.io/99fe6d097fd1cc1f6797ad7b816c98dec3261a37 Log: Merge pull request #1305 from oleg-nenashev/changelog/2.100 Changelog: noting JENKINS-48761 / JENKINS-48754 in Jenkins 2.100. Compare: https://github.com/jenkins-infra/jenkins.io/compare/62362762ff55...99fe6d097fd1

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                panajev Goffredo Marocchi
              • Votes:
                3 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: