Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33886

Can only connect one JNLP3 slave per IP address

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I'm running into an issue with JNLP3. I have three slaves communicating through a proxy, so by the time the traffic gets to the master, they are coming from the same IP address. The port numbers are unique; they are ephemeral ports. With JNLP3, I can't get more than one slave connected at a time. Other slaves connect, but then I get a message:

      WARNING: Making

      {slave name}

      offline because it’s not responding
      Mar 28, 2016 11:35:31 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor

      If I remove the "-Djenkins.slaves.JnlpSlaveAgentProtocol3.enabled=true" switch (reverting to JNLP2), I can connect multiple slaves at the same time, no issues.

        Attachments

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            Haven't investigated, from superficially this is more a bug related to a feature that 'adds security', rather than a security vulnerability.

            Oleg Nenashev Did you intend for this to be filed here?

            Show
            danielbeck Daniel Beck added a comment - Haven't investigated, from superficially this is more a bug related to a feature that 'adds security', rather than a security vulnerability. Oleg Nenashev Did you intend for this to be filed here?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I think it's rather a security implementation-related bug than a security issue.

            Show
            oleg_nenashev Oleg Nenashev added a comment - I think it's rather a security implementation-related bug than a security issue.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The difference is that we could be able to process it as a common bug and to release a fix in remoting without syncing up with the next LTS release

            Show
            oleg_nenashev Oleg Nenashev added a comment - The difference is that we could be able to process it as a common bug and to release a fix in remoting without syncing up with the next LTS release
            Hide
            jeffkayser Jeff Kayser added a comment -

            Sorry for mis-categorizing it. I'm not sure how to change the category (I'm a Jira newbie). Please feel free to change it as desired. My feelings won't be hurt.

            Show
            jeffkayser Jeff Kayser added a comment - Sorry for mis-categorizing it. I'm not sure how to change the category (I'm a Jira newbie). Please feel free to change it as desired. My feelings won't be hurt.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            No problem. We are having a SECURITY team meeting tomorrow, so we will agree on the action plan there
            .

            Show
            oleg_nenashev Oleg Nenashev added a comment - No problem. We are having a SECURITY team meeting tomorrow, so we will agree on the action plan there .
            Hide
            danielbeck Daniel Beck added a comment -

            If there is no way to exploit this, it's not a SECURITY issue. Oleg Nenashev Do you see some possibility for that?

            Show
            danielbeck Daniel Beck added a comment - If there is no way to exploit this, it's not a SECURITY issue. Oleg Nenashev Do you see some possibility for that?
            Hide
            teilo James Nord added a comment -

            As long as we only disconnect the others after the security handshake of the latest, otherwise it could be a DOS attack vector.

            Show
            teilo James Nord added a comment - As long as we only disconnect the others after the security handshake of the latest, otherwise it could be a DOS attack vector.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            My only concern is about the case when multiple slaves manage to connect:
            1) First slave gets JNLP3
            2) Others get decline from JNLP3 and get connected via JNLP2
            3) A user who knows the behavior may be able to retrieve the data from there

            The behavior does not seem to happen according to the description from Jeff.
            I think we should fix it as a common bug.

            Show
            oleg_nenashev Oleg Nenashev added a comment - My only concern is about the case when multiple slaves manage to connect: 1) First slave gets JNLP3 2) Others get decline from JNLP3 and get connected via JNLP2 3) A user who knows the behavior may be able to retrieve the data from there The behavior does not seem to happen according to the description from Jeff. I think we should fix it as a common bug.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            A common CRITICAL bug, of course

            Show
            oleg_nenashev Oleg Nenashev added a comment - A common CRITICAL bug, of course
            Hide
            jeffkayser Jeff Kayser added a comment -

            Hi, all.

            I upgrade master and slaves to 1.655.

            Here is what I observed:

            First JNLP3 connection: connects OK.
            Second JNLP3 connection: connects OK, but kicks off the first slave.
            Third JNLP3 connection: hangs. It is eventually disconnected on the master end. However, The slave never got the shutdown message from the master; you have to shutdown the slave.

            Once the master is setup for JNLP3, I did not see JNLP3-capable slaves downgrading to JNLP2. Not sure that the downgrade logic is working as expected.

            I did verify that it was the number of IPs connected. I stopped the first and second slave, then retried to do JNLP3 connect for the third slave. It connected OK.

            After I had these issues with JNLP3, I reconfigured the master to not support JNLP3. Slaves connected via JNLP2, and all was working properly as before.

            Show
            jeffkayser Jeff Kayser added a comment - Hi, all. I upgrade master and slaves to 1.655. Here is what I observed: First JNLP3 connection: connects OK. Second JNLP3 connection: connects OK, but kicks off the first slave. Third JNLP3 connection: hangs. It is eventually disconnected on the master end. However, The slave never got the shutdown message from the master; you have to shutdown the slave. Once the master is setup for JNLP3, I did not see JNLP3-capable slaves downgrading to JNLP2. Not sure that the downgrade logic is working as expected. I did verify that it was the number of IPs connected. I stopped the first and second slave, then retried to do JNLP3 connect for the third slave. It connected OK. After I had these issues with JNLP3, I reconfigured the master to not support JNLP3. Slaves connected via JNLP2, and all was working properly as before.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Moving it outside security according to the discussion

            Show
            oleg_nenashev Oleg Nenashev added a comment - Moving it outside security according to the discussion
            Hide
            flow86 Florian Doersch added a comment -

            get the same problem,

            4 windows virtual machines connected via jnlp to the jenkins "master" via NAT,
            seems only the "last" machine started works, all others give "channel already closed" on starting a job - but they still think they're connected.

            Show
            flow86 Florian Doersch added a comment - get the same problem, 4 windows virtual machines connected via jnlp to the jenkins "master" via NAT, seems only the "last" machine started works, all others give "channel already closed" on starting a job - but they still think they're connected.
            Hide
            akshay_abd akshay_abd added a comment - - edited

            I tried 2 different simple setups today and could not reproduce the issue:

            • 3 JNLP nodes connect to the master from the same machine
            • 2 JNLP nodes connect to the master from different machines

            In both cases the nodes were able to stay connected and do work. Jeff Kayser - you mentioned that your setup involved a proxy - could you tell me some details so I can mirror your setup?

            Also - if you have time can you confirm that a "simple" setup - ie. nodes connecting without a proxy involved work?

            Show
            akshay_abd akshay_abd added a comment - - edited I tried 2 different simple setups today and could not reproduce the issue: 3 JNLP nodes connect to the master from the same machine 2 JNLP nodes connect to the master from different machines In both cases the nodes were able to stay connected and do work. Jeff Kayser - you mentioned that your setup involved a proxy - could you tell me some details so I can mirror your setup? Also - if you have time can you confirm that a "simple" setup - ie. nodes connecting without a proxy involved work?
            Hide
            jeffkayser Jeff Kayser added a comment -

            Hi, Akshay.

            You don't need a proxy. Just using an intervening Firewall/Router should do the trick. Basically, if your Master is connecting to slaves across the Internet, the JNLP3 logic will have problems.

            You should be able to replicate via:

            Master <=> Firewall/Router <=> Internet <=> Firewall/Router <=> multiple slave machines.

            Or, more simply:

            Master <=> Network <=> Firewall/Router <=> Slaves.

            The idea is that the Firewall/Router will NAT the IP addresses of the slaves to one IP address, and the Master, seeing multiple slaves connecting via the same IP address, will get confused.

            For example:

            Slave IP addresses:
            192.168.10.1
            192.168.10.2
            192.168.10.3

            NAT firewall / Router:

            Private IP <=> Public IP
            192.168.10.1 <=> 1.2.3.4 port 100
            192.168.10.2 <=> 1.2.3.4 port 200
            192.168.10.3 <=> 1.2.3.4 port 300

            Converts all private IP addresses of the slave machines (behind the
            firewall) to the publicly routeable IP address that firewall uses to connect to the Internet.

            Internet:

            Or any other intervening network.

            Master:

            Will see three different slaves try to connect with the same IP address
            (1.2.3.4) and different NAT ports.

            Master will get confused.

            I hope this helps.

            I have slaves that are on a remote network, across the Internet, with an intervening Firewall/Router, and JNLP3 did not work. JNLP2 works fine with that configuration.

            I'll create a simple test to replicate.

            Jeff Kayser
            Jeff.kayser@dbdr.com

            Show
            jeffkayser Jeff Kayser added a comment - Hi, Akshay. You don't need a proxy. Just using an intervening Firewall/Router should do the trick. Basically, if your Master is connecting to slaves across the Internet, the JNLP3 logic will have problems. You should be able to replicate via: Master <=> Firewall/Router <=> Internet <=> Firewall/Router <=> multiple slave machines. Or, more simply: Master <=> Network <=> Firewall/Router <=> Slaves. The idea is that the Firewall/Router will NAT the IP addresses of the slaves to one IP address, and the Master, seeing multiple slaves connecting via the same IP address, will get confused. For example: Slave IP addresses: 192.168.10.1 192.168.10.2 192.168.10.3 NAT firewall / Router: Private IP <=> Public IP 192.168.10.1 <=> 1.2.3.4 port 100 192.168.10.2 <=> 1.2.3.4 port 200 192.168.10.3 <=> 1.2.3.4 port 300 Converts all private IP addresses of the slave machines (behind the firewall) to the publicly routeable IP address that firewall uses to connect to the Internet. Internet: Or any other intervening network. Master: Will see three different slaves try to connect with the same IP address (1.2.3.4) and different NAT ports. Master will get confused. I hope this helps. I have slaves that are on a remote network, across the Internet, with an intervening Firewall/Router, and JNLP3 did not work. JNLP2 works fine with that configuration. I'll create a simple test to replicate. Jeff Kayser Jeff.kayser@dbdr.com
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Starting from JENKINS-34819 (remoting-2.59) it is possible to disable the protocol on the slave side. It should allow working around the issue

            Show
            oleg_nenashev Oleg Nenashev added a comment - Starting from JENKINS-34819 (remoting-2.59) it is possible to disable the protocol on the slave side. It should allow working around the issue
            Hide
            sag47 Sam Gleske added a comment - - edited

            I plan to roll Mac slaves behind a NAT and would prefer their communication to be encrypted. I've voted/watched for a resolution.

            Show
            sag47 Sam Gleske added a comment - - edited I plan to roll Mac slaves behind a NAT and would prefer their communication to be encrypted. I've voted/watched for a resolution.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Sam Gleske In middle-term there is a plan to release JNLP4 (JENKINS-36871).
            The main PR is merged into the remoting master branch towards 3.0

            Show
            oleg_nenashev Oleg Nenashev added a comment - Sam Gleske In middle-term there is a plan to release JNLP4 ( JENKINS-36871 ). The main PR is merged into the remoting master branch towards 3.0
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            JNLP4 protocol is now available in both Jenkins Weekly and LTS lines. JNLP3 is explicitly not recommended for use. IMHO the recommendation would be to migrate from JNLp3 to JNLP4 if any issue happens.

            Show
            oleg_nenashev Oleg Nenashev added a comment - JNLP4 protocol is now available in both Jenkins Weekly and LTS lines. JNLP3 is explicitly not recommended for use. IMHO the recommendation would be to migrate from JNLp3 to JNLP4 if any issue happens.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            docs/protocols.md
            src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java
            http://jenkins-ci.org/commit/remoting/fe2587b7f9d78334e0ab05ab0b95f39b4b600a25
            Log:
            Docs - Noting JENKINS-37302, JENKINS-33886, and JENKINS-34121 in Errata

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: docs/protocols.md src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java http://jenkins-ci.org/commit/remoting/fe2587b7f9d78334e0ab05ab0b95f39b4b600a25 Log: Docs - Noting JENKINS-37302 , JENKINS-33886 , and JENKINS-34121 in Errata
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            docs/protocols.md
            src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java
            http://jenkins-ci.org/commit/remoting/86e13055079fd679a46b06fc7ce54ea1eb33ac1f
            Log:
            Merge pull request #155 from oleg-nenashev/doc/jnlp3_errata

            [Docs] - Noting JENKINS-37302, JENKINS-33886, and JENKINS-34121 in JNLP3 Errata

            Compare: https://github.com/jenkinsci/remoting/compare/b8f10d809829...86e13055079f

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: docs/protocols.md src/main/java/org/jenkinsci/remoting/engine/JnlpProtocol3Handler.java http://jenkins-ci.org/commit/remoting/86e13055079fd679a46b06fc7ce54ea1eb33ac1f Log: Merge pull request #155 from oleg-nenashev/doc/jnlp3_errata [Docs] - Noting JENKINS-37302 , JENKINS-33886 , and JENKINS-34121 in JNLP3 Errata Compare: https://github.com/jenkinsci/remoting/compare/b8f10d809829...86e13055079f
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Closing as "Won't fix", upgrade to JNLP4 is the recommended solution.

            If somebody wants to fix it, please feel free to reopen it and to create a pull request

            Show
            oleg_nenashev Oleg Nenashev added a comment - Closing as "Won't fix", upgrade to JNLP4 is the recommended solution. If somebody wants to fix it, please feel free to reopen it and to create a pull request

              People

              • Assignee:
                Unassigned
                Reporter:
                jeffkayser Jeff Kayser
              • Votes:
                2 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: