Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19844

Maven agent socket bind too inflexible (allow Jenkins in virtualized environment)

    Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: maven-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      When creating the Maven agent Jenkins currently does the following in AbstractMavenProcessFactory:

      static final class AcceptorImpl implements Acceptor, Serializable {
          private transient final ServerSocket serverSocket;
      
          AcceptorImpl() throws IOException {
              // open a TCP socket to talk to the launched Maven process.
              // let the OS pick up a random open port
              this.serverSocket = new ServerSocket();
              serverSocket.bind(null);
              ...
          }
          ...
      }
      

      If the argument to bind is null then Java will bind to a random port on the wildcard interface. This is typically not allowed in a virtualized environment (think RedHat's OpenShift).

      First off, please note that the Javadoc in the JDK is slightly misleading for this method as it says :

      If the address is null, then the system will pick up an ephemeral port and a valid local address to bind the socket.

      The word "local address" in that sentence gives you the impression that Java will bind to 'localhost' interface. That is not what happens. It will bind on the wildcard interface. In many environments such action is not allowed and will fail with java.net.BindException: Permission denied

      (if you are not familiar with the term 'wildcard interface' it simply means all network interfaces, typically shown in netstat as a "*." prefix for IPv4 and as a "::" prefix for IPv6)

      So we need a way to override the address that the bind operation will use.
      Something like:

      String address = System.getenv("JENKINS_MAVEN_AGENT_ADDRESS");
      if (address != null) {
          serverSocket.bind(new InetSocketAddress(InetAddress.getByName(address), 0));
      } else {
          serverSocket.bind(null);
      }
      

      There's a guy (not me) who has cloned Jenkins and made his own Jenkins just for this one purpose. Here's the project : https://github.com/siasia/jenkins/tree/jenkins-1.504-openshift

      It would be much better if this was to become part of standard Jenkins.

        Attachments

          Activity

          Hide
          phansson Peter Hansson added a comment - - edited

          Upon further analysis the client side of the tcp connection also needs to be changed. This means changes to the Jenkins Maven3 CLI Agent or Jenkins Maven3.1.x CLI Agent.

          Within that in Maven3Main.java or Maven31Main.java you'll see this:

           
              ... 
          	String mavenRemoteUseInetEnvVar = System.getenv( "MAVEN_REMOTE_USEINET" ); 
          	
          	boolean mavenRemoteUseInet = Boolean.parseBoolean( mavenRemoteUseInetEnvVar ); 
          	
          	if(mavenRemoteUseInet) { 
          	    InetAddress host = InetAddress.getLocalHost(); 
          		String hostname = host.getHostName(); 
          		System.out.println( "use inet address " + hostname ); 
          		s = new Socket(hostname,tcpPort); 
              } 
          	else s = new Socket((String)null,tcpPort); 
          	... 
          

          If we're using same semantics as for the server side of the connection (explained in the original posting) then that piece of code needs to changed to something like this:

           
              ... 
          	String mavenRemoteUseInetEnvVar = System.getenv( "MAVEN_REMOTE_USEINET" ); 
          	String mavenRemoteUseInetAddr = System.getenv( "JENKINS_MAVEN_AGENT_ADDRESS" );
          
          	boolean mavenRemoteUseInet = Boolean.parseBoolean( mavenRemoteUseInetEnvVar ); 
          	
          	if(mavenRemoteUseInet) { 
          	    String hostname; 
          		if (mavenRemoteUseInetAddr == null) { 
          		    InetAddress host = InetAddress.getLocalHost(); 
          			hostname = host.getHostName(); 
          		} else { 
          		    hostname = mavenRemoteUseInetAddr; 
          		} 
          		System.out.println( "use inet address " + hostname ); 
          		s = new Socket(hostname,tcpPort); 
          	} 
          	    else s = new Socket((String)null,tcpPort);
          	... 
          

          This works well for me.

          On another note, I would say that the MAVEN_REMOTE_USEINET is mis-named. It has nothing to do as such with Maven, but is indeed invented by Hudson/Jenkins. It should therefore, IMHO, have been named with a Hudson/Jenkins prefix.

          Let's just re-iterate what the MAVEN_REMOTE_USEINET does (before the change): If defined and equals true it makes the Maven agent connect using the host's own network name as the endpoint address. If MAVEN_REMOTE_USEINET is not defined (which is the default) or is false then the connection will be attempted on the loopback interface. So INET is used in both cases for the connection it is just that whether which network interface will be used. (a further reason why the variable is strangely named).

          Frankly MAVEN_REMOTE_USEINET was never a stellar solution to begin with. The reason is that determining a host's own network name is not a precise science. On a host with 4 NICs then what happens? (I can give you a hint: the order of the network devices as seen from the OS will be what determines your answer from InetAddress.getLocalHost().). In other words : if MAVEN_REMOTE_USEINET works for you it is probably down to luck more than anything else. This is of course exaggerated as most hosts don't have that many NICs and in the end the answer from InetAddress.getLocalHost() is as expected. But precise it ain't.

          Show
          phansson Peter Hansson added a comment - - edited Upon further analysis the client side of the tcp connection also needs to be changed. This means changes to the Jenkins Maven3 CLI Agent or Jenkins Maven3.1.x CLI Agent . Within that in Maven3Main.java or Maven31Main.java you'll see this: ... String mavenRemoteUseInetEnvVar = System .getenv( "MAVEN_REMOTE_USEINET" ); boolean mavenRemoteUseInet = Boolean .parseBoolean( mavenRemoteUseInetEnvVar ); if (mavenRemoteUseInet) { InetAddress host = InetAddress.getLocalHost(); String hostname = host.getHostName(); System .out.println( "use inet address " + hostname ); s = new Socket(hostname,tcpPort); } else s = new Socket(( String ) null ,tcpPort); ... If we're using same semantics as for the server side of the connection (explained in the original posting) then that piece of code needs to changed to something like this: ... String mavenRemoteUseInetEnvVar = System .getenv( "MAVEN_REMOTE_USEINET" ); String mavenRemoteUseInetAddr = System .getenv( "JENKINS_MAVEN_AGENT_ADDRESS" ); boolean mavenRemoteUseInet = Boolean .parseBoolean( mavenRemoteUseInetEnvVar ); if (mavenRemoteUseInet) { String hostname; if (mavenRemoteUseInetAddr == null ) { InetAddress host = InetAddress.getLocalHost(); hostname = host.getHostName(); } else { hostname = mavenRemoteUseInetAddr; } System .out.println( "use inet address " + hostname ); s = new Socket(hostname,tcpPort); } else s = new Socket(( String ) null ,tcpPort); ... This works well for me. On another note, I would say that the MAVEN_REMOTE_USEINET is mis-named. It has nothing to do as such with Maven, but is indeed invented by Hudson/Jenkins. It should therefore, IMHO, have been named with a Hudson/Jenkins prefix. Let's just re-iterate what the MAVEN_REMOTE_USEINET does (before the change): If defined and equals true it makes the Maven agent connect using the host's own network name as the endpoint address. If MAVEN_REMOTE_USEINET is not defined (which is the default) or is false then the connection will be attempted on the loopback interface. So INET is used in both cases for the connection it is just that whether which network interface will be used. (a further reason why the variable is strangely named). Frankly MAVEN_REMOTE_USEINET was never a stellar solution to begin with. The reason is that determining a host's own network name is not a precise science. On a host with 4 NICs then what happens? (I can give you a hint: the order of the network devices as seen from the OS will be what determines your answer from InetAddress.getLocalHost() .). In other words : if MAVEN_REMOTE_USEINET works for you it is probably down to luck more than anything else. This is of course exaggerated as most hosts don't have that many NICs and in the end the answer from InetAddress.getLocalHost() is as expected. But precise it ain't.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/resources/hudson/tasks/junit/CaseResult/summary.jelly
          core/src/main/resources/hudson/tasks/test/MatrixTestResult/index.jelly
          core/src/main/resources/hudson/tasks/test/MetaTabulatedResult/body.jelly
          core/src/main/resources/lib/hudson/aggregated-failed-tests.jelly
          core/src/main/resources/lib/hudson/failed-test.jelly
          http://jenkins-ci.org/commit/jenkins/a84d1bca7aa3303a096304362e562a57a1b1a321
          Log:
          [FIXED JENKINS-19844]

          Merged pull request #966

          Compare: https://github.com/jenkinsci/jenkins/compare/d45d18ed4e43...a84d1bca7aa3

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/resources/hudson/tasks/junit/CaseResult/summary.jelly core/src/main/resources/hudson/tasks/test/MatrixTestResult/index.jelly core/src/main/resources/hudson/tasks/test/MetaTabulatedResult/body.jelly core/src/main/resources/lib/hudson/aggregated-failed-tests.jelly core/src/main/resources/lib/hudson/failed-test.jelly http://jenkins-ci.org/commit/jenkins/a84d1bca7aa3303a096304362e562a57a1b1a321 Log: [FIXED JENKINS-19844] Merged pull request #966 Compare: https://github.com/jenkinsci/jenkins/compare/d45d18ed4e43...a84d1bca7aa3
          Hide
          odoepner Oliver Doepner added a comment -

          This issue was closed by mistake.

          The fix mentioned in the comments is for JENKINS-19884 "Improve test failure summary appearance" and has nothing to do with this issue here : JENKINS-19844 "Maven agent socket bind too inflexible"

          Probably someone got the similar issue numbers (19884 and 19844) mixed up.

          Show
          odoepner Oliver Doepner added a comment - This issue was closed by mistake. The fix mentioned in the comments is for JENKINS-19884 "Improve test failure summary appearance" and has nothing to do with this issue here : JENKINS-19844 "Maven agent socket bind too inflexible" Probably someone got the similar issue numbers (19884 and 19844) mixed up.
          Hide
          randroid Roberto Andrade added a comment -

          Have a PR on Github waiting to be merged for this: https://github.com/jenkinsci/maven-plugin/pull/43

          Show
          randroid Roberto Andrade added a comment - Have a PR on Github waiting to be merged for this: https://github.com/jenkinsci/maven-plugin/pull/43
          Hide
          lweller Lucien Weller added a comment - - edited

          Sorry reopened by mistake...

          @Roberto Andrade: you're pull request seams not to be fully ok. Can you fix it?

          Show
          lweller Lucien Weller added a comment - - edited Sorry reopened by mistake... @Roberto Andrade: you're pull request seams not to be fully ok. Can you fix it?
          Hide
          ndeloof Nicolas De Loof added a comment -

          https://github.com/jenkinsci/maven-plugin/commit/a30fe29cc919afbd6650371cb995050ebdfef26a introduces TcpSocketHostLocator that can be used on client side to locate TCP channel IP. I guess it could be extended to do the same server side, better than relying on env variables which look like backdoors vs node adequate configuration.

          Show
          ndeloof Nicolas De Loof added a comment - https://github.com/jenkinsci/maven-plugin/commit/a30fe29cc919afbd6650371cb995050ebdfef26a introduces TcpSocketHostLocator that can be used on client side to locate TCP channel IP. I guess it could be extended to do the same server side, better than relying on env variables which look like backdoors vs node adequate configuration.

            People

            • Assignee:
              Unassigned
              Reporter:
              phansson Peter Hansson
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: