Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32542

Maven project failed with "Connection refused" when built in a docker Container

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Building a "Maven project" in a Docker container fails with a "connection refused" error message whereas the same build is successful in a "Freestyle project" (also build in a Docker container plugin).

      [ael.validate-in-docker-mvnproject] $ docker exec --tty --user 500: dc69be1aed975dbff977348d060329d75d1377ecbefaf929a269a11fc5e40373 env ANT_HOME=/opt/apache-ant-1.8.2 CLASSPATH= EXECUTOR_NUMBER=0 JAVA_HOME=/opt/jdk1.7.0_45 JENKINS_HOME=/data/EDT/jenkins M2_HOME=/opt/apache-maven-3.0.4 MAVEN_HOME=/opt/apache-maven-3.0.4 "MAVEN_OPTS=-Xms1g -Xmx1g -XX:MaxPermSize=256m -Dfile.encoding=Cp1252" no_proxy=/var/run/docker.sock ORACLE_HOME=/opt/oracle11/product/11.2.0/client/ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/jdk1.7.0_45/bin:/opt/apache-ant-1.8.2/bin:/opt/apache-maven-3.0.4/bin TERM=xterm WORKSPACE=/data/EDT/jenkinsSlave_recette_1/c3f621fe/workspace/ael.validate-in-docker-mvnproject java -cp /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-agent.jar:/opt/apache-maven-3.0.4/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /opt/apache-maven-3.0.4 /data/EDT/jenkinsSlave_recette_1/slave.jar /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-interceptor.jar /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-interceptor-commons.jar 44848
      Exception in thread "main" java.net.ConnectException: Connection refused
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:579)
      at java.net.Socket.connect(Socket.java:528)
      at java.net.Socket.<init>(Socket.java:425)
      at java.net.Socket.<init>(Socket.java:208)
      at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:136)
      at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:70)
      Stopping Docker container after build completion

        Attachments

          Issue Links

            Activity

            Hide
            zcahill Zane Cahill added a comment -

            We're having the same issue... In our case, as Nicolas De Loof considered, the slave is running within our JNLP docker container deployed upon Amazon EC2 Container Service, using his plugin.

            The problem, it seems, stems from the network bridging; each container being in its own network and no "advanced" option within the CloudBees Docker Custom Build Environment Plugin to link containers. Although with a dynamically produced slave container name by the Amazon EC2 Container Service plugin, I guess this may need to utilise an environment variable if/when this feature becomes available?

            Setting the "Network bridge" to host as Aaron Corley suggest does not help in our case as the containers are still on separate networks. However, I'm presuming if Nicolas De Loof's plugin had an advanced option to configure the "Network Bridge" for the Slave and setting it to "host" could cure this issue. However there may be other consequences to this...

            For the time being we can build with a Freestyle project, but if I get a chance to review the plugin(s) sources I'll see if I can contribute a fix.

            Show
            zcahill Zane Cahill added a comment - We're having the same issue... In our case, as Nicolas De Loof considered, the slave is running within our JNLP docker container deployed upon Amazon EC2 Container Service, using his plugin . The problem, it seems, stems from the network bridging; each container being in its own network and no "advanced" option within the CloudBees Docker Custom Build Environment Plugin to link containers. Although with a dynamically produced slave container name by the Amazon EC2 Container Service plugin, I guess this may need to utilise an environment variable if/when this feature becomes available? Setting the "Network bridge" to host as Aaron Corley suggest does not help in our case as the containers are still on separate networks. However, I'm presuming if Nicolas De Loof 's plugin had an advanced option to configure the "Network Bridge" for the Slave and setting it to "host" could cure this issue. However there may be other consequences to this... For the time being we can build with a Freestyle project, but if I get a chance to review the plugin(s) sources I'll see if I can contribute a fix.
            Hide
            petermount petermount added a comment -

            I've had a docker java slave running happily for a few weeks now but after adding the Cloudbees docker build environment plugin (may be a coincidence) it failed with this same issue, with one minor difference in that it included a hostname in the logs:

            Modules changed, recalculating dependency graph
            Established TCP socket on dockerhost:48877
            maven32-agent.jar already up to date
            maven32-interceptor.jar already up to date
            maven3-interceptor-commons.jar already up to date
            [opendata-common] $ /opt/jdk//bin/java -cp /opt/jenkins/maven32-agent.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/boot/plexus-classworlds-2.5.2.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/conf/logging jenkins.maven3.agent.Maven32Main /opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3 /opt/jenkins/slave.jar /opt/jenkins/maven32-interceptor.jar /opt/jenkins/maven3-interceptor-commons.jar dockerhost:48877
            Exception in thread "main" java.net.ConnectException: Connection refused
            	at java.net.PlainSocketImpl.socketConnect(Native Method)
            	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
            	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
            	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
            	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
            	at java.net.Socket.connect(Socket.java:589)
            	at java.net.Socket.connect(Socket.java:538)
            	at java.net.Socket.<init>(Socket.java:434)
            	at java.net.Socket.<init>(Socket.java:211)
            	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:141)
            	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:71)
            ERROR: Failed to launch Maven. Exit code = 1
            

            Whilst looking for other possible causes for this I found JENKINS-29674 which stated:

            This is by intention, maven job type won't work with a docker container as it open an arbitrary socket for communication between slave agent and maven process, which we can't discover at container launch time

            So as the exception gave me a hostname & as this was using my own docker images (both jenkins server and slaves) I added as part of the slaves entrypoint script the following:

            echo "127.0.0.1 dockerhost" >>/etc/hosts
            

            Now this may be a fluke that it's working & I don't know where dockerhost is being defined (it's not in any of my config) but this may help in finding the underlying issue.

            Environment details: This was with Jenkins 2.5 (first spotted with 2.3) and an SSH based slave, both running within separate docker images. My docker images that have this fix in are area51/jenkins:2.5 & area51/jenkins-slave:2.5

            Show
            petermount petermount added a comment - I've had a docker java slave running happily for a few weeks now but after adding the Cloudbees docker build environment plugin (may be a coincidence) it failed with this same issue, with one minor difference in that it included a hostname in the logs: Modules changed, recalculating dependency graph Established TCP socket on dockerhost:48877 maven32-agent.jar already up to date maven32-interceptor.jar already up to date maven3-interceptor-commons.jar already up to date [opendata-common] $ /opt/jdk//bin/java -cp /opt/jenkins/maven32-agent.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/boot/plexus-classworlds-2.5.2.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/conf/logging jenkins.maven3.agent.Maven32Main /opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3 /opt/jenkins/slave.jar /opt/jenkins/maven32-interceptor.jar /opt/jenkins/maven3-interceptor-commons.jar dockerhost:48877 Exception in thread "main" java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.<init>(Socket.java:434) at java.net.Socket.<init>(Socket.java:211) at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:141) at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:71) ERROR: Failed to launch Maven. Exit code = 1 Whilst looking for other possible causes for this I found JENKINS-29674 which stated: This is by intention, maven job type won't work with a docker container as it open an arbitrary socket for communication between slave agent and maven process, which we can't discover at container launch time So as the exception gave me a hostname & as this was using my own docker images (both jenkins server and slaves) I added as part of the slaves entrypoint script the following: echo "127.0.0.1 dockerhost" >>/etc/hosts Now this may be a fluke that it's working & I don't know where dockerhost is being defined (it's not in any of my config) but this may help in finding the underlying issue. Environment details: This was with Jenkins 2.5 (first spotted with 2.3) and an SSH based slave, both running within separate docker images. My docker images that have this fix in are area51/jenkins:2.5 & area51/jenkins-slave:2.5
            Hide
            mryan Manuel Ryan added a comment -

            I'm using the kubernetes plugin to provision fungible slaves in a cluster for each build and I have this issue too.

            Maybe the "dockerhost" hostname should actually be "jenkinshost" and be obtained by running somethig like "hostname -i" on the jenkins host performing the build (the slave container in my case) instead of the "ip route in an alpine:3.2 container" technique.

            Some other issues remain as the port needs to be exposed in the slave pod.

            • The kubernetes plugin needs to be able to specify port exposition in slave templates
            • The jenkins-maven plugin needs to support using a fixed port for the jvm agent -> build host connection. (this would cause collisions in most cases, but not when using the kubernetes plugin as each build has it's own IP address).
            Show
            mryan Manuel Ryan added a comment - I'm using the kubernetes plugin to provision fungible slaves in a cluster for each build and I have this issue too. Maybe the "dockerhost" hostname should actually be "jenkinshost" and be obtained by running somethig like "hostname -i" on the jenkins host performing the build (the slave container in my case) instead of the "ip route in an alpine:3.2 container" technique. Some other issues remain as the port needs to be exposed in the slave pod. The kubernetes plugin needs to be able to specify port exposition in slave templates The jenkins-maven plugin needs to support using a fixed port for the jvm agent -> build host connection. (this would cause collisions in most cases, but not when using the kubernetes plugin as each build has it's own IP address).
            Hide
            mryan Manuel Ryan added a comment -

            It all boils down to the "getDocker0" method in the custom-build-plugin. We should not assume that the address of the jenkins (master or slave) is the default gateway of the container it starts. The basic example is jenkins running in a container with a shared docker socket.

            I think a portable solution could be to allow master/slave provisionners to specify and environment variable (for the master/slave process) with the value computed depending on their networking environment (there are so many different scenarios, we can give sensible defaults, but we also need to allow full customization).

            The environment variable could be something like JENKINS_MAVEN_PLUGIN_HOST. I'll try to submit a PR tomorrow for this.

            Show
            mryan Manuel Ryan added a comment - It all boils down to the "getDocker0" method in the custom-build-plugin. We should not assume that the address of the jenkins (master or slave) is the default gateway of the container it starts. The basic example is jenkins running in a container with a shared docker socket. I think a portable solution could be to allow master/slave provisionners to specify and environment variable (for the master/slave process) with the value computed depending on their networking environment (there are so many different scenarios, we can give sensible defaults, but we also need to allow full customization). The environment variable could be something like JENKINS_MAVEN_PLUGIN_HOST. I'll try to submit a PR tomorrow for this.
            Hide
            prinztob Tobias Prinz added a comment -

            Have the same issue. Any update on the PR?

            Show
            prinztob Tobias Prinz added a comment - Have the same issue. Any update on the PR?

              People

              • Assignee:
                Unassigned
                Reporter:
                scarrier Sylvie Carrier
              • Votes:
                10 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated: