Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22993

Maven plugin version 2.3 freezes the ssh node if it runs with the mvn -T flag.

XMLWordPrintable

      The problem started occurring after upgrading from maven plugin 2.1.

      The node becomes entirely unresponsive and any other builds or scm pollings also freeze on that node. The jenkins node need to be manually reconnected to clear the fault.
      If a git SCM poll was active on the node then the jenkins master must be restarted for the polling status page to clear.
      The problem does not seem to be intermittent and is easy to reproduce for us.

      Attaching jstacks and a mocked multimodule job that replicates the problem.

      Using SSH slaves plugin 1.6 and Jenkins LTS 1.554.1.
      Java 7u51 x86-64.

      Running on the master node without ssh slave plugin does not cause a freeze.
      Running without the -T flag does not cause a freeze.
      Running with maven-plugin 2.1 does not cause a freeze.
      Have not tested with maven plugin version 2.2.

      Tested with maven 3.1.1. Maven 3.2.1 with -T flag does not seem supported by the plugin at all.
      Goals: compile -T 4
      MAVEN_OPTS=-Xms128M -Xmx1G -verbose:gc

      Jenkins is started via ubuntu upstart with the "java -jar jenkins.war" method. It is however manually installed.
      Jenkins JVM options: -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/tmp/jenkins-test/loggc.txt -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xmx1024m -Xms1024m -XX:SoftRefLRUPolicyMSPerMB=2000 -XX:MaxPermSize=192m -server -Djava.awt.headless=true -Dhudson.master.headless=true -Dhudson.webstart.headless=true -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -DJENKINS_HOME=/local/jenkins-test -jar /local/jenkins-test/jenkins.war --javaHome=/opt/local/dev_tools/java/x64/jdk1.7.0_51/jre --prefix=/jenkins --logfile=/local/jenkins-test/jenkins.log --httpListenAddress=127.0.0.1 --httpPort=18080 --httpsListenAddress=127.0.0.1 --httpsPort=-1 --ajp13ListenAddress=127.0.0.1 --ajp13Port=-1 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

      Node JVM options: -Xms8m -Xmx128m -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/local/jenkins-test-agents/jenkinsadm/loggc.txt -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
      No out of memory problem indications in -Xloggc or -verbose:gc.

      I have tried with and without these, no effect:
      Use private Maven repository: Local to workspace
      Run headless
      Disable triggering of downstream projects
      Disable automatic site documentation artifact archiving
      Disable automatic artifact archiving

      The "Build modules in parallel" functionality is not a viable option for us, we require that the build be done within the same job.

      Tried both with and without the "chrt -b 0" batch scheduler. Normally we start the nodes with the batch scheduler as a prefix.
      Tried to run nodes both on RHEL 6.2 over lan and Ubuntu 14.04 on localhost. Both are multicore machines.

      I have renamed some server and usernames for security reasons.
      Hanging job name: ThomasHerrlin_ticket_3348
      Hanging slave node: server030

      Jenkins console output:

      ...
      Executing Maven: -B -f /ssd/jenkins-test-agents/jenkinsadm/workspace/ThomasHerrlin_ticket_3348/pom.xml -T4 compile
      [GC 33024K->4405K(125952K), 0.0085580 secs]
      [GC 37429K->6340K(125952K), 0.0109360 secs]
      [INFO] Scanning for projects...
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Build Order:
      [INFO]
      [INFO] my-app1
      [INFO] my-app2
      [INFO] my-app3
      [INFO] my-app4
      [INFO] buildall_test

      [HUDSON] Collecting dependencies info
      [HUDSON] Collecting dependencies info[INFO] Building with 4 threads

      [HUDSON] Collecting dependencies info
      [HUDSON] Collecting dependencies info

      At this point it freezes.

            schristou Steven Christou
            thomas_herrlin Thomas Herrlin
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: