Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22993

Maven plugin version 2.3 freezes the ssh node if it runs with the mvn -T flag.

    Details

    • Similar Issues:

      Description

      The problem started occurring after upgrading from maven plugin 2.1.

      The node becomes entirely unresponsive and any other builds or scm pollings also freeze on that node. The jenkins node need to be manually reconnected to clear the fault.
      If a git SCM poll was active on the node then the jenkins master must be restarted for the polling status page to clear.
      The problem does not seem to be intermittent and is easy to reproduce for us.

      Attaching jstacks and a mocked multimodule job that replicates the problem.

      Using SSH slaves plugin 1.6 and Jenkins LTS 1.554.1.
      Java 7u51 x86-64.

      Running on the master node without ssh slave plugin does not cause a freeze.
      Running without the -T flag does not cause a freeze.
      Running with maven-plugin 2.1 does not cause a freeze.
      Have not tested with maven plugin version 2.2.

      Tested with maven 3.1.1. Maven 3.2.1 with -T flag does not seem supported by the plugin at all.
      Goals: compile -T 4
      MAVEN_OPTS=-Xms128M -Xmx1G -verbose:gc

      Jenkins is started via ubuntu upstart with the "java -jar jenkins.war" method. It is however manually installed.
      Jenkins JVM options: -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/tmp/jenkins-test/loggc.txt -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xmx1024m -Xms1024m -XX:SoftRefLRUPolicyMSPerMB=2000 -XX:MaxPermSize=192m -server -Djava.awt.headless=true -Dhudson.master.headless=true -Dhudson.webstart.headless=true -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -DJENKINS_HOME=/local/jenkins-test -jar /local/jenkins-test/jenkins.war --javaHome=/opt/local/dev_tools/java/x64/jdk1.7.0_51/jre --prefix=/jenkins --logfile=/local/jenkins-test/jenkins.log --httpListenAddress=127.0.0.1 --httpPort=18080 --httpsListenAddress=127.0.0.1 --httpsPort=-1 --ajp13ListenAddress=127.0.0.1 --ajp13Port=-1 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

      Node JVM options: -Xms8m -Xmx128m -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/local/jenkins-test-agents/jenkinsadm/loggc.txt -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
      No out of memory problem indications in -Xloggc or -verbose:gc.

      I have tried with and without these, no effect:
      Use private Maven repository: Local to workspace
      Run headless
      Disable triggering of downstream projects
      Disable automatic site documentation artifact archiving
      Disable automatic artifact archiving

      The "Build modules in parallel" functionality is not a viable option for us, we require that the build be done within the same job.

      Tried both with and without the "chrt -b 0" batch scheduler. Normally we start the nodes with the batch scheduler as a prefix.
      Tried to run nodes both on RHEL 6.2 over lan and Ubuntu 14.04 on localhost. Both are multicore machines.

      I have renamed some server and usernames for security reasons.
      Hanging job name: ThomasHerrlin_ticket_3348
      Hanging slave node: server030

      Jenkins console output:

      ...
      Executing Maven: -B -f /ssd/jenkins-test-agents/jenkinsadm/workspace/ThomasHerrlin_ticket_3348/pom.xml -T4 compile
      [GC 33024K->4405K(125952K), 0.0085580 secs]
      [GC 37429K->6340K(125952K), 0.0109360 secs]
      [INFO] Scanning for projects...
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Build Order:
      [INFO]
      [INFO] my-app1
      [INFO] my-app2
      [INFO] my-app3
      [INFO] my-app4
      [INFO] buildall_test

      [HUDSON] Collecting dependencies info
      [HUDSON] Collecting dependencies info[INFO] Building with 4 threads

      [HUDSON] Collecting dependencies info
      [HUDSON] Collecting dependencies info

      At this point it freezes.

        Attachments

        1. jenkins.jstack.xz
          8 kB
          Thomas Herrlin
        2. simpleproject.tar.xz
          1 kB
          Thomas Herrlin
        3. slave.jstack.xz
          2 kB
          Thomas Herrlin

          Issue Links

            Activity

            Hide
            usammmy usammmy added a comment -

            We are also noticing the same behavior with Jenkins 1.554.1 and maven-plugin 2.3

            Show
            usammmy usammmy added a comment - We are also noticing the same behavior with Jenkins 1.554.1 and maven-plugin 2.3
            Hide
            thomas_herrlin Thomas Herrlin added a comment - - edited

            Probably related to https://issues.jenkins-ci.org/browse/JENKINS-23098
            Preliminary tests in lab environment indicate that it has been fixed with maven plugin 2.5.
            Tested on Jenkins 1.554.3 with my attached simple maven project.

            Show
            thomas_herrlin Thomas Herrlin added a comment - - edited Probably related to https://issues.jenkins-ci.org/browse/JENKINS-23098 Preliminary tests in lab environment indicate that it has been fixed with maven plugin 2.5. Tested on Jenkins 1.554.3 with my attached simple maven project.
            Hide
            schristou Steven Christou added a comment -

            Duplicate of JENKINS-23098. This issue has been resolved in version 2.5 of the maven-plugin.

            Show
            schristou Steven Christou added a comment - Duplicate of JENKINS-23098 . This issue has been resolved in version 2.5 of the maven-plugin.

              People

              • Assignee:
                schristou Steven Christou
                Reporter:
                thomas_herrlin Thomas Herrlin
              • Votes:
                2 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: