Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15931

Build is hanging in xUnit plugin processing

    Details

    • Similar Issues:

      Description

      When using Amazon EC2 plugin, builds on newly created EC2 instances sometimes hang.
      We have two permanent slaves for a certain label (e.g. 'unittest'),
      which is generated from an AMI.
      The same AMI is specified within Amazon EC2 plugin settings.

      We have a job which can be executed concurrently
      When we invoke three builds at one time, two permanent slaves are exhausted
      and the new one is created.

      The problem is that the build on the new slave hangs at the end of it
      where xUnit plugin is aggregating the test result.

      [CHECKSTYLE] Collecting checkstyle analysis files...
      [CHECKSTYLE] Computing warning deltas based on reference build #850
      [FINDBUGS] Collecting findbugs analysis files...
      [FINDBUGS] Computing warning deltas based on reference build #850
      Archiving artifacts
      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing JUnit
      [xUnit] [INFO] - [JUnit] - 581 test report file(s) were found with the pattern '**/testresult/**/*.xml' relative to '/var/lib/jenkins/workspace/400_Precommit_Check_Branch' for the testing framework 'JUnit'.
      

      After aborting the build, the following error is shown.

      ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception
      java.lang.InterruptedException
      	at java.lang.Object.wait(Native Method)
      	at hudson.remoting.Request.call(Request.java:146)
      	at hudson.remoting.Channel.call(Channel.java:665)
      	at hudson.FilePath.act(FilePath.java:841)
      	at hudson.FilePath.act(FilePath.java:825)
      	at org.jenkinsci.plugins.xunit.XUnitPublisher.performTests(XUnitPublisher.java:170)
      	at org.jenkinsci.plugins.xunit.XUnitPublisher.performXUnit(XUnitPublisher.java:115)
      	at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:92)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:779)
      	at hudson.model.Build$BuildExecution.post2(Build.java:183)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
      	at hudson.model.Run.execute(Run.java:1541)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:236)
      Email was triggered for: Failure
      Sending email for trigger: Failure
      

      I tried to capture the thread dump, but both master and the target slave had EMPTY thread dump while another slave had its own.
      I'd appreciate if someone give me advice.

      Jenkins ver. 1.491
      xUnit plugin 1.51
      Amazon EC2 plugin 1.17

        Attachments

        1. hs_err_pid15419.log
          92 kB
        2. jstack_master.txt
          105 kB
        3. jstack_slave.out
          13 kB
        4. lsof_slave.out
          6 kB

          Activity

          Hide
          voellm Eckhard Völlm added a comment -

          Here i added the full Java trace from core dump of the hanging Jenkins Jobs.

          Show
          voellm Eckhard Völlm added a comment - Here i added the full Java trace from core dump of the hanging Jenkins Jobs.
          Hide
          voellm Eckhard Völlm added a comment - - edited

          Jenkins version 1.6.43
          xUnit version 1.99

          java version "1.7.0_91"
          OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1)
          OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)
          Linux slave 4.2.0-19-lowlatency #23~14.04.1-Ubuntu SMP PREEMPT Thu Nov 12 13:19:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

          Here a thread where Java threads also were hanging at "java.util.zip.ZipFile.getEntry":
          https://github.com/aws/aws-sdk-java/issues/238
          same as we observe here. They also found a solution, IMHO:
          "And yes, using this system property gets rid of the thread blocks :+1: Thank you so much for the suggestion!"

          Show
          voellm Eckhard Völlm added a comment - - edited Jenkins version 1.6.43 xUnit version 1.99 java version "1.7.0_91" OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1) OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode) Linux slave 4.2.0-19-lowlatency #23~14.04.1-Ubuntu SMP PREEMPT Thu Nov 12 13:19:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Here a thread where Java threads also were hanging at "java.util.zip.ZipFile.getEntry": https://github.com/aws/aws-sdk-java/issues/238 same as we observe here. They also found a solution, IMHO: "And yes, using this system property gets rid of the thread blocks :+1: Thank you so much for the suggestion!"
          Hide
          francisu Francis Upton added a comment -

          Does not appear to be an ec2 plugin issue.

          Show
          francisu Francis Upton added a comment - Does not appear to be an ec2 plugin issue.
          Hide
          voellm Eckhard Völlm added a comment - - edited

          Yes, we do not have ec2 plugin in use, i can confirm that.
          And we decided to try an upgrade of Java version now, using oracle 1.8.0_66-b17 since last week,
          Since we did that, there was no hanging, but time is too short to say something, not many test were running since snowzilla hit eastcost.

          Propose to switch Subject, to "Build is hanging in xUnit plugin processing"

          Show
          voellm Eckhard Völlm added a comment - - edited Yes, we do not have ec2 plugin in use, i can confirm that. And we decided to try an upgrade of Java version now, using oracle 1.8.0_66-b17 since last week, Since we did that, there was no hanging, but time is too short to say something, not many test were running since snowzilla hit eastcost. Propose to switch Subject, to "Build is hanging in xUnit plugin processing"
          Hide
          nfalco Nikolas Falco added a comment -

          Please try with lastest version of Jenkins and the plugin, i could not reproduce this scenario also if detailed with thread dump.

          Show
          nfalco Nikolas Falco added a comment - Please try with lastest version of Jenkins and the plugin, i could not reproduce this scenario also if detailed with thread dump.

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              tmgw165 Hiroko Tamagawa
            • Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: