Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12672

Jobs take a long time to complete - get stuck sending success/failure email

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: perforce-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.450
      Perforce plugin 1.3.7
      Red Hat Enterprise Linux 4.5
    • Similar Issues:

      Description

      We've noticed that sometimes our jobs get in a state where they're taking a long time to complete. Investigation reveals that the maven build has completing successfully, and it's "stuck" trying to send a success email.

      Left alone, the build will eventually complete but a 10 minute build may take several hours to succeed and blocks other jobs since the executors are busy. Manual monitoring and intervention is usually required to free up the jobs.

      Here's an example thread taken from a Jenkins thread dump that shows the "stuck" job. It appears to be processing the change list to retrieve the checkin user(s) that checked in code. Why is this taking so long?

      The exact stack changes with each thread dump, but it is consistent up to PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111).

      "Executor #2 for master : executing markview-7.0.x_document-server #103" prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000]
      java.lang.Thread.State: RUNNABLE
      at java.util.ArrayList.size(ArrayList.java:177)
      at org.dom4j.tree.BackedList.<init>(BackedList.java:36)
      at org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373)
      at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392)
      at org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428)
      at org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233)
      at org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98)
      at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180)
      at org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140)
      at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102)
      at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674)
      at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213)
      at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234)
      at org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159)
      at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185)
      at hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111)
      at hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18)
      at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808)
      at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782)
      at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356)
      at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366)
      at hudson.model.User.getProjects(User.java:402)
      at hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38)
      at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100)
      at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514)
      at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331)
      at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251)
      at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243)
      at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675)
      at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022)
      at hudson.model.Run.run(Run.java:1453)
      at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:238)

        Attachments

          Activity

          Hide
          sharon_xia sharon xia added a comment -

          What's worse is: This build cannot be deleted as it says java.exe is using the log file.

          Show
          sharon_xia sharon xia added a comment - What's worse is: This build cannot be deleted as it says java.exe is using the log file.
          Hide
          rpetti Rob Petti added a comment -

          What perforce plugin version are you using?

          Show
          rpetti Rob Petti added a comment - What perforce plugin version are you using?
          Hide
          sharon_xia sharon xia added a comment -

          perforce plugin? We don't use perforce plugin.

          We were using email-ext 2.14 plugin. I just updated it to the latest 2.37.2.2. Hope this could be resolved.

          Show
          sharon_xia sharon xia added a comment - perforce plugin? We don't use perforce plugin. We were using email-ext 2.14 plugin. I just updated it to the latest 2.37.2.2. Hope this could be resolved.
          Hide
          oleg_nenashev Oleg Nenashev added a comment - - edited

          I suppose that the high-priority handling of user-specified e-mail address (JENKINS-19433, https://github.com/jenkinsci/mailer-plugin/pull/12) will resolve performance issues of resolvers for users, who have registered in Jenkins.

          Show
          oleg_nenashev Oleg Nenashev added a comment - - edited I suppose that the high-priority handling of user-specified e-mail address ( JENKINS-19433 , https://github.com/jenkinsci/mailer-plugin/pull/12 ) will resolve performance issues of resolvers for users, who have registered in Jenkins.
          Hide
          rpetti Rob Petti added a comment -

          Please read tickets before reopening them... This issue is specific to the perforce plugin.

          Show
          rpetti Rob Petti added a comment - Please read tickets before reopening them... This issue is specific to the perforce plugin.

            People

            • Assignee:
              rpetti Rob Petti
              Reporter:
              epalazzo Ed Palazzo
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: