Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19215

Consistent daily morning deadlocks

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Environment:
      Linux jenkins 2.6.32-43-server #97-Ubuntu SMP Wed Sep 5 16:56:41 UTC 2012 x86_64 GNU/Linux
    • Similar Issues:

      Description

      For more than a week, we have experienced deadlocks every morning.

      We suspect it occurs when someone hits the web ui for the first time that day. We do have nightly rebuilds, but the deadlock doesn't appear to occur until much later in the morning. The deadlocks don't occur at the same time, but they are consistent every morning.

      We have tried a couple things so far:
      Upgrade from 1.520 to 1.525
      Replace Winstone with Jetty (suggested by nanoyme #jenkins@freenode)

      I've attached the jstack traces for when jenkins gets wedged and after a fresh restart as well as the plugins list.

      Would like some pointers on what to diagnose next. We were hoping replacing winstone with jetty was going to work.

        Attachments

          Activity

          Hide
          milki milki mlk added a comment - - edited

          Possibly related, looks like the irc plugin actually killed the winstone server. See attached irbot.kills.winstone log

          Show
          milki milki mlk added a comment - - edited Possibly related, looks like the irc plugin actually killed the winstone server. See attached irbot.kills.winstone log
          Hide
          milki milki mlk added a comment -

          With the Java Melody monitoring plugin, we have concluded that memory is not an issue immediately before Jenkins freezes.

          Show
          milki milki mlk added a comment - With the Java Melody monitoring plugin, we have concluded that memory is not an issue immediately before Jenkins freezes.
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          In https://issues.jenkins-ci.org/browse/JENKINS-20051 winstone also goes down, but the issue has been caused by Outage of File Descriptors and LDAP issues w/o any deadlocks.

          Seems that automatic failover of a web container could be useful.

          Show
          oleg_nenashev Oleg Nenashev added a comment - In https://issues.jenkins-ci.org/browse/JENKINS-20051 winstone also goes down, but the issue has been caused by Outage of File Descriptors and LDAP issues w/o any deadlocks. Seems that automatic failover of a web container could be useful.
          Hide
          danielbeck Daniel Beck added a comment -

          Is this still a problem for you?

          Show
          danielbeck Daniel Beck added a comment - Is this still a problem for you?
          Hide
          milki milki mlk added a comment -

          Upgrading Java to 7u55, increasing PermGen to 128m, and adding a ton more ram seems to do the trick.

          The irc plugin can still kill winstone and there are still File Descriptor leaks. But Jenkins no longer consistently deadlocks at least.

          Show
          milki milki mlk added a comment - Upgrading Java to 7u55, increasing PermGen to 128m, and adding a ton more ram seems to do the trick. The irc plugin can still kill winstone and there are still File Descriptor leaks. But Jenkins no longer consistently deadlocks at least.

            People

            • Assignee:
              Unassigned
              Reporter:
              milki milki mlk
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: