Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28175

config change deadlock Jenkins when pircx.shutdown() is invoked

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: ircbot-plugin
    • Labels:
      None
    • Environment:
      jenkins 1.596.2
      ircbot-plugin 2.26
    • Similar Issues:

      Description

      We have recently upgraded the IRC plugin from 2.25 to 2.26. On configuration change, the ircbot plugin invokes PircBotX.shutdown(). For some reason it never finish and the conf change is stalled.

      A side effect is that jobs sending notifications ends up being blocked waiting for an instance of the irc connection provider. The only fix is to restart Jenkins entirely.

      Our bug has a few more explanations https://phabricator.wikimedia.org/T96183 and a full thread dump attached https://phabricator.wikimedia.org/P584

      Here are the blocked threads:

      Two jobs are blocked:

      "Executor #2 for integration-slave-trusty-1016 : executing browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce #234" prio=5 BLOCKED
      "Executor #1 for integration-slave-trusty-1012 : executing browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce #494" prio=5 BLOCKED
      
      	hudson.plugins.ircbot.v2.IRCConnectionProvider.getInstance(IRCConnectionProvider.java:14)
      	hudson.plugins.ircbot.IrcPublisher.getIMConnection(IrcPublisher.java:102)
      	hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:374)
      	hudson.plugins.im.IMPublisher.notifyChatsOnBuildEnd(IMPublisher.java:585)
      	hudson.plugins.im.IMPublisher.notifyOnBuildEnd(IMPublisher.java:304)
      	hudson.plugins.im.IMPublisher.perform(IMPublisher.java:291)
              ...
      

      A configuration submit change is blocked as well:

      "Handling POST /ci/configSubmit from X.X.X.X : RequestHandlerThread[#1683]" daemon prio=5 WAITING
      	sun.misc.Unsafe.park(Native Method)
      	java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
      	java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
      	java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
      	java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
      	org.pircbotx.Channel.getMode(Channel.java:127)
      	org.pircbotx.Channel.getModeArgument(Channel.java:182)
      	org.pircbotx.Channel.getChannelKey(Channel.java:239)
      	org.pircbotx.PircBotX.shutdown(PircBotX.java:2872)
      	hudson.plugins.ircbot.v2.IRCConnection.close(IRCConnection.java:102)
      	hudson.plugins.im.IMConnectionProvider.releaseConnection(IMConnectionProvider.java:92)
      	hudson.plugins.ircbot.v2.IRCConnectionProvider.setDesc(IRCConnectionProvider.java:19)
      	hudson.plugins.ircbot.IrcPublisher$DescriptorImpl.configure(IrcPublisher.java:336)
      	jenkins.model.Jenkins.configureDescriptor(Jenkins.java:2915)
      	jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2878)
              ...
      

      Some other related threads:

      "JenkinsIsBusyListener-thread" daemon prio=5 BLOCKED
       hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:83)
       hudson.plugins.im.JenkinsIsBusyListener.setStatus(JenkinsIsBusyListener.java:118)
       hudson.plugins.im.JenkinsIsBusyListener.updateIMStatus(JenkinsIsBusyListener.java:109)
       hudson.plugins.im.JenkinsIsBusyListener.access$000(JenkinsIsBusyListener.java:20)
       hudson.plugins.im.JenkinsIsBusyListener$3.run(JenkinsIsBusyListener.java:98)
       java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
       java.util.concurrent.FutureTask.run(FutureTask.java:262)
       java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
       java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       java.lang.Thread.run(Thread.java:745)
      
      "IM-Reconnector-Thread" daemon prio=5 BLOCKED
       hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:175)
       java.lang.Thread.run(Thread.java:745)
      

      Looking at the git changelog I noticed https://github.com/jenkinsci/ircbot-plugin/commit/98b0105a743d062abf957c285cdded06fedd3fa3 which changes in IRCConnection.close:

      - this.pircConnection.disconnect();
      + this.pircConnection.shutdown(true);
      

      Which was done to fix a leak (JENKINS-25349).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kutzi kutzi
                Reporter:
                hashar Antoine Musso
              • Votes:
                3 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: