Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32926

Recurring "Too many health check executions queued" warning in Jenkins log

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Not A Defect
    • Component/s: metrics-plugin
    • Labels:
      None
    • Environment:
      Jenkins version 1.647, metrics plugin 3.1.2.5
    • Similar Issues:

      Description

      I've recently started seeing this warning popping up in my Jenkins system log:

      Feb 12, 2016 3:26:43 PM WARNING jenkins.metrics.util.HealthChecksThreadPool dropOldestInQueue

      Too many health check executions queued, dropping oldest one. This may mean some health checks are taking too long to execute: jenkins.metrics.util.HealthChecksThreadPool@3cf11acf[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 503], queue size=1, health checks=[disk-space, plugins, temporary-space, thread-deadlock] (4) [java.util.concurrent.FutureTask@8afd366]

      Feb 12, 2016 3:26:43 PM WARNING com.codahale.metrics.health.HealthCheckRegistry runHealthChecks

      Error executing health check thread-deadlock
      java.util.concurrent.CancellationException
      at java.util.concurrent.FutureTask.report(FutureTask.java:121)
      at java.util.concurrent.FutureTask.get(FutureTask.java:188)
      at com.codahale.metrics.health.HealthCheckRegistry.runHealthChecks(HealthCheckRegistry.java:102)
      at jenkins.metrics.api.Metrics$HealthChecker.execute(Metrics.java:592)
      at jenkins.metrics.api.Metrics$HealthChecker.access$500(Metrics.java:402)
      at jenkins.metrics.api.Metrics$HealthChecker$3.run(Metrics.java:532)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      I only noticed it today but I look at the logs every couple of days at least, and I haven't seen it until now. A Google search doesn't return any other instances of it appearing, so beyond a quick glance at the source code I can't say for sure what's causing it. As a one time thing I'd write it off but I'm seeing it repeated every couple of minutes following my last reboot. Should I be concerned about this?

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          I consistently see a similar message in my 1.642.1 installation, and in the Docker based configurations that I use.

          I see:

          Feb 12, 2016 9:52:43 PM jenkins.metrics.util.HealthChecksThreadPool dropOldestInQueue
          WARNING: Too many health check executions queued, dropping oldest one. This may mean some health checks are taking too long to execute: jenkins.metrics.util.HealthChecksThreadPool@7a7d5210[Running, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 2], queue size=1, health checks=[disk-space, plugins, temporary-space, thread-deadlock] (4) [java.util.concurrent.FutureTask@74bab2f7]
          Feb 12, 2016 9:52:43 PM com.codahale.metrics.health.HealthCheckRegistry runHealthChecks
          WARNING: Error executing health check temporary-space
          java.util.concurrent.CancellationException
                  at java.util.concurrent.FutureTask.report(FutureTask.java:121)
                  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
                  at com.codahale.metrics.health.HealthCheckRegistry.runHealthChecks(HealthCheckRegistry.java:102)
                  at jenkins.metrics.api.Metrics$HealthChecker.execute(Metrics.java:592)
                  at jenkins.metrics.api.Metrics$HealthChecker.access$500(Metrics.java:402)
                  at jenkins.metrics.api.Metrics$HealthChecker$3.run(Metrics.java:532)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                  at java.lang.Thread.run(Thread.java:745)
          
          Show
          markewaite Mark Waite added a comment - I consistently see a similar message in my 1.642.1 installation, and in the Docker based configurations that I use. I see: Feb 12, 2016 9:52:43 PM jenkins.metrics.util.HealthChecksThreadPool dropOldestInQueue WARNING: Too many health check executions queued, dropping oldest one. This may mean some health checks are taking too long to execute: jenkins.metrics.util.HealthChecksThreadPool@7a7d5210[Running, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 2], queue size=1, health checks=[disk-space, plugins, temporary-space, thread-deadlock] (4) [java.util.concurrent.FutureTask@74bab2f7] Feb 12, 2016 9:52:43 PM com.codahale.metrics.health.HealthCheckRegistry runHealthChecks WARNING: Error executing health check temporary-space java.util.concurrent.CancellationException at java.util.concurrent.FutureTask.report(FutureTask.java:121) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.codahale.metrics.health.HealthCheckRegistry.runHealthChecks(HealthCheckRegistry.java:102) at jenkins.metrics.api.Metrics$HealthChecker.execute(Metrics.java:592) at jenkins.metrics.api.Metrics$HealthChecker.access$500(Metrics.java:402) at jenkins.metrics.api.Metrics$HealthChecker$3.run(Metrics.java:532) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
          Hide
          hamster21 Hans Meyer added a comment - - edited

          This also occurs on Windows Server 2012R2 with Jenkins version 1.642.1 and seemingly prevents the proper startup of Jenkins.

          Metrics Plugin has version 3.1.2.4

          Update
          Jenkins is starting correctly, seems I have issues with access from the outside of the host running Jenkins

          I see:

          Feb 23, 2016 10:14:10 AM jenkins.metrics.util.HealthChecksThreadPool dropOldestInQueue
          WARNING: Too many health check executions queued, dropping oldest one. This may mean some health checks are taking too long to execute: jenkins.metrics.util.HealthChecksThreadPool@17e54be[Running, pool size = 4, active threads = 4, queued tasks = 1, completed tasks = 12], queue size=1, health checks=[disk-space, plugins, temporary-space, thread-deadlock] (4)
          Feb 23, 2016 10:14:10 AM com.codahale.metrics.health.HealthCheckRegistry runHealthChecks
          WARNING: Error executing health check temporary-space
          java.util.concurrent.CancellationException
          	at java.util.concurrent.FutureTask.report(Unknown Source)
          	at java.util.concurrent.FutureTask.get(Unknown Source)
          	at com.codahale.metrics.health.HealthCheckRegistry.runHealthChecks(HealthCheckRegistry.java:102)
          	at jenkins.metrics.api.Metrics$HealthChecker.execute(Metrics.java:592)
          	at jenkins.metrics.api.Metrics$HealthChecker.access$500(Metrics.java:402)
          	at jenkins.metrics.api.Metrics$HealthChecker$3.run(Metrics.java:532)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          	at java.util.concurrent.FutureTask.run(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          	at java.lang.Thread.run(Unknown Source)
          
          Show
          hamster21 Hans Meyer added a comment - - edited This also occurs on Windows Server 2012R2 with Jenkins version 1.642.1 and seemingly prevents the proper startup of Jenkins. Metrics Plugin has version 3.1.2.4 Update Jenkins is starting correctly, seems I have issues with access from the outside of the host running Jenkins I see: Feb 23, 2016 10:14:10 AM jenkins.metrics.util.HealthChecksThreadPool dropOldestInQueue WARNING: Too many health check executions queued, dropping oldest one. This may mean some health checks are taking too long to execute: jenkins.metrics.util.HealthChecksThreadPool@17e54be[Running, pool size = 4, active threads = 4, queued tasks = 1, completed tasks = 12], queue size=1, health checks=[disk-space, plugins, temporary-space, thread-deadlock] (4) Feb 23, 2016 10:14:10 AM com.codahale.metrics.health.HealthCheckRegistry runHealthChecks WARNING: Error executing health check temporary-space java.util.concurrent.CancellationException at java.util.concurrent.FutureTask.report(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at com.codahale.metrics.health.HealthCheckRegistry.runHealthChecks(HealthCheckRegistry.java:102) at jenkins.metrics.api.Metrics$HealthChecker.execute(Metrics.java:592) at jenkins.metrics.api.Metrics$HealthChecker.access$500(Metrics.java:402) at jenkins.metrics.api.Metrics$HealthChecker$3.run(Metrics.java:532) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
          Hide
          stephenconnolly Stephen Connolly added a comment -

          This is typically indicative of some problem in your Jenkins instance. Did you look at JENKINS_HOME/logs/health-checker.log to see what was reporting?

          (I have added a check to prevent the healthchecks running before Jenkins completes startup in case that was causing the effects Hans Meyer was reporting)

          Show
          stephenconnolly Stephen Connolly added a comment - This is typically indicative of some problem in your Jenkins instance. Did you look at JENKINS_HOME/logs/health-checker.log to see what was reporting? (I have added a check to prevent the healthchecks running before Jenkins completes startup in case that was causing the effects Hans Meyer was reporting)

            People

            • Assignee:
              Unassigned
              Reporter:
              nlykins Nick Lykins
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: