Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33050

Monitoring stopped working - exception collecting data

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: monitoring-plugin
    • Labels:
      None
    • Environment:
      RedHat Enterprise Linux 6.1
      JDK 1.8.0_65
      Jenkins 1.625 LTS
    • Similar Issues:

      Description

      I'm encountering an issue where monitoring stops working. Seeing this javamelody exception:

      WARNING net.bull.javamelody.JavaLogger#warn: exception while collecting data: java.lang.IllegalArgumentException
      java.lang.IllegalArgumentException
      at java.nio.Buffer.position(Buffer.java:244)
      at net.bull.javamelody.RrdNioBackend.read(RrdNioBackend.java:147)
      at org.jrobin.core.RrdBackend.readInt(RrdBackend.java:224)
      at org.jrobin.core.RrdPrimitive.readInt(RrdPrimitive.java:63)
      at org.jrobin.core.RrdInt.get(RrdInt.java:55)
      at org.jrobin.core.Archive.<init>(Archive.java:69)
      at org.jrobin.core.RrdDb.<init>(RrdDb.java:244)
      at org.jrobin.core.RrdDb.<init>(RrdDb.java:203)
      at org.jrobin.core.RrdDb.<init>(RrdDb.java:266)
      at org.jrobin.core.RrdDbPool.requestRrdDb(RrdDbPool.java:103)
      at net.bull.javamelody.JRobin.addValue(JRobin.java:334)
      at net.bull.javamelody.Collector.collectJRobinValues(Collector.java:491)
      at net.bull.javamelody.Collector.collectJavaInformations(Collector.java:382)
      at net.bull.javamelody.Collector.collect(Collector.java:320)
      at net.bull.javamelody.Collector.collectWithoutErrors(Collector.java:306)
      at net.bull.javamelody.Collector.collectLocalContextWithoutErrors(Collector.java:295)
      at net.bull.javamelody.FilterContext$CollectTimerTask.run(FilterContext.java:56)
      at java.util.TimerThread.mainLoop(Timer.java:555)
      at java.util.TimerThread.run(Timer.java:505)

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Hi Dan Tranter
            Please specify the plugin version.

            It seems like a missing bounds check in https://github.com/javamelody/javamelody/blob/master/javamelody-core/src/main/java/net/bull/javamelody/RrdNioBackend.java#L146-L149 . The underlying buffer management code is http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/nio/Buffer.java#235 , so there's only one way to get this exception.

            Show
            oleg_nenashev Oleg Nenashev added a comment - Hi Dan Tranter Please specify the plugin version. It seems like a missing bounds check in https://github.com/javamelody/javamelody/blob/master/javamelody-core/src/main/java/net/bull/javamelody/RrdNioBackend.java#L146-L149 . The underlying buffer management code is http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/nio/Buffer.java#235 , so there's only one way to get this exception.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I've assigned the issue to Emeric

            Show
            oleg_nenashev Oleg Nenashev added a comment - I've assigned the issue to Emeric
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Follow-up issue in JavaMelody project: https://github.com/javamelody/javamelody/issues/533

            Show
            oleg_nenashev Oleg Nenashev added a comment - Follow-up issue in JavaMelody project: https://github.com/javamelody/javamelody/issues/533
            Hide
            evernat evernat added a comment -

            Yes, what is the plugin version?

            Show
            evernat evernat added a comment - Yes, what is the plugin version?
            Hide
            evernat evernat added a comment -

            And also can you copy here the "debugging logs" which are displayed at the botton of the /monitoring page? Thanks

            Show
            evernat evernat added a comment - And also can you copy here the "debugging logs" which are displayed at the botton of the /monitoring page? Thanks
            Hide
            jgbiii James Brown added a comment -

            We'll have to get back to you on the logs, but the version was 1.57 at the time the problem was initially raised. It has now been updated to 1.58.

            Show
            jgbiii James Brown added a comment - We'll have to get back to you on the logs, but the version was 1.57 at the time the problem was initially raised. It has now been updated to 1.58.
            Hide
            dtranter Dan Tranter added a comment -

            Upon update from 1.57 -> 1.58 and restart, monitoring graphs are now displaying again.. However I'm still seeing warnings/exception collecting data, unsure if that's related to previous failure. Unfortunately the current debugging log no longer contains entries from the time of failure:


            Last collect time: 861 ms
            Display time: 678 ms
            Memory overhead estimate: < 5 Mb
            Disk usage: 39 Mb Purge the obsolete files

            JavaMelody 1.58.0
            Debugging logs

            Only the last 50 messages are displayed
            Fri Feb 19 20:07:12 CST 2016 DEBUG Server: jetty/winstone-2.8
            Fri Feb 19 20:07:12 CST 2016 DEBUG Webapp context:
            Fri Feb 19 20:07:12 CST 2016 DEBUG JavaMelody version: 1.58.0
            Fri Feb 19 20:07:12 CST 2016 DEBUG JavaMelody classes loaded from: file:/auto/ncs-jenkins/ncs-jenkins/jenkins/plugins/monitoring/WEB-INF/lib/javamelody-core-1.58.0.jar
            Fri Feb 19 20:07:12 CST 2016 DEBUG Host: host5@123.123.123.123
            Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: storage-directory=//auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring
            Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: http-transform-pattern=/\d+/|/site/.|avadoc/.|/ws/.|obertura/.|estReport/.|iolations/file/.|/user/.|/static/\w/|/adjuncts/\w+/|/bound/[\w\-]+
            Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: custom-reports=Jenkins Info,About Monitoring
            Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: no-database=true
            Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: gzip-compression-disabled=true
            Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: system-actions-enabled=true
            Fri Feb 19 20:07:12 CST 2016 DEBUG log listeners initialized
            Fri Feb 19 20:07:12 CST 2016 DEBUG counters initialized
            Fri Feb 19 20:07:12 CST 2016 DEBUG counters data read from files in /auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring/_host5
            Fri Feb 19 20:07:12 CST 2016 DEBUG collect task scheduled every 60s
            Fri Feb 19 20:07:13 CST 2016 DEBUG first collect of data done
            Fri Feb 19 20:07:13 CST 2016 DEBUG JavaMelody filter init done in 308 ms
            Fri Feb 19 20:07:13 CST 2016 DEBUG counters data read from files in /auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring/nodes
            Sat Feb 20 00:00:29 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 14327 KB
            Sat Feb 20 00:00:32 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 134286 KB
            Sun Feb 21 00:00:35 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 27464 KB
            Sun Feb 21 00:00:39 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 130530 KB
            Sun Feb 21 07:02:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:03:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:04:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:05:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:06:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:07:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:08:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:09:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:10:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:11:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:12:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 07:13:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:21:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:22:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:23:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:24:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:25:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:26:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:27:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:28:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:29:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:30:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:31:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:32:36 CST 2016 WARN exception while collecting data
            Sun Feb 21 08:33:36 CST 2016 WARN exception while collecting data
            Mon Feb 22 00:00:36 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 39034 KB
            Mon Feb 22 00:00:43 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 130487 KB
            Mon Feb 22 03:37:38 CST 2016 WARN exception while collecting data

            Thanks,

            Show
            dtranter Dan Tranter added a comment - Upon update from 1.57 -> 1.58 and restart, monitoring graphs are now displaying again.. However I'm still seeing warnings/exception collecting data, unsure if that's related to previous failure. Unfortunately the current debugging log no longer contains entries from the time of failure: Last collect time: 861 ms Display time: 678 ms Memory overhead estimate: < 5 Mb Disk usage: 39 Mb Purge the obsolete files JavaMelody 1.58.0 Debugging logs Only the last 50 messages are displayed Fri Feb 19 20:07:12 CST 2016 DEBUG Server: jetty/winstone-2.8 Fri Feb 19 20:07:12 CST 2016 DEBUG Webapp context: Fri Feb 19 20:07:12 CST 2016 DEBUG JavaMelody version: 1.58.0 Fri Feb 19 20:07:12 CST 2016 DEBUG JavaMelody classes loaded from: file:/auto/ncs-jenkins/ncs-jenkins/jenkins/plugins/monitoring/WEB-INF/lib/javamelody-core-1.58.0.jar Fri Feb 19 20:07:12 CST 2016 DEBUG Host: host5@123.123.123.123 Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: storage-directory=//auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: http-transform-pattern=/\d+/|/site/. |avadoc/. |/ws/. |obertura/. |estReport/. |iolations/file/. |/user/. |/static/\w /|/adjuncts/\w+/|/bound/ [\w\-] + Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: custom-reports=Jenkins Info,About Monitoring Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: no-database=true Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: gzip-compression-disabled=true Fri Feb 19 20:07:12 CST 2016 DEBUG parameter defined: system-actions-enabled=true Fri Feb 19 20:07:12 CST 2016 DEBUG log listeners initialized Fri Feb 19 20:07:12 CST 2016 DEBUG counters initialized Fri Feb 19 20:07:12 CST 2016 DEBUG counters data read from files in /auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring/_host5 Fri Feb 19 20:07:12 CST 2016 DEBUG collect task scheduled every 60s Fri Feb 19 20:07:13 CST 2016 DEBUG first collect of data done Fri Feb 19 20:07:13 CST 2016 DEBUG JavaMelody filter init done in 308 ms Fri Feb 19 20:07:13 CST 2016 DEBUG counters data read from files in /auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring/nodes Sat Feb 20 00:00:29 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 14327 KB Sat Feb 20 00:00:32 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 134286 KB Sun Feb 21 00:00:35 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 27464 KB Sun Feb 21 00:00:39 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 130530 KB Sun Feb 21 07:02:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:03:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:04:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:05:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:06:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:07:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:08:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:09:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:10:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:11:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:12:36 CST 2016 WARN exception while collecting data Sun Feb 21 07:13:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:21:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:22:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:23:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:24:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:25:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:26:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:27:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:28:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:29:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:30:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:31:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:32:36 CST 2016 WARN exception while collecting data Sun Feb 21 08:33:36 CST 2016 WARN exception while collecting data Mon Feb 22 00:00:36 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 39034 KB Mon Feb 22 00:00:43 CST 2016 DEBUG Obsolete files deleted. JavaMelody disk usage: 130487 KB Mon Feb 22 03:37:38 CST 2016 WARN exception while collecting data Thanks,
            Hide
            evernat evernat added a comment -

            I think that a file in the storage directories (/auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring/**) was corrupted somewhere in the middle of the file. And that javamelody / jrobin can't read it anymore.

            You can try to find if ".rrd" file(s) in the storage directories do not have the same size as the others and to delete it.
            But what is strange is that corruption seems intermittent: from 7:02 to 7:13 and from 8:21 to 8:33 but nothing the day before.

            Anyway, I have added a catch in the code to auto-reset the file in this case. (2eff216)
            And 1.59 release is planned soon including that fix.

            Show
            evernat evernat added a comment - I think that a file in the storage directories (/auto/ncs-jenkins/ncs-jenkins/jenkins/monitoring/**) was corrupted somewhere in the middle of the file. And that javamelody / jrobin can't read it anymore. You can try to find if ".rrd" file(s) in the storage directories do not have the same size as the others and to delete it. But what is strange is that corruption seems intermittent: from 7:02 to 7:13 and from 8:21 to 8:33 but nothing the day before. Anyway, I have added a catch in the code to auto-reset the file in this case. ( 2eff216 ) And 1.59 release is planned soon including that fix.

              People

              • Assignee:
                evernat evernat
                Reporter:
                dtranter Dan Tranter
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: