-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Windows 2008 R2 64bit. 20Gb RAM, virtualised
Through the Melody monitoring plugin and using VisualVM to inspect heap dumps, we have determined that a critical memory leak on our Jenkins master node appears to caused by blocked threads handling requests to the XML API from the Wall Display plugin:
Over a period of 2--3 hours the VM will consume 5Gb of heap space and have 400+ blocked threads.
For example, we have 23550 byte[] instances occupying 1,829,012,514 bytes (84.3%) of heap space in a total heap of 2,170,755,827 in a sample heap dump.
Handling GET /job/Deploy%20FastTrack/api/xml : RequestHandlerThread[#11] hudson.model.User.addProperty(User.java:215) hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:251) sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66) org.kohsuke.stapler.export.Property.writeTo(Property.java:113) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:179) org.kohsuke.stapler.export.Property.writeValue(Property.java:231) org.kohsuke.stapler.export.Property.writeValue(Property.java:168) org.kohsuke.stapler.export.Property.writeValue(Property.java:136) org.kohsuke.stapler.export.Property.writeTo(Property.java:113) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:179) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:176) org.kohsuke.stapler.export.Property.writeValue(Property.java:231) org.kohsuke.stapler.export.Property.writeValue(Property.java:136) org.kohsuke.stapler.export.Property.writeTo(Property.java:113) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:179) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:176) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:176) org.kohsuke.stapler.export.Property.writeValue(Property.java:231) org.kohsuke.stapler.export.Property.writeValue(Property.java:181) org.kohsuke.stapler.export.Property.writeValue(Property.java:136) org.kohsuke.stapler.export.Property.writeTo(Property.java:113) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:179) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:176) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:176) org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:176) org.kohsuke.stapler.export.Model.writeTo(Model.java:152) org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:199) hudson.model.Api.doXml(Api.java:87) sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:282) org.kohsuke.stapler.Function.bindAndInvoke(Function.java:149) org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:88) org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:104) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:196) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:234) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561) org.kohsuke.stapler.Stapler.invoke(Stapler.java:646) org.kohsuke.stapler.Stapler.invoke(Stapler.java:477) org.kohsuke.stapler.Stapler.service(Stapler.java:159) javax.servlet.http.HttpServlet.service(HttpServlet.java:45) winstone.ServletConfiguration.execute(ServletConfiguration.java:249) winstone.RequestDispatcher.forward(RequestDispatcher.java:335) winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:157) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:131) net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:83) org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) winstone.FilterConfiguration.execute(FilterConfiguration.java:195) winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) winstone.FilterConfiguration.execute(FilterConfiguration.java:195) winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) winstone.FilterConfiguration.execute(FilterConfiguration.java:195) winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) winstone.FilterConfiguration.execute(FilterConfiguration.java:195) winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368) winstone.RequestDispatcher.forward(RequestDispatcher.java:333) winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244) winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) java.lang.Thread.run(Thread.java:662)
