Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41100

Memory leak in web interface (Javascript)

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      Tested version: jenkins 2.39
      Browser: Google Chrome

      Issue:

      If I open a browser tab on Jenkins and let it open, the memory consumed by the tab grows up to 2GB (and then I need to close it to be able to work). Consumed memory can be accessed by typing `Shift + Esc` in Google Chrome.

      When I just open the home view, it starts around 52 kB

      I suppose there is a memory leak in the JS code.

      If I launch the Memory profiler in Chrome, it appears that objects are created every 5s and they're never released (See attachment).

      The memory leak seems related to event handling (see 2nd attachment)

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Weird. JENKINS-10912 was supposed to be resolved in 2.23. Could it be related to that anyway?

          Show
          danielbeck Daniel Beck added a comment - Weird. JENKINS-10912 was supposed to be resolved in 2.23. Could it be related to that anyway?
          Hide
          daniel_c_686 Daniel Carrington added a comment - - edited

          Sorry for editing this comment; I needed to fix some typos especially in my Jenkins version number.

          I have this issue too on Jenkins 2.121.1, which I installed in the last week.
          The symptoms are the same: increased memory usage at regular 5s intervals.
          This only appears to occur on pages with the "Build Queue" and "Build Executor Status" widgets.

          I have a lot of executors in my Jenkins instance: between 200 and 210. There's quite a bit of build activity because we have a large number of developers, so I see very quick memory growth.

          I did a memory profile in Chrome of type "Allocation instrumentation on timeline" to see what was leaking.

          I saw a lot of "(closure)", "Detached HTMLTableCellElement" and "HTMLAnchorElement".

          Looking at the HTMLTableCellElements first, I saw some formatted job names. I worked backwards a bit to find that the were all inside "Detached HTMLTableRowElements".
          Here's a sample of the outerHTML for one of those (item names changed, and whitespace/indentation added for clarity):

          <tr>
          <td class="pane" align="right" style="vertical-align: top">3</td>
          <td class="pane"><div style="white-space: normal">
            <a href="/job/my-folder/job/my-jobname/">My Folder » My Job Name</a>
            <table tooltip="Started 20 hr ago<br> Estimated remaining time: 10 hr" style="cursor:pointer" href="/job/my-folder/job/my-jobname/44/console" class="progress-bar " title="Started 20 hr ago<br> Estimated remaining time: 10 hr">
              <tbody><tr>
              <td style="width:66%;" class="progress-bar-done"></td>
              <td style="width:34%" class="progress-bar-left"></td>
            </tr></tbody></table>
          </div></td>
          <td class="pane">
            <a href="/job/my-folder/job/my-jobname/44/" class="model-link inside"><wbr>#44</a>
          </td>
          <td class="pane" align="center" valign="middle">
            <a onclick="if(confirm(&quot;Are you sure you want to abort My Folder » My Job Name #44?&quot;))new Ajax.Request(&quot;/computer/node%20name%20anonymized/executors/2/stop&quot;); return false;" href="/computer/node%20name%20anonymized/executors/2/stop" class="stop-button-link"><img src="/static/175f9ede/images/16x16/stop.png" alt="terminate this build" style="width: 16px; height: 16px; " class="icon-stop icon-sm"></a></td></tr>

          Using the "collect garbage" option in Chrome did not remove these objects.
          I looked at the (closure) objects to see what they were using the "Retainers" panel in the Dev Tools.

          These closures are referenced from "Detached EventListener" objects, referenced by "Detached InternalNode" objects, referenced by "Detached HTMLAnchorElement" objects.

          I did one more thing: "Break on subtree modification" for div#side-panel in the Chrome dev tools.

          That brought me to hudson-behavior.js, "p.replaceChild(node, hist)", which offers some insight into why there are a tree of detached elements: some element inside the tree may have an EventListener that is still referenced somewhere.

          I tried the Firefox Dev tools next (as well as to confirm that the memory leak also occurs in Firefox.

          Tracing call stacks on memory allocations showed that there's an issue with the call stack starting at the very next function call, "Behaviour.applySubtree(node)":

          The _createResponder function in prototype.js seems to be the culprit.
          It seems to always add the passed element to its CACHE array. It never cleans up the CACHE as far as I can tell, unless you unload the page in IE.

          Because the CACHE contains a persistent reference to the HTMLAnchorElements which get event handlers, all of the anchor elements created in the Build Executor Window are persisted. That adds up fast.

          I hope that makes this bug fixable - I had a window where these were visible and it used up 3GB of memory after a few hours.

          Show
          daniel_c_686 Daniel Carrington added a comment - - edited Sorry for editing this comment; I needed to fix some typos especially in my Jenkins version number. I have this issue too on Jenkins 2.121.1, which I installed in the last week. The symptoms are the same: increased memory usage at regular 5s intervals. This only appears to occur on pages with the "Build Queue" and "Build Executor Status" widgets. I have a lot of executors in my Jenkins instance: between 200 and 210. There's quite a bit of build activity because we have a large number of developers, so I see very quick memory growth. I did a memory profile in Chrome of type "Allocation instrumentation on timeline" to see what was leaking. I saw a lot of "(closure)", "Detached HTMLTableCellElement" and "HTMLAnchorElement". Looking at the HTMLTableCellElements first, I saw some formatted job names. I worked backwards a bit to find that the were all inside "Detached HTMLTableRowElements". Here's a sample of the outerHTML for one of those (item names changed, and whitespace/indentation added for clarity): <tr> <td class= "pane" align= "right" style= "vertical-align: top" >3</td> <td class= "pane" ><div style= "white-space: normal" > <a href= "/job/my-folder/job/my-jobname/" >My Folder » My Job Name</a> <table tooltip= "Started 20 hr ago<br> Estimated remaining time: 10 hr" style= "cursor:pointer" href= "/job/my-folder/job/my-jobname/44/console" class= "progress-bar " title= "Started 20 hr ago<br> Estimated remaining time: 10 hr" > <tbody><tr> <td style= "width:66%;" class= "progress-bar-done" ></td> <td style= "width:34%" class= "progress-bar-left" ></td> </tr></tbody></table> </div></td> <td class= "pane" > <a href= "/job/my-folder/job/my-jobname/44/" class= "model-link inside" ><wbr>#44</a> </td> <td class= "pane" align= "center" valign= "middle" > <a onclick= " if (confirm(&quot;Are you sure you want to abort My Folder » My Job Name #44?&quot;)) new Ajax.Request(&quot;/computer/node%20name%20anonymized/executors/2/stop&quot;); return false ;" href= "/computer/node%20name%20anonymized/executors/2/stop" class= "stop-button-link" ><img src= "/ static /175f9ede/images/16x16/stop.png" alt= "terminate this build" style= "width: 16px; height: 16px; " class= "icon-stop icon-sm" ></a></td></tr> Using the "collect garbage" option in Chrome did not remove these objects. I looked at the (closure) objects to see what they were using the "Retainers" panel in the Dev Tools. These closures are referenced from "Detached EventListener" objects, referenced by "Detached InternalNode" objects, referenced by "Detached HTMLAnchorElement" objects. I did one more thing: "Break on subtree modification" for div#side-panel in the Chrome dev tools. That brought me to hudson-behavior.js, "p.replaceChild(node, hist)", which offers some insight into why there are a tree of detached elements: some element inside the tree may have an EventListener that is still referenced somewhere. I tried the Firefox Dev tools next (as well as to confirm that the memory leak also occurs in Firefox. Tracing call stacks on memory allocations showed that there's an issue with the call stack starting at the very next function call, "Behaviour.applySubtree(node)": The _createResponder function in prototype.js seems to be the culprit. It seems to always add the passed element to its CACHE array. It never cleans up the CACHE as far as I can tell, unless you unload the page in IE. Because the CACHE contains a persistent reference to the HTMLAnchorElements which get event handlers, all of the anchor elements created in the Build Executor Window are persisted. That adds up fast. I hope that makes this bug fixable - I had a window where these were visible and it used up 3GB of memory after a few hours.
          Hide
          trejkaz trejkaz added a comment -

          We're seeing something like this also. Overnight, my machine will get to 6~8GB per tab with Jenkins open in it.

          I tracked it down to a cache in Prototype as well, but mine was 'cache' in lowercase instead of 'CACHE' in uppercase.

          Show
          trejkaz trejkaz added a comment - We're seeing something like this also. Overnight, my machine will get to 6~8GB per tab with Jenkins open in it. I tracked it down to a cache in Prototype as well, but mine was 'cache' in lowercase instead of 'CACHE' in uppercase.
          Hide
          trejkaz trejkaz added a comment - - edited

          Here's a story by someone else with the same problem (though they discovered it in TeamCity, instead of Jenkins) who ended up forking Prototype to fix it.

          http://kirblog.idetalk.com/2011/06/prototype-17-memory-leak.html

          They have a patched version of prototype.js with a fix in it. Could this just be incorporated into Jenkins?

          Show
          trejkaz trejkaz added a comment - - edited Here's a story by someone else with the same problem (though they discovered it in TeamCity, instead of Jenkins) who ended up forking Prototype to fix it. http://kirblog.idetalk.com/2011/06/prototype-17-memory-leak.html They have a patched version of prototype.js with a fix in it. Could this just be incorporated into Jenkins?
          Hide
          trejkaz trejkaz added a comment -
          Show
          trejkaz trejkaz added a comment - Is this the only copy of prototype.js to fix? https://github.com/jenkinsci/jenkins/blob/master/war/src/main/webapp/scripts/prototype.js
          Hide
          zioschild Sven Appenrodt added a comment - - edited

          Maybe as fast workaround you can ignore the problem in IE and fix it for chrome and firefox by canging

              if (Object.isUndefined(registry)) {
                CACHE.push(element);
                registry = Element.retrieve(element, 'prototype_event_registry', $H());
              }
          

          to

              if (Object.isUndefined(registry)) {
                if (!Prototype.Browser.IE) {
                  CACHE.push(element);
                }
                registry = Element.retrieve(element, 'prototype_event_registry', $H());
              }
          

          This will prevent the elements getting stored in the array for 'not IE' browser...

          BTW: there is an additional leak in Element.Storage in case of the element is not clearly deleted with the purge command. A workaround may be:

            getStorage: function(element) {
              if (!(element = $(element))) return;
          
              //var uid;
              if (element === window) {
                uid = 0;
              } else {
                if (typeof element._prototypeUID === "undefined"){
                  element._prototypeUID = Element.Storage.UID++;
          	  }
                uid = element._prototypeUID;
              }
          	// -------------------
          	//PATCHED: do not (!!) use Storage - see below
              //if (!Element.Storage[uid])
              //  Element.Storage[uid] = $H();
              //return Element.Storage[uid];
          	
          	// -------------------
          	// PATCHED Storage is not cleaned correctly when not calling "purge" and leads to memleaks
          	// for reusage we add the storage directly to the element
          	if (!element._prototypeStorage)	
          		 element._prototypeStorage = $H();
          	return element._prototypeStorage;
            },
          
          Show
          zioschild Sven Appenrodt added a comment - - edited Maybe as fast workaround you can ignore the problem in IE and fix it for chrome and firefox by canging if ( Object .isUndefined(registry)) { CACHE.push(element); registry = Element.retrieve(element, 'prototype_event_registry' , $H()); } to if ( Object .isUndefined(registry)) { if (!Prototype.Browser.IE) { CACHE.push(element); } registry = Element.retrieve(element, 'prototype_event_registry' , $H()); } This will prevent the elements getting stored in the array for 'not IE' browser... BTW: there is an additional leak in Element.Storage in case of the element is not clearly deleted with the purge command. A workaround may be: getStorage: function(element) { if (!(element = $(element))) return ; // var uid; if (element === window) { uid = 0; } else { if (typeof element._prototypeUID === "undefined" ){ element._prototypeUID = Element.Storage.UID++; } uid = element._prototypeUID; } // ------------------- //PATCHED: do not (!!) use Storage - see below // if (!Element.Storage[uid]) // Element.Storage[uid] = $H(); // return Element.Storage[uid]; // ------------------- // PATCHED Storage is not cleaned correctly when not calling "purge" and leads to memleaks // for reusage we add the storage directly to the element if (!element._prototypeStorage) element._prototypeStorage = $H(); return element._prototypeStorage; },

            People

            • Assignee:
              Unassigned
              Reporter:
              gga Guillaume Gautreau
            • Votes:
              7 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated: