Details

    • Similar Issues:

      Description

      Hi,
      I found memory leak in stapler in class
      https://github.com/stapler/stapler/blob/master/core/src/main/java/org/kohsuke/stapler/framework/io/LargeText.java

      the problem is caused by condition on row 213.

      
              if(completed) {
                  // write everything till EOF
                  byte[] buf = new byte[1024];
                  int sz;
                  while((sz=f.read(buf))>=0)
                      os.write(buf,0,sz);
             }
      

      Unfortunately the fact that building is finished does not mean that we can write everything what remains.
      Our Jenkins instance fell due to this bug. The scenario is quite easy (especially in case of smartfrog plugin where is used only full console, no skipping bytes).
      1. Run job which creates a great log (several GB)
      2. Open a log console of running build which log has had already several GB)
      3. Open full console (without skipping bytes) and let it open in browser
      4. Stop this build (so the log is finished and completed in aforementioned code is true)
      5. You get out of memory (in case that your application has less memory then the size of log), because whole log which was not written is written in one shot to the memory because used output stream is in this case class CharSpool which write everything into memory.

      I suggest not to use completed and write the rest of log in the same way as it is done with completed=false.

      Please add your opinions into comments. Thanks

        Attachments

          Activity

          Hide
          abayer Andrew Bayer added a comment -

          So I'm probably out of my depth here but thought I'd try to understand what's happening. How are we not writing everything? I'm not clear on what the difference, in practice, is between what's being done in the completed case and what's being done otherwise...

          Show
          abayer Andrew Bayer added a comment - So I'm probably out of my depth here but thought I'd try to understand what's happening. How are we not writing everything? I'm not clear on what the difference, in practice, is between what's being done in the completed case and what's being done otherwise...
          Hide
          lvotypkova Lucie Votypkova added a comment -

          Hi, it is about what is being done in the completed case. I will try to be more clear. The current behaviour is that jenkins is serving log by small parts (several lines) in case that build is not done yet (build is still running it is not completed) - and your browser is continuously asking for new data. On opposite case if your build is done (completed) jenkins offers user the tail of log of build and in case that he requires all log, jenkins writes it whole dirrectly to the browser (no javascript, no cashing in jenkins master memory). The problem is when build is not finished and is still running during displaying it to user.

          jenkins gives log by small parts by javascript/ajax - browser continuously asked for new data until builds is finished (in this case it obtains information that there will be no new information and browser stop asking for it). These new data are read in small parts into jenkins memory and sent to the browser as response. The memory leak happens in time when the build is finished during displaying log by parts to the user. So the build is finished and user does not have displayed whole log yet. In this case current behavior decides to sent user the rest of log (which still can be several GB) and inform browser that this is all (do not ask for new information), but it is done by the same algorithm - read into jenkins memory and sent to the browser as response. Is it more clearer?

          Show
          lvotypkova Lucie Votypkova added a comment - Hi, it is about what is being done in the completed case. I will try to be more clear. The current behaviour is that jenkins is serving log by small parts (several lines) in case that build is not done yet (build is still running it is not completed) - and your browser is continuously asking for new data. On opposite case if your build is done (completed) jenkins offers user the tail of log of build and in case that he requires all log, jenkins writes it whole dirrectly to the browser (no javascript, no cashing in jenkins master memory). The problem is when build is not finished and is still running during displaying it to user. jenkins gives log by small parts by javascript/ajax - browser continuously asked for new data until builds is finished (in this case it obtains information that there will be no new information and browser stop asking for it). These new data are read in small parts into jenkins memory and sent to the browser as response. The memory leak happens in time when the build is finished during displaying log by parts to the user. So the build is finished and user does not have displayed whole log yet. In this case current behavior decides to sent user the rest of log (which still can be several GB) and inform browser that this is all (do not ask for new information), but it is done by the same algorithm - read into jenkins memory and sent to the browser as response. Is it more clearer?
          Hide
          jglick Jesse Glick added a comment -

          IIUC the problem is the fact that LargeText.doProgressText uses CharSpool to send a X-Text-Size header. While one solution would in fact be to use multiple requests to send the tail of the log, another would be to get rid of reliance on this header and stream the content directly, allowing the client to figure out for itself how many bytes it received (taking into account newline translation I suppose).

          Show
          jglick Jesse Glick added a comment - IIUC the problem is the fact that LargeText.doProgressText uses CharSpool to send a X-Text-Size header. While one solution would in fact be to use multiple requests to send the tail of the log, another would be to get rid of reliance on this header and stream the content directly, allowing the client to figure out for itself how many bytes it received (taking into account newline translation I suppose).
          Hide
          jglick Jesse Glick added a comment -

          This is not a memory “leak” per se but rather an attempt to overallocate memory on one occasion.

          Show
          jglick Jesse Glick added a comment - This is not a memory “leak” per se but rather an attempt to overallocate memory on one occasion.

            People

            • Assignee:
              Unassigned
              Reporter:
              lvotypkova Lucie Votypkova
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: