Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25876

Log files retained in memory until process is killed

    Details

    • Similar Issues:

      Description

      Apologies if this is in any way redundant to another issue; I found none in my searches.

      I have a central Jenkins system with 16 slaves with a total of 37 executors – at any given time, about 10 executors are in use, and each executor produces about 200KB of logging.

      One problem I've had is that Jenkins is running on a machine that only has about 50GB of space available which means that when my logs run up to 50GB, Jenkins effectively locks up. I've tried to set up some maintenance jobs to take care of this – logrotate, a job within Jenkins itself to run /safeRestart – but those each have their own problems. Logrotate only takes care of the logs that are not actively held in memory (which, for my problems, are all of them); safeRestart does not actually halt the process so the logs are still held in memory.

      As a result, every week or so I have to find a moment where I can prevent new jobs from spawning and manually restart Jenkins (actually killing the old process and starting a new one). This is a pain in the neck and while I could set up a cronjob to do this for me, I have jobs running 24/7 and so it's not a "simple" matter of setting up a script to detect if I have any jobs running, and if not, put it into do-not-start-jobs mode, and at that point, run a hard restart. At the very least, that shouldn't be a work-around I'm forced to do.

      This may be a case of my not using Jenkins "correctly," and I'd love to hear how the community resolves this problem, but I don't see any configuration options in 'Manage Jenkins' or the general Jenkins configuration screen (including under 'Disk usage').

      What I'd prefer is for logs to be handled in a more efficient manner – I'm not the expert, so you may have a better solution, but my optimal solution would be for:

      • Loggers to maintain a maximum (configurable) size, either as a global max or an independent logger max.
      • When logging exceeds that size, files circulate in memory and on disk, effectively locking the maximum file size available.

      I realize that Jenkins is very plugin friendly and perhaps this would be a better plugin than to be built into the core but I don't currently see a good alternative to the workaround of manually killing and restarting my CI system on a weekly basis and in general that seems like a poor business practice.

      Any help is greatly appreciated and I'm willing to help implement solutions if needed.

      Thank you!

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          This is about the slave logs in JENKINS_HOME/logs/slaves/? Or what specifically?

          Show
          danielbeck Daniel Beck added a comment - This is about the slave logs in JENKINS_HOME/logs/slaves/? Or what specifically?
          Hide
          cantimatt Matt Newkirk added a comment -

          Ah. Yes, I think this is specific to JENKINS_HOME/logs/slaves/ – I had initially thought that all logging was involved but my problem seems to be specific to the slaves logs. Thanks for pointing that out.

          Show
          cantimatt Matt Newkirk added a comment - Ah. Yes, I think this is specific to JENKINS_HOME/logs/slaves/ – I had initially thought that all logging was involved but my problem seems to be specific to the slaves logs. Thanks for pointing that out.
          Hide
          danielbeck Daniel Beck added a comment -

          "produces about 200KB of logging" in what time frame? 200KB*37 is still basically nothing if that's the total.

          Show
          danielbeck Daniel Beck added a comment - "produces about 200KB of logging" in what time frame? 200KB*37 is still basically nothing if that's the total.
          Hide
          cantimatt Matt Newkirk added a comment -

          Heh, I really don't know what I meant by 200KB of logging. Right now, my jenkins instance is consuming about 50GB of space (in the slaves logs) every 4 days or so; in the end, I'm less concerned with the rate of space taken so much as the inability to maintain the degree of space taken. I'm not sure why the standard log rotation feature does not seem to apply to these slave logs.

          Show
          cantimatt Matt Newkirk added a comment - Heh, I really don't know what I meant by 200KB of logging. Right now, my jenkins instance is consuming about 50GB of space (in the slaves logs) every 4 days or so; in the end, I'm less concerned with the rate of space taken so much as the inability to maintain the degree of space taken. I'm not sure why the standard log rotation feature does not seem to apply to these slave logs.

            People

            • Assignee:
              Unassigned
              Reporter:
              cantimatt Matt Newkirk
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: