Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44483

Large console logs cause infinite loops in slave

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: pipeline, remoting
    • Labels:
    • Environment:
      We run Jenkins using the official docker image version 2.46.2, docker version 17.03.1-ce, Amazon EC2 plugin 1.36, Ubuntu 14.04, Oracle java version 1.8.0_131 for the slave.jar process
    • Similar Issues:

      Description

      I have just been investigating a problem in our jenkins setup that I think might be related to JENKINS-25218. We're using the EC2 plugin and running builds that generate quite large logs (230 MB). At some point during the build, the master loses track of the log and just starts logging the same block of text from the log over and over as long as I let it. The build completes successfully on the slave and nothing bad appears in the Node log in the jenkins UI. But the master continues to fill up the filesystem with the same repeated text forever. I changed the build to log much less and now this isn't happening. We're running 2.46.2. Could this potentially be one of the edge cases?

        Attachments

          Issue Links

            Activity

            chris_phillips Chris Phillips created issue -
            chris_phillips Chris Phillips made changes -
            Field Original Value New Value
            Environment We run Jenkins usin the official docker image version 2.46.2, docker version 17.03.1-ce, Amazon EC2 plugin 1.36, Ubuntu 14.04, Oracle java version 1.8.0_131 for the slave.jar process We run Jenkins using the official docker image version 2.46.2, docker version 17.03.1-ce, Amazon EC2 plugin 1.36, Ubuntu 14.04, Oracle java version 1.8.0_131 for the slave.jar process
            oleg_nenashev Oleg Nenashev made changes -
            Component/s pipeline [ 21692 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels pipeline
            jglick Jesse Glick made changes -
            Link This issue duplicates JENKINS-37575 [ JENKINS-37575 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue depends on JENKINS-38381 [ JENKINS-38381 ]
            gonzzz82 Dima Rudaev made changes -
            Attachment workaround.py [ 39704 ]
            gonzzz82 Dima Rudaev made changes -
            Comment [ Hi guys, I encountered the same issue, and I want to share my observations and a workaround for it :

            It less dependent on a size of a log and more depends on frequency of prints, it happens when it has a huge amount of prints in a very short time period e.g: 1000000 prints in  less then 1 sec.
             

            The idea of workaround is to redirect the output and print in batches (in the attached example of workaround I redirect the stderr of  my_script.py and I allow 1000 prints per second with frequency of  one print of 10 lines each 0.01 second):

             [^workaround.py]

            Hope it helps !!

              ]
            gonzzz82 Dima Rudaev made changes -
            Attachment workaround.py [ 39704 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Duplicate [ 3 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                chris_phillips Chris Phillips
              • Votes:
                4 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: