Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49153

Sh step repeatedly spews its last n output lines until a timeout is hit

    Details

    • Similar Issues:

      Description

      We find this happens often with steps like: 

      sh "docker-compose -f docker-compose-ci.yml logs containername"

      Or test steps that produce a lot of output.

      The outcome is often that a job uses ~5GB of space until it hits our global 1h timeout, even though that step does not have that much output. It's definitely Jenkins spinning on repeating the last few ~100 lines, although the process has already returned.

      As a workaround, if the above line is replaced with:

       

      sh "docker-compose logs containername | gzip > containername.log.gz"
      archive("containername.log.gz")

      then we don't ever experience the hang.

      But for test output that isn't ideal.

      Note that this issue also happens if the command itself has a timeout:

      go test -timeout 10m

      It can still hang for an hour, spewing tons of logs that just repeat the last few pages.

       

        Attachments

          Issue Links

            Activity

            Hide
            abayer Andrew Bayer added a comment -

            Jesse Glick Sam Van Oort - any thoughts on this? Do we have an effective maximum size for durable task logs?

            Show
            abayer Andrew Bayer added a comment - Jesse Glick Sam Van Oort - any thoughts on this? Do we have an effective maximum size for durable task logs?
            Hide
            jglick Jesse Glick added a comment -

            Known bug in a code path slated to be rewritten for other reasons.

            Show
            jglick Jesse Glick added a comment - Known bug in a code path slated to be rewritten for other reasons.
            Hide
            svanoort Sam Van Oort added a comment -

            So much pain with classic logging.

            Show
            svanoort Sam Van Oort added a comment - So much pain with classic logging.
            Hide
            jzila John Zila added a comment -

            Thanks for the response! Should I be using a different type of logging?

            Show
            jzila John Zila added a comment - Thanks for the response! Should I be using a different type of logging?
            Hide
            jglick Jesse Glick added a comment -

            No, there is known workaround in the current codebase.

            Show
            jglick Jesse Glick added a comment - No, there is known workaround in the current codebase.

              People

              • Assignee:
                Unassigned
                Reporter:
                jzila John Zila
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: