Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52165

Use push rather than pull for durable task logging

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      Pipeline Nodes and Processes 2.22

      Description

      Retroactively filing an issue originally treated as part of JENKINS-38381, since it is independently mergeable and has some independent benefits even when using existing (per-step, master-based) log storage.

      Historically, when running durable tasks (sh and kin), running output was handled by having the master send a callable to the agent at intervals, initially short (¼s) but growing exponentially up to some maximum (15s) if the process seems to be idle, and resetting to short again if and when fresh output is detected.

      Not only is there up to a 15s delay in displaying new output, this is wasteful of master and network resources when the process is idle for a long time; and sending a UserRequest and corresponding response involves a fair amount of Java serialization.

      By contrast, non-durable processes (such as those created by Launcher in a freestyle build) use RemoteOutputStream to send content from the agent to master as soon as it is available, minimizing network traffic; Remoting is also able to optimize this kind of traffic by sending low-overhead Chunk packets of tailored sizes.

      Durable tasks should switch from the policy of pulling log output to having the agent push log output. The content cannot be detected immediately, since we are effectively tailing a log file, but it can be detected quickly after the log file is updated with minimal overhead.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Link This issue blocks JENKINS-38381 [ JENKINS-38381 ]
            Hide
            jglick Jesse Glick added a comment -

            This change supersedes the code affected by JENKINS-37575.

            Show
            jglick Jesse Glick added a comment - This change supersedes the code affected by JENKINS-37575 .
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-37575 [ JENKINS-37575 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "durable-task PR 60 (Web Link)" [ 21102 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-durable-task-step PR 63 (Web Link)" [ 21103 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "JEP-210 (Web Link)" [ 21243 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-52954 [ JENKINS-52954 ]
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            Resolution Fixed [ 1 ]
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Released in Durable Task version 1.26 and Pipeline Nodes and Processes 2.22.

            Show
            dnusbaum Devin Nusbaum added a comment - Released in Durable Task version 1.26 and Pipeline Nodes and Processes 2.22.
            dnusbaum Devin Nusbaum made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            Released As Pipeline Nodes and Processes 2.22
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-durable-task-step PR 84 (Web Link)" [ 21956 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-53888 [ JENKINS-53888 ]
            Hide
            jglick Jesse Glick added a comment -

            Disabling this mode in workflow-durable-task-step 2.25 until some reported issues can be tracked down and fixed.

            Show
            jglick Jesse Glick added a comment - Disabling this mode in workflow-durable-task-step 2.25 until some reported issues can be tracked down and fixed.
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54643 [ JENKINS-54643 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54081 [ JENKINS-54081 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: