Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47172

Provide more granular control of pipeline flownode persistence

    Details

    • Similar Issues:
    • Sprint:
      Pipeline - October, Pipeline - December

      Description

      As a pipeline user, I WANNA GO FAST.  

      Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions.  As-is, every action we attach triggers a persistence cycle and we attach several usually.

      We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. 

      Work items:

      • Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
      • Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
      • Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
      • Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions 

      This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected.  An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.

        Attachments

          Issue Links

            Activity

            svanoort Sam Van Oort created issue -
            svanoort Sam Van Oort made changes -
            Field Original Value New Value
            Epic Link JENKINS-47170 [ 185575 ]
            svanoort Sam Van Oort made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            svanoort Sam Van Oort made changes -
            Summary Pipeline Saves FlowNodes Multiple Times Redundantly Provide more granular control of pipeline persistence
            svanoort Sam Van Oort made changes -
            Summary Provide more granular control of pipeline persistence Provide more granular control of pipeline flownode persistence
            svanoort Sam Van Oort made changes -
            Link This issue blocks JENKINS-47173 [ JENKINS-47173 ]
            jamesdumay James Dumay made changes -
            Sprint Pipeline - October [ 406 ]
            svanoort Sam Van Oort made changes -
            Description As a pipeline user, I WANNA GO FAST.  :)

            Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions.  As-is, every action we attach triggers a persistence cycle and we attach several usually.

            We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. 

            Work items:
             * Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
             * Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
             * Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
             * Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions 
             * Provide a way for Steps to mark that they are going to attach their own actions before running work, so we can wait on that to persist them. Useful for steps that add labels, etc.

            This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected.  An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.
            As a pipeline user, I WANNA GO FAST.  :)

            Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions.  As-is, every action we attach triggers a persistence cycle and we attach several usually.

            We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. 

            Work items:
             * Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
             * Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
             * Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
             * Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions 

            This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected.  An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.
            jamesdumay James Dumay made changes -
            Sprint Pipeline - October [ 406 ] Pipeline - October, Pipeline - December [ 406, 446 ]
            svanoort Sam Van Oort made changes -
            Status In Progress [ 3 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]
            svanoort Sam Van Oort made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            svanoort Sam Van Oort made changes -
            Status Reopened [ 4 ] Closed [ 6 ]
            Resolution Done [ 10000 ]
            svanoort Sam Van Oort made changes -
            Labels performance performance project-cheetah

              People

              • Assignee:
                svanoort Sam Van Oort
                Reporter:
                svanoort Sam Van Oort
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: