Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47172

Provide more granular control of pipeline flownode persistence

    Details

    • Similar Issues:
    • Sprint:
      Pipeline - October, Pipeline - December

      Description

      As a pipeline user, I WANNA GO FAST.  

      Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions.  As-is, every action we attach triggers a persistence cycle and we attach several usually.

      We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. 

      Work items:

      • Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
      • Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
      • Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
      • Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions 

      This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected.  An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java
            src/main/java/org/jenkinsci/plugins/workflow/support/PipelineIOUtils.java
            src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogActionImpl.java
            src/main/java/org/jenkinsci/plugins/workflow/support/storage/BulkFlowNodeStorage.java
            src/main/java/org/jenkinsci/plugins/workflow/support/storage/FlowNodeStorage.java
            src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java
            src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/AbstractStorageTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/BulkStorageTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/MockFlowExecution.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamStorageTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/StorageTestUtils.java
            http://jenkins-ci.org/commit/workflow-support-plugin/5f8c0433d1b0d8d5f5062eedba0b791ffad449ec
            Log:
            Merge pull request #47 from svanoort/defer-action-write-mk2

            Granular flownode persistence and new node storage engine JENKINS-47172 JENKINS-47173

            Compare: https://github.com/jenkinsci/workflow-support-plugin/compare/c0c4bbef3e36...5f8c0433d1b0

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java src/main/java/org/jenkinsci/plugins/workflow/support/PipelineIOUtils.java src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/support/storage/BulkFlowNodeStorage.java src/main/java/org/jenkinsci/plugins/workflow/support/storage/FlowNodeStorage.java src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/AbstractStorageTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/BulkStorageTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/MockFlowExecution.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamStorageTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/StorageTestUtils.java http://jenkins-ci.org/commit/workflow-support-plugin/5f8c0433d1b0d8d5f5062eedba0b791ffad449ec Log: Merge pull request #47 from svanoort/defer-action-write-mk2 Granular flownode persistence and new node storage engine JENKINS-47172 JENKINS-47173 Compare: https://github.com/jenkinsci/workflow-support-plugin/compare/c0c4bbef3e36...5f8c0433d1b0
            Hide
            svanoort Sam Van Oort added a comment -

            Feature released with workflow-support 2.17 – and consumed by downstream plugins

            Show
            svanoort Sam Van Oort added a comment - Feature released with workflow-support 2.17 – and consumed by downstream plugins

              People

              • Assignee:
                svanoort Sam Van Oort
                Reporter:
                svanoort Sam Van Oort
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: