As a user of pipeline, I'm tired of it chewing up my storage IOPS and spitting them out, and honestly I don't care about recovering from dirty JVM crashes sometimes.
Can I please have a durability setting (or settings) to signal when I do and do not care about pipelines being durable enough to survive masters' failures, and when I do not care, stop writing to disk as much?
- workflow-api - APIs for durability
- workflow-support - potentially use the durability API to determine when persistence is invoked (maybe)
- workflow-cps - track and use the durability settings to control persistence of the program, execution, flownodes, and if I'm feeling particularly clever, logs.
- workflow-job - whatever is needed to pass in durability settings
- workflow-multibranch - properties for durability that can be attached to branches via some sort of branch property policy.