Devin Nusbaum Daniel Beck IIUC what is being described, this actually maps to a really obnoxious bug I've been investigating for several weeks that has been blocking release of a fix/improvement to persistence (with threading implications due to the use of synchronization during I/O).
It seems to be on the Pipeline end itself though – it relates to how the Pipeline job interacts with the OneOffExecutor created when it throws an AsynchronousExecution upon running. The Pipeline may even get marked as completed, but somehow the listener that terminates the Pipeline is not invoked – I'm simplifying grossly here, of course, in reality there's a very complex asynchronous chain of events with a complex threading model underlying all this.
The behavior can be traced to the upon-resume situation if state was incompletely persisted AFAICT, but it requires somewhat precise timing to trigger the events.
Some situations that cause this behavior have probably been solved by prior fixes, but clearly not all of them.