When pods get deleted for any reason, there is a log/exception like so:
The job then appears to hang indefinitely until a timeout is reached or it's stopped manually.
In our use case (k8s using preemptible vms) we actually expect pods to be deleted mid build and want to be able to handle pod deletion with a retry.
I have not been able to find a way to handle this in declarative syntax.
For testing, using a very simple declarative example:
But the exception does not actually trigger the failure block when the pod is killed.
Is there currently any best practice to handle the deletion of a pod? Are there any timeout parameters that would be useful in this case?
I'm happy to add a PR to the Readme after learning