Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33761

Ability to disable Pipeline durability and "resume" build.

    XMLWordPrintable

    Details

    • Sprint:
      Blue Ocean 1.4 - beta 2, Pipeline - December
    • Similar Issues:

      Description

      Having some state being generated at the each node during execution, resuming builds after jenkins restarts or nodes reboots are just not feasible sometimes and can result in infinite hangs in some cases.  Also, providing durability results in extensive writes to disk that can bring performance crashing down. 

      It would be great to be able to specify that jobs don't resume upon interruptions, but rather just fail. This would increase the robustness of the system ideally, since upon nodes restarting, they quickly pick up jobs that tries to resume and hangs exhausting all available executors quickly.

      Implementation notes:

      • Requires a new OptionalJobProperty on the job, optionally a new BranchProperty in workflow-multibranch-plugin that echoes that same property
      • Needs some way to signal to storage (workflow-support) and execution (workflow-cps) that the pipeline is running with resume OFF to hint that they can use faster nondurable execution.

        Attachments

          Issue Links

            Activity

            Hide
            svanoort Sam Van Oort added a comment -

            Mike Kozell Have you tried the betas for the pipeline plugins that are currently in the experimental update center?  I'm fairly sure I fixed an error of this category when hardening the work in workflow-cps – this is also the same beta that provides the ability to prevent individual flows from resuming.  With a little work in the script console it should be possible to write a quick script to invoke that on all currently running builds. 

            Show
            svanoort Sam Van Oort added a comment - Mike Kozell  Have you tried the betas for the pipeline plugins that are currently in the experimental update center?  I'm fairly sure I fixed an error of this category when hardening the work in workflow-cps – this is also the same beta that provides the ability to prevent individual flows from resuming.  With a little work in the script console it should be possible to write a quick script to invoke that on all currently running builds. 
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java
            src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java
            src/main/java/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty.java
            src/main/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty.java
            src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty/config-details.jelly
            src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/config-details.jelly
            src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/help.html
            src/test/java/org/jenkinsci/plugins/workflow/job/MemoryCleanupTest.java
            src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunRestartTest.java
            src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunTest.java
            src/test/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobPropertyTest.java
            http://jenkins-ci.org/commit/workflow-job-plugin/5d3b91a68514d74422cc4ec5bc67d99418d7962c
            Log:
            Merge pull request #75 from svanoort/disable-pipeline-resume-JENKINS-33761

            Provide job property for durability hints & add ability to disable pipeline resume JENKINS-33761

            Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/2dfc94ac80bc...5d3b91a68514

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowJob.java src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java src/main/java/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty.java src/main/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty.java src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DisableResumeJobProperty/config-details.jelly src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/config-details.jelly src/main/resources/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobProperty/help.html src/test/java/org/jenkinsci/plugins/workflow/job/MemoryCleanupTest.java src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunRestartTest.java src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunTest.java src/test/java/org/jenkinsci/plugins/workflow/job/properties/DurabilityHintJobPropertyTest.java http://jenkins-ci.org/commit/workflow-job-plugin/5d3b91a68514d74422cc4ec5bc67d99418d7962c Log: Merge pull request #75 from svanoort/disable-pipeline-resume- JENKINS-33761 Provide job property for durability hints & add ability to disable pipeline resume JENKINS-33761 Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/2dfc94ac80bc...5d3b91a68514
            Hide
            svanoort Sam Van Oort added a comment -

            Released with... uh, well take a look at the Jenkins Pipeline Handbook entry on scaling pipeline for versions.

            Show
            svanoort Sam Van Oort added a comment - Released with... uh, well take a look at the Jenkins Pipeline Handbook entry on scaling pipeline for versions.
            Hide
            gregcovertsmith Greg Smith added a comment -

            For those watching, found direct link Sam mentioned:

            https://jenkins.io/doc/book/pipeline/scaling-pipeline/

            Show
            gregcovertsmith Greg Smith added a comment - For those watching, found direct link Sam mentioned: https://jenkins.io/doc/book/pipeline/scaling-pipeline/
            Hide
            mkozell Mike Kozell added a comment - - edited

            Sam Van Oort

            After upgrading Jenkins with the following, I was not able to reproduce the issue after a build timeout, cancelling a build, and restarting Jenkins in the middle of a build.

            Jenkins 2.89.4
            Pipeline 2.5
            Pipeline API 2.26
            Pipeline Nodes and Processes 2.19
            Pipeline Step API 2.14
            Scripts Security 1.41
            durabilityHint=PERFORMANCE_OPTIMIZED
            org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty
            Groovy Sandbox = disabled
            Java = 1.8.0_162

            Although my jobs correctly didn't resume after Jenkins restart, I did see the message below in the build logs.

            Resuming build at Sat Feb 24 06:38:10 UTC 2018 after Jenkins restart
             [Pipeline] End of Pipeline
             java.io.IOException: Cannot resume build – was not cleanly saved when Jenkins shut down.
            Show
            mkozell Mike Kozell added a comment - - edited Sam Van Oort After upgrading Jenkins with the following, I was not able to reproduce the issue after a build timeout, cancelling a build, and restarting Jenkins in the middle of a build. Jenkins 2.89.4 Pipeline 2.5 Pipeline API 2.26 Pipeline Nodes and Processes 2.19 Pipeline Step API 2.14 Scripts Security 1.41 durabilityHint=PERFORMANCE_OPTIMIZED org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty Groovy Sandbox = disabled Java = 1.8.0_162 Although my jobs correctly didn't resume after Jenkins restart, I did see the message below in the build logs. Resuming build at Sat Feb 24 06:38:10 UTC 2018 after Jenkins restart [Pipeline] End of Pipeline java.io.IOException: Cannot resume build – was not cleanly saved when Jenkins shut down.

              People

              • Assignee:
                svanoort Sam Van Oort
                Reporter:
                jtilander Jim Tilander
              • Votes:
                47 Vote for this issue
                Watchers:
                49 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: