Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47173

Offer a high-performance storage engine for pipeline at some cost to resumability

    Details

    • Similar Issues:
    • Sprint:
      Pipeline - October, Pipeline - December

      Description

      As a user of pipeline, I WANNA GO FAST.  

      To do that, I'm okay with sacrificing some of the ability to resume a pipeline after a hard Jenkins crash, as long as it means pipelines run fast.  It would be nice if I can still resume the pipeline after a clean/safe restart of the master, though!

      This differs from https://issues.jenkins-ci.org/browse/JENKINS-30896 because for this we're sacrificing some of the guarantees pipeline normally provides.

      Concrete needs:

      • Implement storage engine (workflow-support)
      • Implement options to select that storage engine (workflow-job)
      • Implement ability to RUN that storage engine (workflow-cps)

        Attachments

          Issue Links

            Activity

            svanoort Sam Van Oort created issue -
            svanoort Sam Van Oort made changes -
            Field Original Value New Value
            Epic Link JENKINS-47170 [ 185575 ]
            svanoort Sam Van Oort made changes -
            Link This issue is related to JENKINS-33761 [ JENKINS-33761 ]
            svanoort Sam Van Oort made changes -
            Link This issue is related to JENKINS-30896 [ JENKINS-30896 ]
            svanoort Sam Van Oort made changes -
            Link This issue is blocked by JENKINS-47172 [ JENKINS-47172 ]
            svanoort Sam Van Oort made changes -
            Issue Type Improvement [ 4 ] New Feature [ 2 ]
            jamesdumay James Dumay made changes -
            Sprint Pipeline - October [ 406 ]
            svanoort Sam Van Oort made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            svanoort Sam Van Oort added a comment -

            Based on a discussion I had today, it may be helpful to have a Jenkins system property (or option) to control what the default storage engine and Durability Mode for pipeline are.

            Show
            svanoort Sam Van Oort added a comment - Based on a discussion I had today, it may be helpful to have a Jenkins system property (or option) to control what the default storage engine and Durability Mode for pipeline are.
            jamesdumay James Dumay made changes -
            Sprint Pipeline - October [ 406 ] Pipeline - October, Pipeline - December [ 406, 446 ]
            svanoort Sam Van Oort made changes -
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java
            src/main/java/org/jenkinsci/plugins/workflow/support/PipelineIOUtils.java
            src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogActionImpl.java
            src/main/java/org/jenkinsci/plugins/workflow/support/storage/BulkFlowNodeStorage.java
            src/main/java/org/jenkinsci/plugins/workflow/support/storage/FlowNodeStorage.java
            src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java
            src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/AbstractStorageTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/BulkStorageTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/MockFlowExecution.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamStorageTest.java
            src/test/java/org/jenkinsci/plugins/workflow/support/storage/StorageTestUtils.java
            http://jenkins-ci.org/commit/workflow-support-plugin/5f8c0433d1b0d8d5f5062eedba0b791ffad449ec
            Log:
            Merge pull request #47 from svanoort/defer-action-write-mk2

            Granular flownode persistence and new node storage engine JENKINS-47172 JENKINS-47173

            Compare: https://github.com/jenkinsci/workflow-support-plugin/compare/c0c4bbef3e36...5f8c0433d1b0

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java src/main/java/org/jenkinsci/plugins/workflow/support/PipelineIOUtils.java src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/support/storage/BulkFlowNodeStorage.java src/main/java/org/jenkinsci/plugins/workflow/support/storage/FlowNodeStorage.java src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/AbstractStorageTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/BulkStorageTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/MockFlowExecution.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamStorageTest.java src/test/java/org/jenkinsci/plugins/workflow/support/storage/StorageTestUtils.java http://jenkins-ci.org/commit/workflow-support-plugin/5f8c0433d1b0d8d5f5062eedba0b791ffad449ec Log: Merge pull request #47 from svanoort/defer-action-write-mk2 Granular flownode persistence and new node storage engine JENKINS-47172 JENKINS-47173 Compare: https://github.com/jenkinsci/workflow-support-plugin/compare/c0c4bbef3e36...5f8c0433d1b0
            Hide
            svanoort Sam Van Oort added a comment -

            Released with workflow-support 2.17

            Show
            svanoort Sam Van Oort added a comment - Released with workflow-support 2.17
            svanoort Sam Van Oort made changes -
            Status In Progress [ 3 ] Closed [ 6 ]
            Resolution Done [ 10000 ]
            Hide
            basil Basil Crow added a comment -

            I'm afraid I have to report a possible regression. After the following plugin upgrade this weekend to pull in Durablity Settings, a significant number of my pipeline jobs failed to re-hydrate cleanly:

            @@ -72,14 +72,14 @@ timestamper:1.8.9
             token-macro:2.3
             windows-slaves:1.3.1
             workflow-aggregator:2.5
            -workflow-api:2.24
            +workflow-api:2.25
             workflow-basic-steps:2.6
             workflow-cps-global-lib:2.9
            -workflow-cps:2.42
            -workflow-durable-task-step:2.17
            -workflow-job:2.16
            -workflow-multibranch:2.16
            +workflow-cps:2.43
            +workflow-durable-task-step:2.18
            +workflow-job:2.17
            +workflow-multibranch:2.17
             workflow-scm-step:2.6
             workflow-step-api:2.14
            -workflow-support:2.16
            +workflow-support:2.17
            

            Here is the stack trace:

            Resuming build at Sat Jan 27 13:35:48 PST 2018 after Jenkins restart
            Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: ???
            Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: ???
            Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting for next available executor on machine1
            Waiting for next available executor on machine2
            Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting for next available executor on machine2
            [Pipeline] End of Pipeline
            Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting for next available executor on machine2
            java.lang.IllegalStateException: JENKINS-37121: something already locked /path/to/regression-run-workspace/jenkins-regression/machine1/workspace@20
            	at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:75)
            	at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:51)
            	at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92)
            	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            Caused: java.io.IOException: Failed to load build state
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:814)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:812)
            	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:861)
            	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
            	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            Finished: FAILURE
            

            I can already see one problem:

            Waiting for next available executor on machine1
            Waiting for next available executor on machine2
            

            This is not only different from what I used to see, but it doesn't make logical sense. When I've done restarts like this in the past, before I upgraded the plugins to get Durability Settings, this used to say:

            Waiting for next available executor on machine1
            

            Here is my theory about what is going on. The job has this structure:

            node('machine2') {
                [...]
            }
            node('machine1') {
                node('machine2') {
                    [...]
                }
                [...] <- long-running operation that was running before the restart
            }
            

            The long-running operation was running before I restarted Jenkins. Note that I did not change any durability settings, I just upgraded the plugins to the latest version. My jobs are running at MAX_SURVIVABILITY. When Jenkins came up and re-hydrated, this previously would just wait for machine1 to be back online, then the job would resume. Apparently Jenkins now wants to wait for both machine1 and machine2 to have free executors before the job will resume, even though we are no longer running on machine2 (because the inner node block has gone out of scope and we have disconnected from machine2). In this case, machine2 happens to have a very small # of executors, so the job can't immediately claim an executor slot on machine2. Instead, it has to wait a long time for one to free up. By this time, a brand new (and unrelated) job has started and claimed the workspace, so we hit the exception.

            If my theory is correct, there is a regression in the recent Durability Settings work. We should not be trying to grab an executor slot for a node that we have already disconnected from when rehydrating. This wasn't the case before the plugin upgrade, but is the case after the plugin upgrade. This took down a large percentage of my regression test runs this weekend.

            Another (possibly separate) issue is that we continued to try to grab an executor on machine2 AFTER the "End of Pipeline" message, then threw an IllegalStateException.

            Show
            basil Basil Crow added a comment - I'm afraid I have to report a possible regression. After the following plugin upgrade this weekend to pull in Durablity Settings, a significant number of my pipeline jobs failed to re-hydrate cleanly: @@ -72,14 +72,14 @@ timestamper:1.8.9 token-macro:2.3 windows-slaves:1.3.1 workflow-aggregator:2.5 -workflow-api:2.24 +workflow-api:2.25 workflow-basic-steps:2.6 workflow-cps-global-lib:2.9 -workflow-cps:2.42 -workflow-durable-task-step:2.17 -workflow-job:2.16 -workflow-multibranch:2.16 +workflow-cps:2.43 +workflow-durable-task-step:2.18 +workflow-job:2.17 +workflow-multibranch:2.17 workflow-scm-step:2.6 workflow-step-api:2.14 -workflow-support:2.16 +workflow-support:2.17 Here is the stack trace: Resuming build at Sat Jan 27 13:35:48 PST 2018 after Jenkins restart Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: ??? Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: ??? Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting for next available executor on machine1 Waiting for next available executor on machine2 Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting for next available executor on machine2 [Pipeline] End of Pipeline Waiting to resume part of sybase-ase_full » SYBASE_RHEL69_16.0 #26: Waiting for next available executor on machine2 java.lang.IllegalStateException: JENKINS-37121: something already locked /path/to/regression-run-workspace/jenkins-regression/machine1/workspace@20 at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:75) at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:51) at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) Caused: java.io.IOException: Failed to load build state at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:814) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:812) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:861) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE I can already see one problem: Waiting for next available executor on machine1 Waiting for next available executor on machine2 This is not only different from what I used to see, but it doesn't make logical sense. When I've done restarts like this in the past, before I upgraded the plugins to get Durability Settings, this used to say: Waiting for next available executor on machine1 Here is my theory about what is going on. The job has this structure: node('machine2') { [...] } node('machine1') { node('machine2') { [...] } [...] <- long-running operation that was running before the restart } The long-running operation was running before I restarted Jenkins. Note that I did not change any durability settings, I just upgraded the plugins to the latest version. My jobs are running at MAX_SURVIVABILITY. When Jenkins came up and re-hydrated, this previously would just wait for machine1 to be back online, then the job would resume. Apparently Jenkins now wants to wait for both machine1 and machine2 to have free executors before the job will resume, even though we are no longer running on machine2 (because the inner node block has gone out of scope and we have disconnected from machine2). In this case, machine2 happens to have a very small # of executors, so the job can't immediately claim an executor slot on machine2. Instead, it has to wait a long time for one to free up. By this time, a brand new (and unrelated) job has started and claimed the workspace, so we hit the exception. If my theory is correct, there is a regression in the recent Durability Settings work. We should not be trying to grab an executor slot for a node that we have already disconnected from when rehydrating. This wasn't the case before the plugin upgrade, but is the case after the plugin upgrade. This took down a large percentage of my regression test runs this weekend. Another (possibly separate) issue is that we continued to try to grab an executor on machine2 AFTER the "End of Pipeline" message, then threw an IllegalStateException.
            Hide
            svanoort Sam Van Oort added a comment -

            Basil Crow Thank you for reporting your issues, and I'm sorry to hear they bit you rather hard.  I've been doing some investigation on this – it sounds like you may be hitting a pre-existing bug that we saw reported couldn't reproduce (very similar symptoms around use of nesting blocks and executors).  However whether it's pre-existing or a regression from this, if a nested pair of nodes enables us to reproduce (and thus solve it), then you can look forward to a fix soonishly. 

            Show
            svanoort Sam Van Oort added a comment - Basil Crow Thank you for reporting your issues, and I'm sorry to hear they bit you rather hard.  I've been doing some investigation on this – it sounds like you may be hitting a pre-existing bug that we saw reported couldn't reproduce (very similar symptoms around use of nesting blocks and executors).  However whether it's pre-existing or a regression from this, if a nested pair of nodes enables us to reproduce (and thus solve it), then you can look forward to a fix soonishly. 
            Hide
            svanoort Sam Van Oort added a comment -

            Basil Crow I do not appear to be able to reproduce what you describe with restarts using the new version – after digging deeper, my suspicion is that it might be an issue with the program data persisted before shutdown (using older plugin versions), or as you say it may be a race condition where a job somehow claims a workspace it is not entitled to (since it should be owned by another job) – though I've not seen such before. 

            The WorkspaceListLease un/pickling logic itself doesn't have any alterations, and we did a fair bit of testing around upgrade/downgrade scenarios.  But of course, this logic is quite subtle, complex, and in some cases prior bugs fixed in the course of this could only be observed if actions occurred in specific time windows.

             

            Have you observed / are you able to reproduce this behavior during restarts with the new plugin versions in place?

            Show
            svanoort Sam Van Oort added a comment - Basil Crow I do not appear to be able to reproduce what you describe with restarts using the new version – after digging deeper, my suspicion is that it might be an issue with the program data persisted before shutdown (using older plugin versions), or as you say it may be a race condition where a job somehow claims a workspace it is not entitled to (since it should be owned by another job) – though I've not seen such before.  The WorkspaceListLease un/pickling logic itself doesn't have any alterations, and we did a fair bit of testing around upgrade/downgrade scenarios.  But of course, this logic is quite subtle, complex, and in some cases prior bugs fixed in the course of this could only be observed if actions occurred in specific time windows.   Have you observed / are you able to reproduce this behavior during restarts with the new plugin versions in place?
            svanoort Sam Van Oort made changes -
            Labels performance performance project-cheetah
            Hide
            georgecnj George C added a comment -

            I'm also seeing something similar with a pipeline job. This happened after I did two "Reload Configuration From Disk" operations about 5 minutes apart. We see that the job's queue only shows builds up to #6, but builds newer than 6 exist. This is using Jenkins 2.46.3

             

             

            [Pipeline] End of Pipeline
            java.lang.IllegalStateException: JENKINS-37121: something already locked /Users/hudson/build/workspace/ci-ios-bento-sample-apps-apple
                    at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:75)
                    at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:51)
                    at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            Caused: java.io.IOException: Failed to load build state
                    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:697)
                    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:695)
                    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:744)
                    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
                    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)
            Finished: FAILURE

             

            Show
            georgecnj George C added a comment - I'm also seeing something similar with a pipeline job. This happened after I did two "Reload Configuration From Disk" operations about 5 minutes apart. We see that the job's queue only shows builds up to #6, but builds newer than 6 exist. This is using Jenkins 2.46.3     [Pipeline] End of Pipeline java.lang.IllegalStateException: JENKINS-37121: something already locked /Users/hudson/build/workspace/ci-ios-bento-sample-apps-apple         at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:75)         at org.jenkinsci.plugins.workflow.support.pickles.WorkspaceListLeasePickle$1.tryResolve(WorkspaceListLeasePickle.java:51)         at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) Caused: java.io.IOException: Failed to load build state         at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:697)         at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:695)         at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:744)         at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)         at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang. Thread .run( Thread .java:745) Finished: FAILURE  
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-49086 [ JENKINS-49086 ]
            Hide
            basil Basil Crow added a comment -

            Sam Van Oort Thanks for your reply and for looking into this.

            Have you observed / are you able to reproduce this behavior during restarts with the new plugin versions in place?

            No, I haven't observed this again and I haven't been able to reproduce it. I have also started doing clean shutdowns with the /exit API to ensure that the build queue gets properly flushed to disk before doing Jenkins upgrades.

            Thanks again!

            Show
            basil Basil Crow added a comment - Sam Van Oort Thanks for your reply and for looking into this. Have you observed / are you able to reproduce this behavior during restarts with the new plugin versions in place? No, I haven't observed this again and I haven't been able to reproduce it. I have also started doing clean shutdowns with the /exit API to ensure that the build queue gets properly flushed to disk before doing Jenkins upgrades. Thanks again!
            Hide
            svanoort Sam Van Oort added a comment -

            Basil Crow George C I just released workflow-cps 2.47 and workflow-job 2.18 which include a ton of fixes to related logic and likely fix the cause(s) of this – if you see this issue recur, could you please open a new JIRA issue and assign to me. Thanks!

            Show
            svanoort Sam Van Oort added a comment - Basil Crow George C I just released workflow-cps 2.47 and workflow-job 2.18 which include a ton of fixes to related logic and likely fix the cause(s) of this – if you see this issue recur, could you please open a new JIRA issue and assign to me. Thanks!

              People

              • Assignee:
                svanoort Sam Van Oort
                Reporter:
                svanoort Sam Van Oort
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: