Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41945

Eventual NPE while running many concurrent declarative pipelines

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: workflow-cps-plugin
    • Labels:
      None
    • Environment:
      openjdk version "1.8.0_45-internal"
      Ubuntu 15.04
      workflow-cps 2.26
      pipeline-model-definition 1.0.1
      core 2.32.2
    • Similar Issues:

      Description

      After an hour or so of running 101 concurrent "hello world" pipeline scripts, the Jenkins server becomes unresponsive. The first signs of trouble in the log around here - I'll attach the entire log as well. The attached one probably has a bunch of noise in it from some earlier work.

      The pipeline script itself looks exactly like this:

      pipeline {
          
          agent any
          
          stages {
                  stage("TestingTesting") {
                  steps {
                      sh 'echo Hello world'
                  }
              }
          }
      }
      
      INFO: DeclarativePipelines/inlineThing34 #32 completed: SUCCESS
      Feb 10, 2017 8:27:44 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution cleanUpHeap
      WARNING: failed to clean up memory from Owner[DeclarativePipelines/inlineThing37/32:DeclarativePipelines/inlineThing37 #32]
      java.lang.NullPointerException
      	at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57)
      	at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
      	at java.lang.reflect.Field.get(Field.java:393)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpGlobalClassSet(CpsFlowExecution.java:1034)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:966)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:968)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:968)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:968)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:957)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:957)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpHeap(CpsFlowExecution.java:941)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:363)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      

        Attachments

        1. hs_err_pid27051.log
          453 kB
        2. jenkins.log
          781 kB
        3. jenkins--failure.log
          4.01 MB
        4. logs.tar.gz
          617 kB
        5. screenshot-1.png
          screenshot-1.png
          861 kB

          Issue Links

            Activity

            Hide
            kshultz Karl Shultz added a comment -

            The standard pipelines ran overnight, and are still running this morning.

            Show
            kshultz Karl Shultz added a comment - The standard pipelines ran overnight, and are still running this morning.
            Hide
            kshultz Karl Shultz added a comment -

            Ran a test with 60 declarative pipelines running every minute, instead of 100. Still hit an out of memory condition. I'll attach a tarball of the logs. Inside that, jenkins.log.1 has a lot more historical data in it. jenkins.log is sort of its last gasp. Jenkins also generated an error report file. I'll attach that as well. logs.tar.gz hs_err_pid27051.log

            Show
            kshultz Karl Shultz added a comment - Ran a test with 60 declarative pipelines running every minute, instead of 100. Still hit an out of memory condition. I'll attach a tarball of the logs. Inside that, jenkins.log.1 has a lot more historical data in it. jenkins.log is sort of its last gasp. Jenkins also generated an error report file. I'll attach that as well. logs.tar.gz hs_err_pid27051.log
            Hide
            jglick Jesse Glick added a comment -

            These standard pipelines feel like they run considerably faster than the declarative ones from before.

            Not surprising; I think to compare apples to apples you would need to make the standard Pipeline scripts much more complicated.

            Show
            jglick Jesse Glick added a comment - These standard pipelines feel like they run considerably faster than the declarative ones from before. Not surprising; I think to compare apples to apples you would need to make the standard Pipeline scripts much more complicated.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/1155b889193713f7fedaef8de8688128136d4fe4
            Log:
            [FIXED JENKINS-41945] Ignore null elements in ClassInfo.globalClassSet.items.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java http://jenkins-ci.org/commit/workflow-cps-plugin/1155b889193713f7fedaef8de8688128136d4fe4 Log: [FIXED JENKINS-41945] Ignore null elements in ClassInfo.globalClassSet.items.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/6fe6ea969a36ceebbfe81a9ad587ec251da8d0f9
            Log:
            Merge pull request #107 from jglick/cleanUpGlobalClassSet-NPE-JENKINS-41945

            JENKINS-41945 Ignore null elements in ClassInfo.globalClassSet.items

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java http://jenkins-ci.org/commit/workflow-cps-plugin/6fe6ea969a36ceebbfe81a9ad587ec251da8d0f9 Log: Merge pull request #107 from jglick/cleanUpGlobalClassSet-NPE- JENKINS-41945 JENKINS-41945 Ignore null elements in ClassInfo.globalClassSet.items

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                kshultz Karl Shultz
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: